Microsoft Copilot can now ‘see’ what’s on your screen in Edge

Copilot Vision is present escaped successful Edge.

Copilot Vision, Microsoft’s AI adjunct diagnostic that tin construe what’s connected your surface and assistance you usage apps, is present disposable for escaped usage wrong the Edge browser, Mustafa Suleyman, CEO of Microsoft AI, announced connected Bluesky today. Vision is simply a “talk-based experience,” arsenic Microsoft calls it, meaning you usage it by speaking into the air, past waiting for Copilot to respond.

Suleyman says if you opt into the feature, Copilot Vision tin “literally spot what you spot connected screen.” Suleyman suggests having Copilot Vision usher you done a look portion you navigator oregon having it “decode” occupation descriptions “and leap close into customized interrogation prep oregon screen missive brainstorming.” (Although it might not beryllium the champion idea to usage AI for your resume.) According to a Microsoft enactment page, “Copilot Vision whitethorn item portions of the surface to assistance you find applicable information,” but it doesn’t really click links oregon bash thing connected your behalf.

Broader, system-wide Copilot Vision features are inactive limited to Copilot Pro subscribers. With a subscription, Vision expands beyond Edge, letting you inquire it to assistance you usage features successful Photoshop oregon video editing software, oregon usher you done a crippled similar Minecraft, arsenic it did for The Verge’s Tom Warren earlier this month.

To effort retired Copilot Vision, unfastened this nexus to Microsoft’s website successful the Edge browser. That should springiness you a punctual to opt into the feature, and erstwhile you’ve fixed permission, you tin unfastened the Copilot sidebar portion connected a website, click the microphone icon, and your Vision league begins, signified by a chime and your browser changing its hue. 

Or that’s however it should go. In my case, it took a mates of tries earlier Edge asked if I wanted to opt in. And erstwhile I could opt successful and initiate a Vision session, the controls ne'er appeared — arsenic of this writing, I simply person a connection floating implicit the bottommost of my browser that says “One moment…” But I’m utilizing a reasonably old, underpowered laptop, truthful your mileage whitethorn vary.

According to Microsoft, the institution logs Copilot’s responses to you but doesn’t cod your inputs, images, oregon leafage content. When you’re acceptable to halt sharing your surface with Copilot, you tin either extremity the league oregon adjacent the browser window.