Windows 11 is getting even more AI-powered tools that can "see" the user's screen. The new Copilot Vision tool has raised privacy and security concerns because it sends data to Microsoft's servers.

Microsoft seems to be going all-in on AI these days, so it's worth recapping its different tools. Copilot is its overall package of AI tools designed to integrate with Windows, including a chatbot that answers questions and integrated tools in Word that offer to help write documents. There's also Copilot+ which adds extra features and only works on specific computers with more powerful processors.

There's also Recall, a more controversial Windows 11 feature that takes regular screenshots of the user's desktop to create a local database of past activity. The idea is to make it easier to ask specific questions about information the user has previous worked on, without them having to remember the specific application, document or website they had open.

Windows Recall prompted a major privacy debate and Microsoft made it an opt-in only tool, while noting all data is stored and processed on the user's computer, stored in encrypted format, and anything that appears to be sensitive information such as a credit card number is not captured.

Live Screen Analysis

Now Microsoft is adding Copilot Vision, which brings together elements of all these features by analyzing and answering questions about what's currently displayed on the screen. It's only available on Copilot+ machines and is only triggered when the user actively enables it, rather than constantly running.

Microsoft says the tool "...acts as your second set of eyes, able to analyze content, help when you're lost, provide insights and answer your questions as you go. Whether you're browsing, working or deep in a project, Copilot Vision offers instant insights and answers." (Source: windows.com)

It also hypes the way the tool can not only answer questions about how to perform a particular task in a window or app, but will even actively show where to click and what to do.

Remote Servers

While opinions will vary about how well the tool works and whether its purpose is worthwhile, the real fuss is likely to be about the fact that it doesn't run entirely on the user's machine. Instead, some data is sent to Microsoft servers to allow for more powerful (and thus more accurate) analysis and text recognition.

Microsoft says it won't store this data long-term and will not use it to personalize ads or to train its AI models. (Source: theregister.com)

What's Your Opinion?

Would you use this feature if it was available on your PC? Does it matter if the data goes to Microsoft's servers? Is making the feature optional and not running in the background a suitable balance of control for users?