Google Will Take Control Of Your PC

We recently reported on an AI model that could take control of a computer, for example to complete a form. Now a Google leak reveals its planning a similar technology.

Anthropic recently released a beta update to its AI model Claude. It includes the ability to move a mouse cursor by taking a screenshot and calculating the number of pixels to figure out how far and in what direction to move. The company gave the example of the tool automatically copying data from a spreadsheet to fill out an online form.

"Jarvis" Leaks

Now Google has let slip it's about to release a similar tool. It unintentionally added a listing in the Chrome extension store and quickly pulled it, but has confirmed that "Jarvis" will be launched in December.

While some users were able to download Jarvis before it was pulled, they weren't able to run it as it required specific permissions they couldn't yet enable. That means we only have limited details of what it does and how it does it.

It's a browser extension, which means users can simply install it in Chrome and use it without having to open any standalone applications.

The listing described Jarvis as "a helpful companion that surfs the web with you" and reportedly mentioned using it for tasks such as booking a flight or buying something online. (Source: theinformation.com)

December Debut

The Information had previously reported on the project, noting it will work through Google's own Gemini AI model. That's due an update in December, which is likely when Jarvis will be intentionally made public.

It seems to be a logical development of a long term effort by Google to offer more automation. In 2018 it revealed it was working on a similar tool called Duplex using different technology. Instead of "moving" a mouse, it was a voice assistant that could use synthesized speech to, for example, phone a restaurant and make a booking. (Source: standard.co.uk)

These technologies do raise some ethical and legal questions. For example, it's unclear how fraud protection for payment cards would apply if a bug (or a cyber attack) meant the tool "made" purchases that the user did not expect.

What's Your Opinion?

Are you excited by such technologies? Would you be more likely to use a tool that was an official Google-made Chrome extension? Do these solve real problems?