ChatGPT Can Now Control a Robot Arm

Two researchers at UC Berkeley and ETH Zurich have harnessed the power of OpenAI's GPT-4o large language model to teach cheap robot arms to clean up spills.

It's a clever demonstration of how AI language models, like the one behind OpenAI's popular AI chatbot ChatGPT, can be harnessed to do tasks that on the surface have little to do with language.

It took the pair of roboticists only four days to teach the robot arms, which were given access only to a regular sponge, to identify a nearby spill.

A video shared by UC Berkeley roboticist Jannik Grothusen shows the robot being asked what it sees in front of it.

"I see a sponge and a small spill on a surface," the robot answered in perfect English.

"Explain what you do and clean up the table," the researcher instructed.

"I'll use the robot arm to clean the table surface," the robot replied helpfully. "First, I'll check for available motion skills to pick up the sponge and wipe the table, then I'll execute the sequence to clean the spill."

Without skipping a beat, the robot arm jumps into action and does exactly what it promised to do.

We built a GPT-4o-powered cleaning robot.- $250 for the robot arms- 4 days to build

Open source is truly democratizing the field of robotics.@KasparJanssen pic.twitter.com/DEJECQML0k

— Jannik Grothusen (@JannikGrothusen) November 2, 2024

According to Grothusen, the arms' motions were trained on roughly 100 demos.

The arms themselves are entirely open source and can be built at home with the help of a YouTube playlist.

A context-aware "multimode agent" called LangChain served as a framework to translate the input and output of the LLM into robot movements using reinforcement training.

Grothusen argued in a LinkedIn post that the experiment is a "proof-of-concept for a robot control architecture" that includes a "visual language model for human-robot interaction, reasoning, and orchestration."

It also "demonstrates how open-source is beginning to democratize the field of robotics," given the robot arm's affordable price tag and an entirely open-source learning algorithm.

Whether the concept will ever turn into a fully-fledged cleaning robot that can wipe up spills in your home, however, remains unclear.

More on AI and robots: Tesla's Robots Were Just Remotely Controlled Dummies, Analyst Confirms

Share This Article