With AI, MIT researchers teach a robot to build furniture by just asking

Researchers at the Massachusetts Institute of Technology this week announced they developed a “speech-to-reality” system. This AI-driven workflow allows the MIT team to provide input to a robotic arm and “speak objects into existence,” creating things like furniture in as little as five minutes.

The system uses a robotic arm mounted on a table that can understand spoken input from a human. For example, a person could tell the robot, “I want a simple stool,” and the robot would then construct the stool out of the modular components.

So far, the university researchers have used the speech-to-reality system to create stools, shelves, chairs, a small table, and even decorative items such as a dog statue.

MIT project focuses on bits and atoms

“We’re connecting natural language processing, 3D generative AI, and robotic assembly,” explained Alexander Htet Kyaw, an MIT graduate student and Morningside Academy for Design (MAD) fellow. “These are rapidly advancing areas of research that haven’t been brought together before in a way that you can actually make physical objects just from a simple speech prompt.”

The idea started when Kyaw, a graduate student in the departments of Architecture and Electrical Engineering and Computer Science, took Prof. Neil Gershenfeld’s course, “How to Make Almost Anything.”

In that class, he built the speech-to-reality system. After the class, Kyaw continued working on the project at the MIT Center for Bits and Atoms (CBA), directed by Gershenfeld. He collaborated with graduate students Se Hwan Jeon of the Department of Mechanical Engineering and Miana Smith of CBA.

How does the system work?

The speech-to-reality system begins with speech recognition that processes the user’s request using a large language model (LLM). Next, 3D generative AI creates a digital mesh representation of the object, and a voxelization algorithm breaks down the 3D mesh into assembly components.

After that, geometric processing modifies the AI-generated assembly to account for fabrication and physical constraints associated with the real world. This includes the number of components, overhangs, and connectivity of the geometry.

This is followed by the creation of a feasible assembly sequence and automated path planning for the robotic arm to assemble physical objects from user prompts.

By using natural language, the system makes design and manufacturing more accessible to people without expertise in 3D modeling or robotic programming, asserted the MIT team. And, unlike 3D printing, which can take hours or days, this system can assemble objects within minutes.

“This project is an interface between humans, AI, and robots to co-create the world around us,” Kyaw said. “Imagine a scenario where you say ‘I want a chair,’ and within five minutes, a physical chair materializes in front of you.”

Kyaw plans to make improvements to the system

Examples of objects constructed by a robotic arm in response to voice commands like ‘a shelf with two tiers’ and ‘I want a tall dog.’ | Source: Alexander Kyaw, MIT

The MIT team said it has immediate plans to improve the weight-bearing capability of the furniture by changing the means of connecting the cubes from magnets to more robust connections.

“We’ve also developed pipelines for converting voxel structures into feasible assembly sequences for small, distributed mobile robots, which could help translate this work to structures at any size scale,” Smith said.

The team used modular components to eliminate the waste that goes into making physical objects by disassembling and then reassembling them into something different. For instance, they could turn a sofa into a bed when the user no longer needs the sofa.

Because Kyaw also has experience using gesture recognition and augmented reality to interact with robots in the fabrication process, he is currently working on incorporating both speech and gestural control into the speech-to-reality system. Kyaw said he was inspired by the replicators in the Star Trek franchise and the robots in the animated film Big Hero 6.

“I want to increase access for people to make physical objects in a fast, accessible, and sustainable manner,” he said. “I’m working toward a future where the very essence of matter is truly in your control. One where reality can be generated on demand.”

The team presented its paper, “Speech to Reality: On-Demand Production using Natural Language, 3D Generative AI, and Discrete Robotic Assembly,” at the Association for Computing Machinery (ACM) Symposium on Computational Fabrication held at MIT on Nov. 21.