Now the robot is able to grab objects that no other robot has been able to grab before. – A real “Matrix robot,” says researcher Ekrem Misimi.

The robot is now able to grab objects of various sizes and geometries with much higher accuracy than before.

– By training the AI agent, i.e., the software, in a virtual world, it can be used directly in the real world without additional training, says SINTEF researcher Ekrem Misimi. He calls it groundbreaking – and explains why:

– An agent is something that acts, performs an action. An AI agent, however, is specialized software that uses AI to perceive its environment, perform actions, and learn from its experiences to achieve specific goals.

Both at home and in industry, we encounter objects with strange geometric shapes, smooth surfaces, or those that can only be grasped from the side. All of these variations are challenging for a robot trying to grab an object. Just think of all the challenges that different types of food pose in terms of fragility, softness, and shape. Grasping them without damaging or compromising their quality is a significant challenge.

Additionally, there are objects you must reach for and possibly grasp from the side. When these types of objects come in all shapes and sizes, it becomes very difficult to automate tasks and operations. Simply because, until now, robots neither know how to perform tasks with such objects nor how to handle new, unknown objects they’ve never encountered before.

This is something a research team from the GentleMAN project, consisting of SINTEF and NTNU, has figured out. They are now teaching robots how to grasp such objects. At the same time, they continuously update the claw’s position and orientation in real time.

– We are developing new knowledge and robot-based solutions to enable robots to grasp all types of objects, including food or other industrial objects with challenging shapes. By combining the robot’s learned ability to perceive through 3D vision and artificial intelligence, we are now able to teach robots these grasping operations more effectively, says Misimi.

– A robot without knowledge input is essentially dumb. That’s why we have to provide it with the information it needs for the tasks it must perform. Teaching the robot to grasp and perform other advanced manipulations of both rigid and, especially, deformable objects remains extremely challenging, admits Misimi.

However, by creating a simulated training environment, they have managed to train the robot on several scenarios where it has the opportunity to learn on its own and explore until it masters a task in a satisfactory manner.

– The hardest part has been getting the robot to learn the task in the simulation and then transferring that knowledge from the simulation environment to the real world, explains Sverre Herland, PhD student in the GentleMAN project.

This is especially challenging because an object looks different in the real world than it does in a simulation. Therefore, it’s not a given that the AI agent will succeed.

The AI technique used allows the agent to learn on its own with minimal human guidance.

– We also see that the agent is good at generalizing the learned skills, meaning it can handle unknown objects that it has never seen before, which is confirmed by the high grasping accuracy with such objects, the researchers explain.

Here, you can take a look at the researchers’ experiment:
https://www.youtube.com/watch?v=bwPf8Imvook

So, what can this grasping technology be used for? Many of the grasping and manipulation tasks humans perform in the food industry are often complicated, at least for robots, and also require practice for humans.

– For a robot to perform such operations close to what we can do with our own hands, new technology based on artificial intelligence needs to be in place, and that’s what we’re contributing to with this project, says Misimi.

He believes that robotics will be crucial to keeping the industry in Europe – thus contributing, for example, to more sustainable food production while also supporting food security in a time when the global need for food is rising.

– The technology we’re developing will, among other things, mean that food no longer needs to be sent to the other side of the globe for processing. The quality of the food will also improve since it doesn’t have to be transported so far, says Misimi.

Interested in mre material on artificial intelligence? Check out the website Elements of AI.

GentleMAN is a 4.5-year project funded by IKTPLUSS in the Research Council of Norway with 16 million NOK. The project started in 2019 and is concluding now.

Source: IEEE International Conference on Robotic Automation – ICRA 2024. Sverre Herland, Kerstin Bach, Ekrem Misimi. 6DoF Closed-Loop Grasping with Reinforcement Learning.