Fully automated processes have become an integral part of industrial production. Modern robots ensure the safe handling of finished products and workpieces. They often pick specific objects from a container to prepare them for further process steps, such as processing, assembly, separation, or packaging. Bin picking also plays a key role in quality assurance, where parts that are lying disorderly in a bin are automatically picked, then sorted and placed at their new location in the correct alignment, for example on a pallet. This allows for the items to be checked quickly and efficiently for any errors and to be rejected, if necessary. These robots are equipped with high-resolution 3D cameras and integrated machine vision software so they can “see” the respective objects and grasp them accurately.
Objects can have a wide range of geometries and shapes
It is important that the robots can precisely identify the parts to be inspected and their optimum gripping points. One challenge in this context is that objects can have a wide range of geometries and shapes. Therefore, a detailed CAD model of each individual object would have to be created for reliable identification rates. While this is a quite common workflow in industrial production, such a model might not be available in other scenarios – for example picking items in warehouses or picking deformable objects in the food industry such as fruit, vegetables, or bags of potato chips.
In these cases, the bin picking process must function without corresponding CAD models, which means that the exact shape of the objects is unknown in advance, making the gripping process more difficult. In addition, a crate can also contain several objects with different geometries as well as deformable objects, all of which still have to be gripped safely, even when the surface properties of the individual parts vary greatly – for example, they can have different textures, be partially transparent or even reflective.
And in manufacturing companies there can also exist a wide range of different parts, so the bin picking process not only has to cover many different geometries, but also varying object sizes and image data from different camera or sensor types. For each variation, the corresponding gripping processes must run smoothly to shorten throughput times, increase productivity, and reduce costs.
Reliably gripping objects with an unknown appearance
Addressing all these challenges requires a sophisticated and modern bin picking process. To ensure reliable object recognition and gripping processes, the use of highly developed machine vision systems is the ideal approach. These systems include, for example, a dedicated “3D Gripping Point Detection” technology. The most advanced versions have a decisive advantage compared to others of its kind: It can also be used to grip objects whose external appearance is not known in advance, and this paves the way for new bin picking applications. 3D Gripping Point Detection identifies potential gripping surfaces for vacuum suction cups on suitable objects so that they can be picked up reliably. A CAD model or other information regarding the appearance of the object is not required, which enables bin picking for a variety of object categories with different geometries. Another advantage is that the technology can also be used to reliably grip flexible, i.e. deformable objects.
In many cases, such features are based on artificial intelligence (AI), which in turn relies on a deep learning network that has been pre-trained with numerous images from industrial applications. This ensures a particularly reliable identification rate. The technology is an integral part of state-of-the-art machine vision software.
In addition, there are already development efforts that will lead to “3D Gripping Point Detection”-technologies to be supplemented with sophisticated enhancements for retraining, which will enable users to adapt the bin picking process more effectively and flexibly to specific hardware requirements or environmental conditions. This allows, among other advantages, for greater independence from certain sensor types and more consistent results in sophisticated application scenarios.
Perfectly positioning 60 plastic bags per minute
To take robot-assisted gripping of objects to the next level, high-end machine vision software offers additional features and technologies to combine with “3D Gripping Point Detection”. For example, a Spanish developer of digital inspection systems uses a machine vision solution to fully automate and optimize a pick-and-place application. The use case: The precise identification of plastic bags so that robots can recognize them and grip them securely. The challenges: The bags are translucent, contain various loose sets of small parts, and come in many different shapes. In some cases, they are also crumpled and completely disorganized on the conveyor belt, which makes identification and automatic gripping more difficult.
To ensure a reliable recognition rate and robust gripping processes, the system utilizes a combination of classic machine vision methods and modern deep learning technologies, all provided by one software. The system is comprehensively trained with sample images based on sophisticated AI algorithms, enabling the software to learn the many different characteristics and appearances of the bags. This ensures highly accurate identification results – even with an almost infinite variance of objects. Even overlapping and stacked bags can be reliably gripped and picked in this way. The machine vision software can analyze up to 60 bags per minute in detail and identify them precisely, enabling the automation of the entire bag positioning process and ensuring increased productivity.
Precisely gripping and processing knee implants with complex shapes
In another use case, machine vision software supports the robot-assisted, automated handling of components in the medical sector. The challenge: Reliably detecting, picking and processing randomly aligned and sometimes highly reflective components for knee implants with complex shapes. A 3D image processing application developed in collaboration with an Irish research institute makes it possible to specifically identify and locate the parts so that the robot can pick them up robustly and repeatably and place them down again safely. A particular challenge in this case was the wide range of surfaces of the implants – from matte to high-gloss and reflective. In addition, the components are partially concealed by container walls, arranged randomly, and must be picked and placed from boxes of different sizes.
To overcome these challenges, “shape-based 3D matching” is used. It can locate objects precisely and consistently, even under difficult conditions – for example if the parts are rotated, scaled, distorted in perspective, locally deformed, or partially covered, located outside the image, or subject to non-linear lighting fluctuations. As a result of using the technology, the healthcare company benefits from a reliable and efficient robot-based pick-and-place application that increases productivity and reduces costs.
