3D Sensors in Machine Vision

For several years now the technologies used in machine vision have gotten more sophisticated, and at the same time the applications for quality checks are solved with a new ease. Industrial image processing seems to have focused on the challenge of 3D sensors for their purpose and more and more methods are refined and ready to use. So machine vision has taken a turn towards 3D. Therefore a lot of applications can be solved more economically, including the inspection of adhesive beads or weld seams, as well as bin-picking of unsorted parts or precision measurement of sheet metal in the production process.

And the software to evaluate and measure in a point cloud has also developed considerably over the last few years: from only transforming image data into point clouds to measuring, counting and matching in a point cloud. So what about the ways to get 3D image data?

As most players in the image processing industry know, there are several different ways to get a 3D image. Probably the most traditional is the laser triangulation method, which can be used in vertical markets such as wood and lumber, rubber and tire, automotive and measurement of the axis, metal and cast iron industry, or other applications such as measurement of roadway surfaces.

For laser triangulation a precise calibration of the camera on a structured light source such as a laser line projection is necessary to assure a high sample rate of more than 1 kHz under even high environmental temperatures. Usually the test object is moving under the 3D sensor to capture a 3D point-cloud. That means a camera inspects the laser line which is projected onto the object and calculates the height information from the line profile. While moving the object under the camera several profiles are created, which are used for the completion of a three dimensional image. The typical setup contains a laser, which is directly over the test object and a camera, which is mounted in an angle of 30° to the laser. But other geometric combinations of laser and camera are also possible. For example, to get a more accurate height resolution the angle between camera and laser can be widened. But it has to be noted that with a smaller angle, more light is reaching the camera and therefore the evaluation is more stable.

The height resolution can be calculated for each setup.

The height resolution can be calculated for each setup, just as shown here.

Nowadays there is more and more software, which can deal with 3D image data. The software can transform the captured data into point clouds, which can be compared directly and makes the analysis much easier.

Besides the laser triangulation method, there is also a method called “fringe projection.” The basic principle is also the triangulation, but the whole surface of the test object is captured with one take. The laser projects the light in a fringe pattern, so the object does not have to be moved under the sensor. The light is projected from an angle of 30° to the object and the camera looks straight down onto the object.

From at least three fringe projections in rapid succession with sinusoidal intensity progression, the accurate lateral fringe positions for all points of the surface are determined.

The measuring field can be scaled from less than a millimeter to more than one meter, but the resolution is scaled respectively too. To make a long story short, fringe projection can be used for small and big test objects, due to its measurement speed and also precise measurement in high resolution. Fringe projection is used for industrial inspection tasks such as inspection of form deviations, completeness, the position of component parts or volume measurement, etc. But one has to keep in mind that the fringe projection is sensitive to ambient light.

The 3D stereo camera is another method. It has existed for many years and is used more and more for robot or commissioning applications. The stereo image processing uses the same principle as the human eye, the stereoscopic offset. To get a 3D image the method is based on the calibration of two cameras. But as test objects do not always have the same specific characteristics, a random pattern projection is often used. The distortion and the offset of the pattern between the two cameras allow a cost effective solution, with an accuracy between the fringe projection and time-of-flight methods.

Then there is the way of training the software with images of good and bad parts. In case of the connector images of bent pins can train the system to recognize defect connectors. And this training process can be extended by continuing to feed the system with images.

A few years ago it was said that of all the methods the time-of-flight (ToF) method is least suitable for industrial use because of its limited resolution. Most ToF-cameras have a resolution below VGA and the z-resolution is relatively low, with a repeat accuracy in centimeters. But there are already some cameras on the market with a megapixel resolution. Time-of-flight cameras use a technique similar to the radar engineering. The integrated illumination sends an IR-impulse and the sensor measures the time, which is needed to reflect the light.

It is used more and more these days for 3D object detection, but not for precise measurements. A growing application area is the loading and unloading of pallets with a robot.

As mentioned earlier, the software to process machine vision such as measurements in a point cloud plays a major role in 3D vision. So the “brain” for 3D exists. But does it learn like the human brain? And how to train it?

The conventional way is of course to program the software in such a way that the inspection program detects the bad parts, such as a connector with bent pins. The user programs which height, position, wobble circle, etc. the pins should have and every deviation from the programmed characteristics is a bad part.

Then there is the way of training the software with images of good and bad parts. In the case of the connector images of bent pins, you can train the system to recognize defect connectors. And this training process can be extended by continuing to feed the system with images.

And what about deep learning, the much used word and new hobby horse of image processing? But what is it?

Artificial neural networks are generally presented as systems of interconnected “neurons” which exchange messages between each other.

Deep learning is just another name for artificial neural networks (ANN for short) but in a more refined and easier avatar. They have existed for more than 40 years.

Artificial neural networks are generally presented as systems of interconnected “neurons” which exchange messages between each other. The connections have numeric weights that can be tuned based on experience, making neural nets adaptive to inputs and capable of learning.

Is it an advantage? It’s a growing trend in machine learning due to some favorable results in applications where the target function is very complex and the data sets are large.

In deep learning, ANNs are automatically extracting features instead of manual extraction in feature engineering. Take example of image as input. Instead of us taking an image and hand computing features like distribution of colors, image histograms, distinct color count, etc., we just have to feed the raw images in ANN.

Deep learning helps in the automated world, where processes are not exactly predictable or else are running more effectively without being planned beforehand. Besides, deep learning creates the conditions, which make it possible for plants and robots to make independent and reliable decisions.

Machine learning use cases are for example: image classification, object detection, localization, medical imaging and interpretation, seismic imaging and interpretation, etc.

The conclusion is that the machine vision industry is looking with great expectations to new possibilities in 3D imaging as well as artificial neural networks and deep learning. Let’s see how it will develop.