CN110032971B - Foreign object detection method and detection system for mobile platform based on monocular camera - Google Patents

Abstract

The invention discloses a mobile platform foreign object detection method and detection system based on a monocular camera. The system consists of an image acquisition module including a monocular camera and a main controller. First, an image acquisition module is installed on a periodic low-speed moving platform (such as a robotic arm), and images are collected at different positions of the mobile platform for multiple times and processed. For example, if it is determined that there is a foreign body, the main controller calculates the spatial position, size and color of the foreign body. and other information, and send the calculation results to the upper-level control system. With the help of a mobile platform, the invention uses a monocular camera to obtain information such as the position of the target, which can achieve the same effect as a multi-eye camera, and can be applied to the detection of whether a workpiece is dropped or other low-speed foreign objects in industrial sites (such as automated assembly lines).

Description

Foreign object detection method and detection system for mobile platform based on monocular camera

technical field

The invention relates to the field of machine vision and image processing, in particular to a foreign body state detection method and detection system when a foreign body occurs.

Background technique

The technology that uses computers to acquire and process images is called machine vision technology. To be precise, machine vision is the use of optical non-contact sensing devices to automatically receive and interpret images of real scenes to obtain information to control machines or processes. Image processing is a technology that uses computer data processing technology to process image data and obtain effective information. Human vision is best at qualitatively interpreting complex, unstructured scenes, but machine vision is good at quantitatively measuring structured scenes with advantages such as speed, accuracy, and repeatability. For example, on a production line, Machine vision systems can inspect hundreds or even thousands of components per minute. Equipped with cameras and optics of the appropriate resolution, machine vision systems can easily inspect details of objects too small to be seen by the human eye. The application of machine vision in industry can greatly reduce costs and human labor.

The information contained in an image is the digital information of a large number of pixels measured in the order of one hundred thousand or more; in a monochrome image, each pixel contains only one kind of grayscale information; in a color image, each pixel The pixels contain the luminance information of the three colors of RGB. The process of processing such a huge amount of information is quite complicated. If all machine vision researchers study from the lowest level of processing methods, it is obviously inefficient and unrealistic. Fortunately, in the field of machine vision, there is a powerful and open source OpenCV computer vision library. OpenCV is a cross-platform computer vision library based on a BSD license (open source) and can run on Linux, Windows, Android and Mac OS operating systems. It is lightweight and efficient - it consists of a series of C functions and a small number of C++ classes, and provides interfaces in languages such as Python, Ruby, and MATLAB, and implements many general algorithms in image processing and computer vision. OpenCV provides special data structures for image processing and a large number of library functions for processing images, which provides great convenience for machine vision research.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a falling object detection method and detection system based on a monocular camera, in which information such as the position, size and color of low-speed foreign objects can be obtained by using the monocular camera.

The method part of the technical solution for realizing the object of the present invention is: a method for detecting dropped objects on a mobile platform based on a monocular camera, comprising the following steps:

Step 1. Platform selection: select at least 2 key positions in the process of carrying the mobile platform and the platform movement;

Step 2. Obtain reference: collect environmental image information at all key positions as a reference image under the premise of no foreign matter;

Step 3, image acquisition: when the mobile platform moves to any key position, the environmental image information of the current position is collected;

Step 4. Image processing: compare the collected environmental image information with the reference image collected at the same key position to determine whether there is foreign matter; if there is foreign matter, go to step 5, otherwise go back to step 3, and set it in the user settings. The periodic coverage updates the reference image;

Step 5. Multi-point collection: move the platform to the next key position to collect environmental image information;

Step 6, information processing: Integrate the image information collected from at least two key positions to calculate the position, size and color of the foreign object.

A mobile platform drop detection system based on a monocular camera, comprising an image acquisition module, an image processing module and an information analysis module, wherein:

The image acquisition module is connected with the image processing module to obtain environmental image information, converts the obtained light analog information into digital RGB information through the photoelectric sensor, and transmits the information to the image processing module, and transmits time and position labels at the same time;

The image processing module is connected with the information analysis module. The image processing module stores the environmental image information with time and position labels in the memory. First, the current environmental image information is converted into the color gamut and the difference calculation is performed with the environmental image information in the state of no foreign matter. , and the result is binarized and cut, and finally the connected domain is detected after denoising, and the detected result is transmitted to the information analysis module;

The information analysis module is connected with the upper-level control system and the image acquisition module, and is used to process and summarize the information obtained by image processing; first, according to the detection results of the connected domain, the existence of foreign objects is judged. The location point continues to collect environmental image information, and reconfirms whether the foreign body exists. If it is determined that the foreign body exists, the information collected multiple times is collected to calculate the position, size and color information of the foreign body, and report to the superior control system for further processing; if there is no foreign body or After the second acquisition, it is found that the foreign body does not exist, and the image acquisition module is controlled to work normally.

Compared with the prior art, the present invention has the following significant advantages: (1) the effect of multi-camera depth detection can be achieved only by using a monocular camera; (2) redundant detection is allowed, and existing results are continuously corrected , it is convenient to improve the accuracy; (3) The mobile platform can be specially configured, or it can be directly installed on the existing periodic motion low-speed platform such as welding, assembling the robot arm, and has strong adaptability; (4) The image acquisition module allows to expand additional degrees of freedom, enabling flexible multi-view detection.

Description of drawings

FIG. 1 is a structural diagram of a mobile platform foreign object detection system based on a monocular camera of the present invention.

Figure 2 is a typical appearance structure of the system according to the present invention.

Figure 3 shows the expansion 1 degree of freedom installation method of the image acquisition module.

Figure 4 shows the extended 2-DOF installation method of the image acquisition module.

FIG. 5 is a flowchart of the method for detecting foreign objects on a mobile platform based on a monocular camera according to the present invention.

Figure 6 is a three-dimensional model installed on a low-degree-of-freedom manipulator.

FIG. 7 is a method for locating the spatial position of foreign objects.

Fig. 8 is the visual field state before and after the appearance of the foreign body at the key position A in Example 1

Fig. 9 is the result that the foreign matter is detected in the key position A after processing in Example 1

Fig. 10 is the visual field state before and after the appearance of the foreign body at the key position B in Example 1

Figure 11 is the result of detecting the presence of foreign matter at key position B after processing in Example 1

Detailed ways

The present invention will be further described below with reference to the accompanying drawings.

5 and 6, the present invention is based on a monocular camera-based mobile platform foreign body detection method, the steps are as follows:

Step 1. Platform selection: select at least 2 key positions in the process of carrying the mobile platform and the platform movement;

The mobile platform will move to the same spatial position and angle multiple times within a limited time, which is convenient for the image acquisition module to observe the same monitored area from different angles;

Step 2. Obtain reference: collect environmental image information at all key positions as a reference image under the premise of no foreign matter;

Set the update cycle of the reference image, for example, use the newly acquired image without foreign objects to cover the reference image every certain time or every several cycles. If the update period is too short, it will increase the computational burden. If the update period is too long, the error accumulation will affect the judgment effect due to slight changes in the environment, such as changes in sunlight intensity during the day;

Step 3, image acquisition: when the mobile platform moves to any key position, the environmental image information of the current position is collected;

Here, the key position point to be moved to is set as the key position A, the spatial coordinates are (xa, ya, za), and the axis angle of the image acquisition module is Da;

Step 4. Image processing: compare the collected environmental image information with the reference image collected at the same key position to determine whether there is foreign matter; if there is foreign matter, go to step 5, otherwise go back to step 3, and set it in the user settings. The periodic coverage updates the reference image;

After comparing the image information collected at the same key position in different periods and denoising, it can quickly determine whether there is foreign matter in the field of view;

Step 5. Multi-point collection: the mobile platform moves to the next key position to collect environmental image information;

Here, set the second key position point to be moved to the key position B, the spatial coordinates are (xb, yb, zb), and the axis angle of the image acquisition module is Db.

Step 6, information processing: Integrate the image information collected from at least two key positions to calculate the position, size and color of the foreign object.

By morphological processing of the acquired image, it can be obtained that the position of the foreign object position coordinate point T(xt, yt, zt) in the field of view of the key position A relative to the direction vector of point A is D1, as shown in Figure 7 As shown, the position in the visual field of the key position B relative to the direction vector of point B is D2. Combining the coordinates of point A and the coordinates of point B, the equations of the space straight lines AT and BT can be obtained, and the two space straight lines AT and BT can be calculated. The intersection of BT can get the coordinates of point T. Considering that there may be errors in the actual positioning process, resulting in no intersection between the two spatial straight lines calculated, if it is found that the two spatial straight lines do not intersect, take the midpoint of the common perpendiculars of the two different-plane straight lines as the spatial position point of the foreign object. .

Information such as the approximate size and color of the object can be determined from the spatial position points of the object and the original images obtained in steps 3 and 5.

As shown in Figure 1, the mobile platform foreign body detection system based on the monocular camera of the present invention includes an image acquisition module, an image processing module and an information analysis module, wherein:

The image acquisition module is connected with the image processing module to obtain environmental image information, converts the obtained light analog information into digital RGB information through the photoelectric sensor, and transmits the information to the image processing module, and transmits time and position labels at the same time;

The image processing module is connected with the information analysis module, and the image processing module stores the environmental image information with time and position labels in the memory; first, the current environmental image information is converted into the color gamut and the difference calculation is performed with the environmental image information in the state of no foreign matter. , and the result is binarized and cut, and finally the connected domain is detected after denoising, and the detected result is transmitted to the information analysis module;

The information analysis module is connected with the upper-level control system and the image acquisition module, and is used to process and summarize the information obtained by image processing; first, according to the detection results of the connected domain, the existence of foreign objects is judged. The location point continues to collect environmental image information, and reconfirms whether the foreign body exists. If it is determined that the foreign body exists, the information collected multiple times is collected to calculate the position, size and color information of the foreign body, and report to the superior control system for further processing; if there is no foreign body or After the second acquisition, it is found that the foreign body does not exist, and the image acquisition module is controlled to work normally.

Fig. 2 is a typical appearance structure of the system according to the present invention, wherein 1 is a lens, 2 is an industrial camera, 3 is a fixed base, 4 is a main controller, 5 and 6 are communication interfaces; Fig. 3 is an extension of the image acquisition module 1 The installation method of degrees of freedom, Figure 4 is the expansion of the image acquisition module with 2 degrees of freedom installation, where 7 is the horizontal turntable including the steering gear, and 8 is the vertical turntable including the steering gear. The field of view of the image acquisition module.

As shown in Figure 2, the system consists of an image acquisition module and a main controller, where 1 is a lens, 2 is an industrial camera, 3 is a base for fixing 1 and 2; 4 is a main controller, using an embedded control system, It is equipped with Linux system and has basic communication interfaces and network functions. The image processing module and information analysis module in Figure 1 are implemented in the form of main controller software. 5 is the DB9 interface for serial communication, and 6 is the RJ45 interface. for network communication. The camera mounting platform can provide fixed or extended degrees of freedom installation according to specific conditions. The higher the extended degree of freedom, the larger the detectable range and angle. The typical structure of the extended 1 degree of freedom is shown in Figure 3, where 7 is the horizontal The turntable can be precisely rotated by controlling the steering gear, so that the image acquisition module can have a wider horizontal field of view; the typical structure of extended 2 degrees of freedom is shown in Figure 4, where 8 is a vertical turntable, which can be precisely rotated by controlling the steering gear, The image acquisition module can have a wider vertical field of view.

When using the system of the present invention, first select a periodic low-speed moving platform, such as welding, assembling and palletizing robotic arms commonly used in industrial production; if there is no qualified moving platform at the working position, it can be specially designed for this system Configure periodic low-speed motion platforms such as low-degree-of-freedom robotic arms. After selecting the working platform, install the image acquisition module at the appropriate position of the working platform, such as the end, and connect the image acquisition module to the main controller through the data cable. The main controller and the image acquisition module can be mounted on the working platform at the same time or installed in a fixed position. A communication channel needs to be set up between the main controller and the controller of the working platform, so that the position information of the working platform can be obtained at any time. The data capacity of obtaining the position information of the working platform is not large, and it can work in a simplex, but it requires high real-time performance. There cannot be too much delay, it is recommended to use the RS485 serial communication protocol. A communication channel needs to be set up between the main controller and the upper-level control system, and information is reported in time when foreign objects are found.

The present invention will be further described below in conjunction with specific embodiments.

Example 1

The performance of a mobile platform foreign body detection system and its detection method based on a monocular camera proposed by the present invention is verified by experiments, and the situation is as follows:

1) Experimental conditions

Install the extended 1-DOF visual acquisition module on a 3-DOF robotic arm, and determine two key positions A (200, 0, 100) and B (0, 200, 100) (unit is mm, the same below), first collect environmental information, two The images collected at two key positions are shown in Figure 8(a) and Figure 10(a). After placing the foreign body, the images collected at the two key positions are shown in Figure 8(b) and Figure 10(b). 9 and Figure 11, since the image acquisition module has been calibrated in advance, the direction of the foreign object in the field of view of key position A is Dt ₁ (181.016, 462.152, -100), and the direction of the field of view in key position B is Dt ₂ (420.525, 222.554, - 100).

It can be seen that the system clearly and accurately captures the state of foreign objects.

2) Result analysis

After calculation, it can be obtained that the two straight lines in the direction of the position of the foreign object in the visual field of the key position A and the key position B have no intersection. Therefore, the midpoint of the public perpendicular line of the two straight lines is taken as the calculated value of the foreign object position, and the final result is T(351,384,17) (only the integer part is reserved), after manual measurement, the error with the actual position is within ±5mm, which is considered to be a reliable result.

After obtaining Figure 9 and Figure 11, take the centroid pixel coordinates of the foreign object, and take out the RGB values of the pixels at the corresponding positions in Figure 8(b) and Figure 10(b) (245, 235, 0) and (249, 237, 1) , converted from the RGB color gamut to the HSV color gamut, the H values are 57.551 and 57.581 respectively, and the average value of 57.566 can determine that the object belongs to yellow.

After calculating the position of the foreign object, combined with Figure 9 and Figure 11, it can be calculated that the maximum size of the foreign object does not exceed 39mm, which is consistent with the actual situation.

To sum up, the foreign body position T(351,384,17)(mm), the HSV tone value is 57.566, and the maximum size is 39mm.

Claims (4)

1. a mobile platform foreign body detection method based on monocular camera, is characterized in that, comprises the following steps: Step 1. Platform selection: select at least 2 key positions in the process of carrying the mobile platform and the platform movement; Step 2. Obtain reference: collect environmental image information at all key positions as a reference image under the premise of no foreign matter; Step 3, image acquisition: when the mobile platform moves to any key position, the environmental image information of the current position is collected; Step 4. Image processing: compare the collected environmental image information with the reference image collected at the same key position to determine whether there is foreign matter; if there is foreign matter, go to step 5, otherwise go back to step 3, and set it in the user settings. The periodic coverage updates the reference image; Step 5. Multi-point collection: move the platform to the next key position to collect environmental image information; Step 6. Information processing: Integrate the image information collected from at least two key positions to calculate the position, size and color of the foreign object, specifically: According to the obtained image information, the direction of the foreign body relative to the key position point is obtained, that is, the equation of the line connecting the key position point and the centroid of the foreign body, and the common point of the equation of the line connecting the two key position points and the centroid of the foreign body is the foreign body. If there is no common point between the two straight lines, take the center point of the common vertical line of the two straight lines to get the spatial position point of the foreign object; if there are more than two key position points, calculate the intersection of all the straight lines or The center point of the common vertical line, calculate the average or median value, and get the position of the foreign object; at the same time, according to the calculated position of the foreign object and the pixel size of the outer contour of the object in the field of view, calculate the actual contour size of the object by the similar triangle method ; The color of the foreign object is obtained after the RGB information of the foreign object in the field of view is converted by the color gamut. 2. according to the mobile platform foreign body detection method based on monocular camera described in claim 1, it is characterized in that, described in step 1, selects to carry at least 2 key position points in the mobile platform and platform movement process, is specifically: First choose to carry a mobile platform, the platform moves cyclically, and there is a pause process or the minimum speed is lower than 0.5m/s during the movement; In the motion trajectory of the mobile platform, select a position point with a pause process. If there is no pause in the whole process, select a position point with a speed lower than 0.5m/s. 3. according to the mobile platform foreign body detection method based on monocular camera described in claim 1, it is characterized in that: in step 3, obtain the position information of the platform in real time by the controller or the position sensor of the platform, whenever the main controller detects When the platform moves to a key position, the environmental image information of the current position is collected. 4. The method for detecting foreign objects on a mobile platform based on a monocular camera according to claim 1, wherein the image information collected in the previous step is compared with the reference image collected at the same key position as described in step 4 , the specific steps are: 4.1 Perform color gamut conversion on two pictures collected by the platform at the same key position in different motion cycles, and convert the color pictures into grayscale pictures; 4.2 Use the absolute value difference algorithm absdiff to obtain the grayscale difference value of the two pictures; 4.3 Binarize and cut the difference value to obtain the original environmental shape; 4.4 Perform morphological denoising on the results obtained in step 4.3; 4.5 Perform the connected domain detection on the results obtained in step 4.4 to obtain the location distribution of the connected domain in the visual field and make a judgment. If there is a connected domain, it means that there is a foreign body.

Images