CN110032971B - Monocular camera-based mobile platform foreign matter detection method and detection system - Google Patents
Monocular camera-based mobile platform foreign matter detection method and detection system Download PDFInfo
- Publication number
- CN110032971B CN110032971B CN201910292288.7A CN201910292288A CN110032971B CN 110032971 B CN110032971 B CN 110032971B CN 201910292288 A CN201910292288 A CN 201910292288A CN 110032971 B CN110032971 B CN 110032971B
- Authority
- CN
- China
- Prior art keywords
- foreign matter
- platform
- mobile platform
- information
- monocular camera
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/56—Extraction of image or video features relating to colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Investigating Materials By The Use Of Optical Means Adapted For Particular Applications (AREA)
- Image Analysis (AREA)
- Length Measuring Devices By Optical Means (AREA)
Abstract
The invention discloses a monocular camera-based mobile platform foreign matter detection method and a monocular camera-based mobile platform foreign matter detection system. The system consists of an image acquisition module comprising a monocular camera and a main controller. Firstly, an image acquisition module is installed on a periodic low-speed moving platform (such as a mechanical arm), images are acquired for multiple times at different positions of the moving platform and processed, if the foreign matters are determined to exist, a main controller calculates information such as spatial positions, sizes and colors of the foreign matters, and the calculation results are sent to a superior control system. The invention obtains the information such as the position of the target by using the monocular camera by virtue of the mobile platform, can achieve the same effect of the monocular camera, and can be applied to the detection of whether workpieces fall or other low-speed foreign matters appear on industrial fields (such as an automatic assembly line and the like).
Description
Technical Field
The invention relates to the field of machine vision and image processing, in particular to a foreign matter state detection method and a foreign matter state detection system when foreign matters appear.
Background
The technique of acquiring and processing images using a computer is called a machine vision technique. Specifically, machine vision uses an optical non-contact sensing device to automatically receive and interpret images of a real scene to obtain information to control a machine or process. Image processing is a technique of processing image data using a computer data processing technique and acquiring effective information. Human vision is best at qualitatively interpreting complex, unstructured scenes, but machine vision is best at quantitatively measuring structured scenes by virtue of speed, accuracy and repeatability, for example, on a production line, where a machine vision system can detect hundreds or even thousands of elements per minute. With cameras and optical elements of appropriate resolution, machine vision systems can easily inspect detailed features of items that are too small to be seen by the human eye. The industrial application of machine vision can greatly reduce the cost and the labor intensity.
An image contains information which is digital information of a large number of pixel points measured by hundreds of thousands or even higher orders of magnitude; in a monochrome image, each pixel point only contains one type of gray information; in the color image, each pixel point contains brightness information of three colors of RGB. The process of processing such a huge amount of information is rather complex, clearly inefficient and impractical if all machine vision researchers are working from the lowest level of processing. Fortunately, in the field of machine vision, there is a powerful and open-source OpenCV computer vision library, openCV is a cross-platform computer vision library issued based on BSD license (open source), which can run on Linux, windows, android, and Mac OS operating systems. The method is light and efficient, is composed of a series of C functions and a small number of C + + classes, provides interfaces of languages such as Python, ruby, MATLAB and the like, and realizes a plurality of general algorithms in the aspects of image processing and computer vision. OpenCV provides a dedicated data structure for image processing and library functions for processing a large number of images, and provides great convenience for the research of machine vision.
Disclosure of Invention
The invention aims to provide a falling object detection method based on a monocular camera and a detection system thereof.
The method for realizing the technical solution of the invention comprises the following steps: a monocular camera-based mobile platform dropped object detection method comprises the following steps:
and step 6, information processing: and integrating the image information acquired from at least two key positions to calculate the position, size and color of the foreign matter.
The utility model provides a moving platform thing detecting system that drops based on monocular camera, includes image acquisition module, image processing module and information analysis module, wherein:
the image acquisition module is connected with the image processing module to acquire environment image information, the acquired light ray analog information is converted into digital RGB information through the photoelectric sensor, the information is transmitted to the image processing module, and meanwhile, the time and position labels are transmitted;
the image processing module is connected with the information analysis module, stores the environmental image information with time and position labels into a memory, firstly performs color gamut conversion on the current environmental image information, performs difference calculation on the current environmental image information and the environmental image information in an abnormal state, performs binarization cutting on a result, performs connected domain detection after denoising processing, and transmits the detected result to the information analysis module;
the information analysis module is connected with the superior control system and the image acquisition module and is used for processing and summarizing the information obtained by image processing; firstly, judging the existence of foreign matters according to the detection result of a connected domain, if the foreign matters exist, controlling an image acquisition module to continuously acquire environment image information at the next key position point, confirming whether the foreign matters exist again, if the foreign matters exist, summarizing the information acquired for multiple times to calculate the position, size and color information of the foreign matters, and reporting the position, size and color information to a superior control system for further processing; and if the foreign matters do not exist or the foreign matters do not exist after the secondary acquisition, controlling the image acquisition module to normally work.
Compared with the prior art, the invention has the remarkable advantages that: (1) The effect of multi-view camera depth detection can be realized only by using a single-view camera; (2) Redundant detection is allowed, and the existing result is continuously corrected, so that the precision is conveniently improved; (3) The movable platform can be specially configured, and can also be directly arranged on the existing periodic motion low-speed platform, such as welding and mechanical arm assembling, so that the adaptability is strong; (4) The image acquisition module allows for the expansion of additional degrees of freedom, enabling flexible multi-view detection.
Drawings
Fig. 1 is a structural diagram of a foreign matter detection system of a mobile platform based on a monocular camera according to the present invention.
Fig. 2 is a typical appearance structure of the system of the present invention.
Fig. 3 is an extended 1 degree of freedom mounting of the image acquisition module.
Fig. 4 is an extended 2 degree of freedom mounting of the image acquisition module.
FIG. 5 is a flow chart of the foreign matter detection method of the mobile platform based on the monocular camera according to the present invention.
Fig. 6 is a three-dimensional model of a robot mounted for operation on a low degree of freedom robot arm.
Fig. 7 shows a method for locating the spatial position of a foreign object.
FIG. 8 shows the visual field before and after the appearance of a foreign matter at the critical position A in example 1
FIG. 9 shows the results of detecting the presence of foreign matter at the key position A after the treatment in example 1
FIG. 10 shows the visual field before and after the appearance of a foreign matter at the key position B in example 1
FIG. 11 is a result of detecting the presence of foreign matter at the key position B after the treatment in example 1
Detailed Description
The invention is further described below with reference to the accompanying drawings.
With reference to fig. 5 and 6, the method for detecting the foreign matter on the moving platform based on the monocular camera according to the present invention includes the following steps:
the mobile platform can move to the same spatial position and angle for multiple times within limited time, so that the image acquisition module can conveniently observe the same monitored area from different angles;
the period of reference image update is set, for example, the reference image is overlaid with the newly acquired image without the presence of foreign matter at regular intervals or at intervals of several cycles. If the updating period is too short, the calculation load is increased, and if the updating period is too long, the judgment effect is influenced by error accumulation due to slight environmental change such as sunlight intensity change in a day;
setting the first moved key position point as a key position A, wherein the space coordinate is (xa, ya and za), and the axis angle of the image acquisition module is Da;
after image information acquired at the same key position in different periods is compared and subjected to denoising treatment, the existence of foreign objects in a visual field can be rapidly determined;
here, the second moved key position point is set as a key position B, the spatial coordinates are (xb, yb, zb), and the axis angle of the image capturing module is Db.
By performing morphological processing on the acquired image, it can be obtained that the direction vector of the position of the foreign object position coordinate point T (xt, yt, zt) in the field of view of the key position a relative to the point a is D1, that is, as shown in fig. 7, the direction vector of the position in the field of view of the key position B relative to the point B is D2, the equation of the spatial straight lines AT and BT can be obtained by combining the coordinates of the point a and the coordinates of the point B, and the coordinate of the point T can be obtained by calculating the intersection of the two spatial straight lines AT and BT. Considering that errors possibly exist in the actual positioning process, the calculated two spatial straight lines do not have an intersection point, and if the two spatial straight lines are found not to have the intersection point, the midpoint of the common perpendicular lines of the two different-surface straight lines is taken as a spatial position point of the foreign matter.
The information such as the approximate size and color of the object can be determined from the spatial position points of the object and the original images obtained in step 3 and step 5.
As shown in fig. 1, the foreign matter detection system of the mobile platform based on the monocular camera of the present invention comprises an image acquisition module, an image processing module and an information analysis module, wherein:
the image acquisition module is connected with the image processing module to acquire environment image information, the acquired light ray analog information is converted into digital RGB information through the photoelectric sensor, the information is transmitted to the image processing module, and meanwhile, the time and position labels are transmitted;
the image processing module is connected with the information analysis module and stores the environment image information with the time and position labels into the memory; firstly, performing color gamut conversion on current environment image information, performing difference calculation on the current environment image information and environment image information in a foreign object-free state, performing binarization cutting on a result, performing connected domain detection after denoising, and transmitting the detected result to an information analysis module;
the information analysis module is connected with the superior control system and the image acquisition module and is used for processing and summarizing the information obtained by image processing; firstly, judging the existence of foreign matters according to the detection result of a connected domain, if the foreign matters exist, controlling an image acquisition module to continuously acquire environment image information at the next key position point, confirming whether the foreign matters exist again, if the foreign matters exist, summarizing the information acquired for multiple times to calculate the position, size and color information of the foreign matters, and reporting the position, size and color information to a superior control system for further processing; and if the foreign matters do not exist or the foreign matters do not exist after the secondary acquisition, controlling the image acquisition module to normally work.
FIG. 2 is a typical external view of the system of the present invention, in which 1 is a lens, 2 is an industrial camera, 3 is a fixed base, 4 is a main controller, and 5,6 are communication interfaces; fig. 3 is an extended 1-degree-of-freedom mounting manner of an image acquisition module, fig. 4 is an extended 2-degree-of-freedom mounting manner of the image acquisition module, wherein 7 is a horizontal rotary table including a steering engine, 8 is a vertical rotary table including a steering engine, and the image acquisition module of the type shown in fig. 3 and 4 is used for extending the field of view of the image acquisition module.
As shown in fig. 2, the system is composed of an image acquisition module and a main controller, wherein 1 is a lens, 2 is an industrial camera, and 3 is a base for fixing 1 and 2; the image processing module and the information analysis module in the figure 1 are realized in the form of master controller software, 5 is a DB9 interface for serial communication, and 6 is an RJ45 interface for network communication. The camera carrying platform can provide a fixed or expanded freedom degree installation mode according to specific conditions, the higher the expanded freedom degree is, the larger the detectable range and angle are, the expanded 1 freedom degree typical structure is shown in figure 3, wherein 7 is a horizontal turntable, and the image acquisition module can have a wider horizontal view field by controlling a steering engine to rotate accurately; the typical structure of 2 degrees of freedom of extension is as shown in fig. 4, wherein 8 are vertical turntables, and can be precisely rotated by controlling a steering engine, so that the image acquisition module can have a wider vertical visual field.
When the system is used, a periodic low-speed motion platform is selected firstly, such as a welding, assembling and stacking mechanical arm commonly used in industrial production; if the working position does not have a mobile platform which meets the conditions, a periodic low-speed motion platform such as a low-freedom mechanical arm can be specially configured for the system. After the working platform is selected, the image acquisition module is installed at a proper position, such as the tail end, of the working platform, the image acquisition module is connected with the main controller through the data line, and the main controller and the image acquisition module can be simultaneously carried on the working platform or installed at a fixed position. A communication channel needs to be arranged between the main controller and the controller of the working platform, so that the position information of the working platform can be acquired at any time, the data capacity for acquiring the position information of the working platform is not large, the working platform can work in a simplex mode, but the requirement on real-time performance is high, too large delay cannot exist, and an RS485 serial communication protocol is recommended to be used. A communication channel needs to be arranged between the main controller and the superior control system, and information is reported in time when foreign matters are found.
The present invention will be further described with reference to the following specific examples.
Example 1
Experiments verify that the performance of the mobile platform foreign matter detection system based on the monocular camera and the detection method thereof provided by the invention is as follows:
1) Conditions of the experiment
Installing a vision acquisition module with 1 degree of freedom expansion on a 3-degree-of-freedom mechanical arm, determining two key positions A (200, 0, 100) and B (0, 200, 100) (the unit is mm, the same below), firstly acquiring environment information, acquiring images acquired by the two key positions such as a graph (8 (a) and a graph (10 (a), after placing foreign matters, acquiring images acquired by the two key positions such as a graph (8 (B) and a graph (10 (B), and obtaining a graph (9) and a graph (11) after processing, wherein the image acquisition module is calibrated in advance to obtain the direction Dt of the foreign matters in the view field of the key position A 1 (181.016, 462.152, -100) with Dt in the critical position B view direction 2 (420.525,222.554,-100).
It can be seen that the system clearly and accurately captures the state of the foreign body.
2) Analysis of results
The two straight lines in the directions of the positions where the foreign matters are located in the visual fields of the key positions A and the key positions B are obtained through calculation, therefore, the midpoint of the common perpendicular line of the two straight lines is taken as the calculated value of the position of the foreign matters, the final result is T (351, 384 and 17) (only an integral part is reserved), and the error between the actual position and the position is within +/-5 mm through manual measurement, so that the method is considered to be a credible result.
After obtaining fig. 9 and fig. 11, the centroid pixel coordinates of the alien material are taken, RGB values (245, 235, 0) and (249, 237, 1) of the pixels are taken out at corresponding positions of fig. 8 (b) and fig. 10 (b), the RGB color gamut is converted into an HSV color gamut, H values are 57.551 and 57.581 respectively, and the object can be determined to belong to yellow by the average value of 57.566.
After the foreign body position is calculated, the maximum size of the foreign body can be calculated to be not more than 39mm by combining fig. 9 and fig. 11, which is consistent with the actual situation.
In summary, a foreign matter position T (351, 384, 17) (mm) occurs, HSV hue value 57.566, maximum dimension 39mm.
Claims (4)
1. A foreign matter detection method of a mobile platform based on a monocular camera is characterized by comprising the following steps:
step 1, platform selection: selecting a carrying mobile platform and at least 2 key position points in the motion process of the platform;
step 2, obtaining reference: acquiring environmental image information at all key positions as reference images on the premise of no foreign object;
step 3, image acquisition: when the mobile platform moves to any key position, acquiring environmental image information of the current position;
step 4, image processing: comparing the collected environmental image information with a reference image collected at the same key position, and determining whether foreign objects appear or not; if foreign matters appear, executing the step 5, otherwise returning to the step 3, and updating the reference image in a period set by a user;
step 5, multipoint acquisition: moving the platform to the next key position to acquire environment image information;
and step 6, information processing: synthesize the image information that two at least key positions were gathered, calculate the position, size and the colour of foreign matter, specifically do:
according to the obtained image information, obtaining the direction of the foreign matter relative to the key position point, namely an equation of a connecting line of the key position point and the centroid of the foreign matter, wherein the common point of the equations of the connecting line of the two key position points and the centroid of the foreign matter is the spatial position point of the foreign matter, and if the two straight lines have no common point, the central point of the common perpendicular line of the two straight lines is taken to obtain the spatial position point of the foreign matter; if more than two key position points exist, calculating the intersection point of every two straight lines or the central point of a common perpendicular line, and calculating the average value or the median value to obtain the position of the foreign matter; meanwhile, according to the calculated position of the foreign matter and the outer contour pixel size of the object in the visual field, calculating the actual contour size of the object by a similar triangle method; the color of the foreign matter is obtained by the color gamut conversion of the RGB information of the pixel of the foreign matter in the visual field.
2. The monocular camera-based foreign object detection method for a mobile platform according to claim 1, wherein the step 1 of selecting at least 2 key location points for carrying the mobile platform and the platform in the motion process specifically comprises:
firstly, selecting a platform for carrying a mobile platform, wherein the platform periodically and circularly moves, and a pause process exists in the moving process or the lowest speed is lower than 0.5m/s;
and selecting a position point with a stopping process in the motion trail of the mobile platform, and if the whole motion trail does not stop, selecting a position point with the speed lower than 0.5 m/s.
3. The monocular camera-based mobile platform foreign matter detection method according to claim 1, wherein: and 3, acquiring the position information of the platform in real time through a controller or a position sensor of the platform, and acquiring the environmental image information of the current position when the main controller detects that the platform moves to a key position point.
4. The monocular camera-based mobile platform foreign matter detection method according to claim 1, wherein the step 4 of comparing the image information collected in the previous step with the reference image collected at the same key position comprises the specific steps of:
4.1, performing color gamut conversion on two pictures acquired by the platform at the same key position in different motion periods, and converting the color pictures into gray pictures;
4.2, solving the gray difference value of the two pictures by using an absolute value difference algorithm absdiff;
4.3, performing binarization cutting on the difference value to obtain an original environment form;
4.4, performing morphological denoising treatment on the result obtained in the step 4.3;
and 4.5, detecting the connected domain according to the result obtained in the step 4.4, obtaining the position distribution of the connected domain in the visual field, judging, and if the connected domain exists, indicating that foreign matters exist.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910292288.7A CN110032971B (en) | 2019-04-12 | 2019-04-12 | Monocular camera-based mobile platform foreign matter detection method and detection system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910292288.7A CN110032971B (en) | 2019-04-12 | 2019-04-12 | Monocular camera-based mobile platform foreign matter detection method and detection system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110032971A CN110032971A (en) | 2019-07-19 |
| CN110032971B true CN110032971B (en) | 2022-10-14 |
Family
ID=67238202
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910292288.7A Active CN110032971B (en) | 2019-04-12 | 2019-04-12 | Monocular camera-based mobile platform foreign matter detection method and detection system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110032971B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110910369A (en) * | 2019-11-19 | 2020-03-24 | 珠海格力电器股份有限公司 | Production line supervision method and device and storage medium |
| CN110954967A (en) * | 2019-12-11 | 2020-04-03 | 江西莱利电气有限公司 | Device and method for detecting foreign matters in fan |
| CN113012090B (en) * | 2019-12-20 | 2024-03-01 | 中国科学院沈阳计算技术研究所有限公司 | Multi-workpiece quality detection method and device based on movable camera |
| CN111626204B (en) * | 2020-05-27 | 2022-01-11 | 汪海洋 | Railway foreign matter invasion monitoring method and system |
| CN112785587B (en) * | 2021-02-04 | 2024-05-31 | 上海电气集团股份有限公司 | Foreign matter detection method, system, equipment and medium in stacking production process |
| CN114147704B (en) * | 2021-11-18 | 2023-09-22 | 南京师范大学 | Mechanical arm accurate positioning and grabbing method based on depth vision and incremental closed loop |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103258188A (en) * | 2013-04-19 | 2013-08-21 | 上海应用技术学院 | Moving target object detection tracking method based on cross-platform computer vision library |
| CN104590319A (en) * | 2014-06-11 | 2015-05-06 | 北京交通大学 | Device for foreign body invasion detection and method for foreign body invasion detection |
-
2019
- 2019-04-12 CN CN201910292288.7A patent/CN110032971B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103258188A (en) * | 2013-04-19 | 2013-08-21 | 上海应用技术学院 | Moving target object detection tracking method based on cross-platform computer vision library |
| CN104590319A (en) * | 2014-06-11 | 2015-05-06 | 北京交通大学 | Device for foreign body invasion detection and method for foreign body invasion detection |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110032971A (en) | 2019-07-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110032971B (en) | Monocular camera-based mobile platform foreign matter detection method and detection system | |
| CN108592788B (en) | 3D intelligent camera system facing spraying production line and workpiece online measurement method | |
| CN102922521B (en) | A kind of mechanical arm system based on stereoscopic vision servo and real-time calibration method thereof | |
| CN109297413B (en) | Visual measurement method for large-scale cylinder structure | |
| US6088469A (en) | Identifying apparatus and method, position detecting apparatus and method, robot apparatus and color extracting apparatus | |
| CN104385282B (en) | Visual intelligent numerical control system and visual measuring method thereof | |
| CN111470327B (en) | Visual stacking equipment and visual information processing method thereof | |
| EP3290860B1 (en) | Method and system for determining the presence or absence of a part of an assembly within a work cell | |
| CN104842361A (en) | Robotic system with 3d box location functionality | |
| CN114241298A (en) | Tower crane environment target detection method and system based on laser radar and image fusion | |
| CN114638909A (en) | Substation semantic map construction method based on laser SLAM and visual fusion | |
| CN112334760A (en) | Method and device for locating points on complex surfaces in space | |
| CN110378937B (en) | Kinect camera-based industrial mechanical arm man-machine safety distance detection method | |
| CN112518748A (en) | Automatic grabbing method and system of vision mechanical arm for moving object | |
| CN101943969B (en) | P0sition measurement apparatus and p0sition measuring method | |
| CN111856436A (en) | Combined calibration device and calibration method for multi-line laser radar and infrared camera | |
| CN203772217U (en) | Non-contact type flexible online dimension measuring device | |
| CN112789570A (en) | Method and robot system for inputting work area | |
| CN114140534A (en) | Combined calibration method for laser radar and camera | |
| CN109579698A (en) | A kind of intelligence cargo detection system and its detection method | |
| Chemweno et al. | Innovative safety zoning for collaborative robots utilizing Kinect and LiDAR sensory approaches | |
| CN110160455B (en) | Clearance surface difference detection system | |
| CN106625669A (en) | All-directional moving visual robot system | |
| CN111015650A (en) | Industrial robot intelligent vision system and method for determining target position at multiple points | |
| CN114314345A (en) | Intelligent sensing system of bridge crane and working method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20230111 Address after: No. 23, Huashen Avenue, Yuhuatai District, Nanjing City, Jiangsu Province, 210012 Patentee after: Nanjing mowen Technology Co.,Ltd. Address before: 210094 No. 200, Xiaolingwei, Jiangsu, Nanjing Patentee before: NANJING University OF SCIENCE AND TECHNOLOGY |