CN113379743A - Conveyor abnormity detection method and system based on computer vision - Google Patents
Conveyor abnormity detection method and system based on computer vision Download PDFInfo
- Publication number
- CN113379743A CN113379743A CN202110921779.0A CN202110921779A CN113379743A CN 113379743 A CN113379743 A CN 113379743A CN 202110921779 A CN202110921779 A CN 202110921779A CN 113379743 A CN113379743 A CN 113379743A
- Authority
- CN
- China
- Prior art keywords
- conveyor belt
- area
- offset
- conveyor
- preset
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 24
- 230000002159 abnormal effect Effects 0.000 claims abstract description 38
- 238000000034 method Methods 0.000 claims abstract description 38
- 239000013077 target material Substances 0.000 claims abstract description 26
- 230000008859 change Effects 0.000 claims abstract description 19
- 239000000463 material Substances 0.000 claims description 26
- 239000013598 vector Substances 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 6
- 238000013507 mapping Methods 0.000 claims description 5
- 230000002238 attenuated effect Effects 0.000 claims description 3
- 230000005856 abnormality Effects 0.000 description 9
- 230000006870 function Effects 0.000 description 7
- 238000012545 processing Methods 0.000 description 6
- 238000003708 edge detection Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 238000005070 sampling Methods 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 239000000126 substance Substances 0.000 description 2
- 241000669618 Nothes Species 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000005674 electromagnetic induction Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 230000009191 jumping Effects 0.000 description 1
- 238000007477 logistic regression Methods 0.000 description 1
- 238000013178 mathematical model Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/40—Image enhancement or restoration by the use of histogram techniques
-
- G06T5/70—
-
- G06T5/90—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
- G06T7/75—Determining position or orientation of objects or cameras using feature-based methods involving models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20172—Image enhancement details
- G06T2207/20192—Edge enhancement; Edge preservation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30108—Industrial image inspection
- G06T2207/30164—Workpiece; Machine component
Abstract
The invention relates to the technical field of computer vision, in particular to a conveyor abnormity detection method and system based on computer vision. The method comprises the following steps: collecting a real-time image of a conveyor belt; detecting the edge of a conveyor belt of a real-time image, and acquiring an enclosing frame of a target material in the area of the conveyor belt; obtaining a jitter area of the conveyor belt; when the shaking area is the whole area of the conveyor belt, carrying out internal anomaly detection on the conveyor where the conveyor belt is located; when the shaking area is a local area of the conveyor belt, acquiring an abnormal area with a falling risk; generating three-dimensional point cloud data for the target material in the abnormal area, and taking the depth change of a preset two-dimensional plane in the three-dimensional point cloud data as an offset; taking the rotation angle of a preset two-dimensional plane as an offset angle; and when the offset amount and the corresponding offset angle are both larger than the corresponding offset threshold value, locating an abnormal point of the conveyor belt. The embodiment of the invention can accurately and quickly position the position of the conveyor where the conveyor is abnormal.
Description
Technical Field
The invention relates to the technical field of computer vision, in particular to a conveyor abnormity detection method and system based on computer vision.
Background
In the metallurgical industry, many kinds and quantities of materials are needed, and in order to improve the production efficiency, the transportation of bulk materials is mostly realized by a belt conveyor. In the transportation process, due to the problem of a conveyor belt of the conveyor, the risk of jumping, deviation and falling of materials is inevitable. The belt conveyor has the problems in the transportation process, and the damage to the belt conveyor can be caused, and in a serious case, the production accident can be caused.
In practice, the inventors found that the above prior art has the following disadvantages:
at present, the fault detection of the conveying belt in China mainly comprises methods such as manual detection, electromagnetic induction, X-ray nondestructive inspection and the like, but the device has the characteristics that: under the static condition of conveyer belt, carry out the segmentation to the conveyer belt by the manual work and detect and find out the fault location, not only consume a large amount of manpower resources, detection efficiency is lower moreover, appears lou examining the situation easily. The manual detection cannot realize real-time monitoring, only can be used for regularly checking and observing, and has low real-time performance and reliability.
Disclosure of Invention
In order to solve the above technical problems, an object of the present invention is to provide a method and a system for detecting conveyor anomalies based on computer vision, wherein the adopted technical scheme is as follows:
in a first aspect, an embodiment of the present invention provides a method for detecting conveyor abnormality based on computer vision, which includes the following steps:
acquiring a real-time image of a conveyor belt, wherein the real-time image comprises the conveyor belt and materials transported by the conveyor belt;
detecting the edge of the conveyor belt of the real-time image, and acquiring an enclosing frame of the target material in the area of the conveyor belt;
obtaining a jitter area of the conveyor belt through jitter variation of the conveyor belt edge; when the jitter area is the whole area of the conveyor belt, carrying out internal anomaly detection on the conveyor where the conveyor belt is located;
when the shaking area is a local area of the conveyor belt, judging whether the material has a falling risk or not for each shaking area according to the surrounding frame, and acquiring an abnormal area with the falling risk;
generating three-dimensional point cloud data for the target material in the abnormal area, and taking the depth change of a preset two-dimensional plane in the three-dimensional point cloud data as an offset; taking the rotation angle of the preset two-dimensional plane as an offset angle;
and when the offset and the corresponding offset angle are both larger than the corresponding offset threshold, positioning the abnormal point of the conveyor belt according to the offset direction of the offset angle.
Preferably, the method for acquiring the jitter region includes:
acquiring a conveyor belt edge image, calculating the change of the depth information of the conveyor belt edge in continuous multiframe conveyor belt edge images to obtain the jitter degree, traversing the conveyor belt, and acquiring an area corresponding to the jitter degree larger than a jitter threshold value as the jitter area.
Preferably, the method for determining the existence of the falling risk is as follows:
acquiring a first image comprising the surrounding frame and the edge of the conveyor belt, and judging whether the material has a falling risk or not by calculating the vertical distance between the corner point of the surrounding frame in the first image and the edge of the conveyor belt.
Preferably, the method for acquiring the abnormal region includes:
when the materials are in risk of falling, the shaking area with the risk of falling is used as the abnormal area.
Preferably, the method for acquiring the preset two-dimensional plane includes:
extracting initial three-dimensional point cloud data in an initial frame image with offset, giving an initial thermal force value to a pixel point with the minimum depth value in the initial three-dimensional point cloud data, giving the initial thermal force value to pixels in the surrounding frame in the same depth plane, and communicating the pixel points corresponding to the initial thermal force value to form the preset two-dimensional plane.
Preferably, the method for acquiring the preset two-dimensional plane further includes:
and when only one pixel point with the minimum depth value is available, expanding a two-dimensional plane as the preset two-dimensional plane by taking the pixel point as a plane central point.
Preferably, the method for acquiring the offset comprises:
and enabling the initial heat force value to be attenuated according to the change of the frame number, and tracking the position change of the preset two-dimensional plane according to the attenuation of the initial heat force value to obtain the depth value difference of the preset two-dimensional plane in the real-time image of the adjacent frame as the offset.
Preferably, the method for acquiring the offset angle includes:
and acquiring normal vectors of the preset two-dimensional plane corresponding to the offset, mapping the normal vectors in the same coordinate system, and obtaining an included angle between the normal vectors as the offset angle.
In a second aspect, another embodiment of the present invention provides a computer vision-based conveyor abnormality detection system, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the above-mentioned computer vision-based conveyor abnormality detection method when executing the computer program.
The embodiment of the invention at least has the following beneficial effects:
the offset and the offset angle of the target material are calculated in the abnormal area according to the characteristics of the three-dimensional point cloud data of the target material, so that the abnormal position of the conveyor where the conveyor belt is located is positioned, accuracy and rapidness are achieved, and reliability is high.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions and advantages of the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a flow chart illustrating steps of a method for detecting conveyor anomalies based on computer vision according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of the relationship between the enclosure frame and the edge of the conveyor belt for the target material according to one embodiment of the present invention;
fig. 3 is a schematic diagram of a target material on a conveyor belt in accordance with an embodiment of the present invention.
Detailed Description
To further illustrate the technical means and effects of the present invention adopted to achieve the predetermined objects, the following detailed description will be given of a method and a system for detecting conveyor abnormality based on computer vision according to the present invention, with reference to the accompanying drawings and preferred embodiments, and the detailed description thereof will be made below. In the following description, different "one embodiment" or "another embodiment" refers to not necessarily the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
The following describes a specific scheme of a conveyor anomaly detection method and system based on computer vision in detail with reference to the accompanying drawings.
Referring to fig. 1, a flow chart illustrating steps of a method for detecting abnormality of a conveyor based on computer vision according to an embodiment of the present invention is shown, the method including the following steps:
and S001, acquiring a real-time image of the conveyor belt, wherein the real-time image comprises the conveyor belt and the materials transported by the conveyor belt.
And placing an orbit RGB-D camera above the starting end of the conveyor belt, wherein the camera looks down to shoot the material and carries a position sensor to collect the image of the conveyor belt. When the materials on the conveying belt are detected by a frame difference method, the track camera starts to run to the tail end of the conveying belt from the starting end at the same speed as the running speed of the conveying belt, real-time image acquisition is carried out on the conveying belt and the materials conveyed by the conveying belt, and a real-time image sequence is obtained
。
As an example, the conveyor belt in the embodiment of the invention keeps running at a constant speed
The belt length of the conveyor belt is
. The shooting frequency of the camera is 25 frames/s, and the sampling frequency is set to be 5 frames/s.
And S002, detecting the edge of the conveyor belt of the real-time image, and acquiring a surrounding frame of the target material in the area of the conveyor belt.
The method comprises the following specific steps:
1) the acquired image sequence is processed.
In order to reduce the computational complexity among images, the collected image sequence M is subjected to graying processing, the grayed images are subjected to histogram equalization processing, the contrast among the images is improved, and further, denoising is performed by Gaussian filtering. Finally, a series of image processing is carried out to obtain a processed gray level image sequence
。
2) The conveyor belt edge of the real-time image is detected.
The processed gray image sequence
And carrying out edge detection to obtain the edge information of the conveyor belt in the image.
As an example, in the embodiment of the present invention, edge detection is performed by using a Canny operator, and binarization processing of an image after edge detection is performed to determine an edge point, specifically, the processed image is used
And mapping the curve to Hough space, and fitting a Hough straight line to obtain an edge line of the conveyor belt. The Canny operator ignores the internal details of the edge line, and the edge detection precision is higher.
Preferably, the Canny operator is used for edge detection, and meanwhile, local maximum and minimum value straight lines of the linear coordinates are taken to eliminate the influence of texture noise in the conveyor belt.
3) Taking the area between two edge lines of the conveyor belt as the conveyor belt area, and carrying out gray image sequence
And carrying out target detection on the conveying belt area to obtain an enclosing frame of the target material.
As an example, in the embodiment of the present invention, the target detection is performed by using the YOLOV3 algorithm with the structure of Encoder-FC, and in other embodiments, target detection algorithms that can achieve the same effect, such as fast RCNN, SSD, RetinaNet, and the like, may also be used.
The method comprises the following specific steps:
a. for gray level image sequence
The transport band region in (1) is image-encoded and then decoded. The input of the network is a gray image sequence, and the output is the central point of the surrounding frame
And width w, height h and rotation angle of the bounding box
。
b. Convolution and average pooling operations are carried out through a DNN deep neural network, in the process of down-sampling the image, spatial domain features in the image are extracted, and the output of a target encoder is an extracted feature map.
As an example, the DNN deep neural network in the embodiment of the invention adopts DarkNet-53.
c. The foreground and background of the feature map are classified by logistic regression, and the exact bounding box is obtained by Bbox regression.
d. The loss function is the superposition of classification loss function and regression loss function.
As an example, in the embodiment of the present invention, the classification loss function is a cross entropy loss function, and the regression loss function is a mean square error loss function.
The network output result is the characteristic vector of the center point of the surrounding frame of the target material, the width and the height of the surrounding frame and the rotation angle of the surrounding frame
，
Represents the center point of the bounding box, h represents the height of the bounding box,
the width of the bounding box is indicated and the coordinates of the four corner points of the bounding box are determined.
Step S003, obtaining a shaking area of the conveyor belt through the shaking change of the edge of the conveyor belt; when the jitter area is the whole area of the conveyor belt, the internal abnormality detection is performed on the conveyor where the conveyor belt is located.
The method comprises the following specific steps:
1) and acquiring a transmission belt edge image, acquiring the jitter degree by calculating the change of the depth information of the transmission belt edge in the continuous multi-frame transmission belt edge image, traversing the transmission belt, and acquiring a region corresponding to the jitter degree which is greater than the jitter threshold value as a jitter region.
In the edge images of the adjacent frame conveyor belts, the larger the depth change is, the stronger the edge jitter is, and the greater the jitter degree is, so that the jitter degree of the conveyor belt edge is judged through the depth value change of the edge pixel points of the adjacent three-frame conveyor belts
。
Fitting a relation between the depth value and the jitter degree through a mathematical model, wherein the relation is as follows:
wherein the content of the first and second substances,
is shown as
In the frame conveyer edge image
The depth value of each edge pixel point is determined,
is shown as
Frame image No
The depth value of each edge pixel point is determined,
is shown as
Frame image No
And N represents the number of the edge pixel points.
And traversing all the conveyor belt edge images of the conveyor belt, and calculating an area with the jitter degree larger than a jitter threshold value to obtain a jitter area.
When the shaking degree is smaller than the shaking threshold value, the normal shaking condition in the transportation process is realized, and the conveyor has no abnormal fault which causes the material to fall.
As an example, the empirical jitter threshold is obtained according to the output power of the conveyor and the position of the conveyor belt, and the value of the jitter threshold in the embodiment of the present invention is 5.
2) When the entire conveyor belt is in a shaking area, an abnormal fault may occur inside the conveyor, and internal abnormality detection needs to be performed on the conveyor where the conveyor belt is located.
And step S004, when the shaking area is a local area of the conveyor belt, judging whether the materials fall off risk exists in each shaking area according to the surrounding frame, and acquiring an abnormal area with the falling risk.
The method comprises the following specific steps:
1) the method comprises the steps of obtaining a first image comprising a surrounding frame and the edge of a conveyor belt, and judging whether the material has a falling risk or not by calculating the vertical distance between the corner point of the surrounding frame in continuous multi-frame first images and the edge of the conveyor belt.
As shown in fig. 2, when the target material is shifted, the corner point of the corresponding bounding box 201 is shifted, the bounding box corner point 202 close to the edge of the conveyor belt is selected, the vertical distance between the bounding box corner point and the edge 203 of the conveyor belt is calculated, the sum of the changes of the vertical distances of the first images of three continuous frames is compared with the width of the conveyor belt, and when the sum of the changes is greater than or equal to 1/4 of the width of the conveyor belt, the target material has a greater shifting tendency, and the target material is at risk of falling.
The specific calculation formula is as follows:
wherein the content of the first and second substances,
is shown as
The vertical distance of the first image of the frame,
is shown as
The vertical distance of the first image of the frame,
is shown as
The vertical distance of the first image of the frame,
the distance between the two edges of the conveyor belt, i.e. the width of the conveyor belt, is indicated.
2) When the material has the risk of falling, regard the shake area that has the risk of falling as the abnormal area.
For the shaking area obtained in step S003, when there is no risk of dropping in the shaking area, the shaking degree of the shaking area does not cause a large offset influence on the material on the conveyor belt; when the shaking area has a risk of falling, the shaking degree of the shaking area may cause a large offset effect on the material on the conveyor belt, and thus the material falls. Therefore, the shaking area where the risk of falling is present is taken as the abnormal area.
Step S005, generating three-dimensional point cloud data for the target material in the abnormal area, and taking the depth change of a preset two-dimensional plane in the three-dimensional point cloud data as an offset; and taking the rotation angle of the preset two-dimensional plane as an offset angle.
The method comprises the following specific steps:
1) and changing the sampling frequency to obtain more image data of the abnormal area.
Specifically, after the abnormal region is obtained, the sampling frequency of the camera is increased, image data of more abnormal regions is obtained, and the image data is subjected to the same image processing as that in step S002, so that the target image is obtained.
As an example, the sampling frequency of the camera of the abnormal area in the embodiment of the invention is increased from 5 frames/s to 10 frames/s.
2) And generating three-dimensional point cloud data for the target materials in the abnormal area.
And carrying out three-dimensional point cloud mapping on the depth information of the target image pixel points to generate three-dimensional point cloud data of the target material.
3) And taking the depth change of a preset two-dimensional plane in the three-dimensional point cloud data as an offset.
The method comprises the following specific steps:
a. and acquiring a preset two-dimensional plane.
Referring to fig. 3, the initial three-dimensional point cloud data in the initial frame target image with the offset is extracted, the initial thermal force value is given to the pixel point with the minimum depth value in the initial three-dimensional point cloud data, the initial thermal force value is given to the pixel point in the bounding box of the same depth plane, and the pixel points corresponding to the initial thermal force value are connected to form the preset two-dimensional plane
。
And when only one pixel point with the minimum depth value is provided, expanding a two-dimensional plane as a preset two-dimensional plane by taking the pixel point as a plane central point.
As an example, the initial heat value in the embodiment of the present invention is 1.0.
b. An offset is calculated.
And attenuating the initial thermal value according to the change of the frame number, and tracking the position change of the preset two-dimensional plane according to the attenuation of the initial thermal value to obtain the depth value difference of the preset two-dimensional plane in the real-time image of the adjacent frame as the offset.
Setting the thermal value attenuation of the corresponding pixel point between the adjacent frames to be 0.1, namely setting the thermal value of a preset two-dimensional plane in the next frame target image corresponding to the initial frame to be 0.9 as a second preset two-dimensional plane
Second predetermined two-dimensional plane
And a predetermined two-dimensional plane
Being the same plane in different target images. Making the second predetermined two-dimensional plane
And a predetermined two-dimensional plane
The depth values of the adjacent target images are differentiated to obtain the offset between the adjacent target images. And then continuously iterating, and continuously attenuating the thermal value of the pixel point until the thermal value is attenuated to 0 or the offset is greater than the length offset threshold value, and finally obtaining the dynamic change of the preset two-dimensional plane between continuous frames.
It should be noted that the second predetermined two-dimensional plane
The depth value of the image is the minimum depth value of the pixel point in the current plane.
4) And taking the rotation angle of the preset two-dimensional plane as an offset angle.
And acquiring normal vectors of a preset two-dimensional plane corresponding to the offset, mapping the normal vectors in the same coordinate system, and obtaining an included angle between the normal vectors as an offset angle.
In acquiring a preset two-dimensional plane
And a second predetermined two-dimensional plane
Meanwhile, acquiring a corresponding normal vector, and calculating an included angle between the two normal vectors as a preset two-dimensional plane
And a second predetermined two-dimensional plane
The offset angle therebetween.
And S006, when the offset amount and the corresponding offset angle are both larger than the corresponding offset threshold value, positioning the abnormal point of the conveyor belt according to the offset direction of the offset angle.
And when the offset of the target material is greater than the length offset threshold and the offset angle is greater than the angle offset threshold, locating an abnormal point when the current position is abnormal.
Specifically, the offset direction of the offset angle is obtained, and when the offset direction of the target material is leftward offset, the abnormal point is located on the right side of the initial position of the target material; similarly, when the deviation direction of the target material is rightward deviation, the abnormal point is on the left side of the initial position of the target material, and the current position of the camera is recorded, namely the abnormal point of the conveyor where the conveyor belt is located.
As an example, an empirical length deviation threshold and an angle deviation threshold are obtained according to the influence of the length of the conveyor belt on the conveyor belt and the influence of the conveyor belt abnormal point on the transported material of the conveyor, in an embodiment of the present invention, the length deviation threshold is 3, and the angle deviation threshold is 10 °.
In summary, the embodiment of the invention collects the real-time image of the conveyor belt, wherein the real-time image comprises the conveyor belt and the materials transported by the conveyor belt; detecting the edge of a conveyor belt of a real-time image, and acquiring an enclosing frame of a target material in the area of the conveyor belt; obtaining a jitter area of the conveyor belt through jitter variation of the edge of the conveyor belt; when the shaking area is the whole area of the conveyor belt, carrying out internal anomaly detection on the conveyor where the conveyor belt is located; when the shaking areas are local areas of the conveyor belt, judging whether the materials have falling risks or not according to the surrounding frame for each shaking area, and acquiring abnormal areas with the falling risks; generating three-dimensional point cloud data for the target material in the abnormal area, and taking the depth change of a preset two-dimensional plane in the three-dimensional point cloud data as an offset; taking the rotation angle of a preset two-dimensional plane as an offset angle; and when the offset and the corresponding offset angle are both larger than the corresponding offset threshold, positioning the abnormal point of the conveyor belt according to the offset direction of the offset angle. The embodiment of the invention can accurately and quickly position the abnormal position of the conveyor through the offset and the offset angle of the material, and has high reliability.
The embodiment of the invention also provides a conveyor abnormality detection system based on computer vision, which comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein the processor realizes the steps of the method when executing the computer program. Since the conveyor abnormality detection method based on computer vision is described in detail above, it is not described in detail.
It should be noted that: the precedence order of the above embodiments of the present invention is only for description, and does not represent the merits of the embodiments. And specific embodiments thereof have been described above. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (9)
1. A conveyor abnormity detection method based on computer vision is characterized by comprising the following steps:
acquiring a real-time image of a conveyor belt, wherein the real-time image comprises the conveyor belt and materials transported by the conveyor belt;
detecting the edge of the conveyor belt of the real-time image, and acquiring an enclosing frame of the target material in the area of the conveyor belt;
obtaining a jitter area of the conveyor belt through jitter variation of the conveyor belt edge; when the jitter area is the whole area of the conveyor belt, carrying out internal anomaly detection on the conveyor where the conveyor belt is located;
when the shaking area is a local area of the conveyor belt, judging whether the material has a falling risk or not for each shaking area according to the surrounding frame, and acquiring an abnormal area with the falling risk;
generating three-dimensional point cloud data for the target material in the abnormal area, and taking the depth change of a preset two-dimensional plane in the three-dimensional point cloud data as an offset; taking the rotation angle of the preset two-dimensional plane as an offset angle;
and when the offset and the corresponding offset angle are both larger than the corresponding offset threshold, positioning the abnormal point of the conveyor belt according to the offset direction of the offset angle.
2. The method according to claim 1, wherein the jitter region is obtained by:
acquiring a conveyor belt edge image, calculating the change of the depth information of the conveyor belt edge in continuous multiframe conveyor belt edge images to obtain the jitter degree, traversing the conveyor belt, and acquiring an area corresponding to the jitter degree larger than a jitter threshold value as the jitter area.
3. The method of claim 1, wherein the determination that there is a risk of dropping is performed by:
acquiring a first image comprising the surrounding frame and the edge of the conveyor belt, and judging whether the material has a falling risk or not by calculating the vertical distance between the corner point of the surrounding frame in the first image and the edge of the conveyor belt.
4. The method according to claim 1, wherein the abnormal region is obtained by:
when the materials are in risk of falling, the shaking area with the risk of falling is used as the abnormal area.
5. The method according to claim 1, wherein the preset two-dimensional plane is obtained by:
extracting initial three-dimensional point cloud data in an initial frame image with offset, giving an initial thermal force value to a pixel point with the minimum depth value in the initial three-dimensional point cloud data, giving the initial thermal force value to pixels in the surrounding frame in the same depth plane, and communicating the pixel points corresponding to the initial thermal force value to form the preset two-dimensional plane.
6. The method according to claim 5, wherein the method for acquiring the preset two-dimensional plane further comprises:
and when only one pixel point with the minimum depth value is available, expanding a two-dimensional plane as the preset two-dimensional plane by taking the pixel point as a plane central point.
7. The method of claim 5, wherein the offset is obtained by:
and enabling the initial heat force value to be attenuated according to the change of the frame number, and tracking the position change of the preset two-dimensional plane according to the attenuation of the initial heat force value to obtain the depth value difference of the preset two-dimensional plane in the real-time image of the adjacent frame as the offset.
8. The method according to claim 1, wherein the offset angle is obtained by:
and acquiring normal vectors of the preset two-dimensional plane corresponding to the offset, mapping the normal vectors in the same coordinate system, and obtaining an included angle between the normal vectors as the offset angle.
9. Computer vision based conveyor anomaly detection system comprising a memory, a processor and a computer program stored in said memory and executable on said processor, characterized in that said processor implements the steps of the method according to any of claims 1-8 when executing said computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110921779.0A CN113379743B (en) | 2021-08-12 | 2021-08-12 | Conveyor abnormity detection method and system based on computer vision |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110921779.0A CN113379743B (en) | 2021-08-12 | 2021-08-12 | Conveyor abnormity detection method and system based on computer vision |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113379743A true CN113379743A (en) | 2021-09-10 |
| CN113379743B CN113379743B (en) | 2021-10-29 |
Family
ID=77576834
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110921779.0A Active CN113379743B (en) | 2021-08-12 | 2021-08-12 | Conveyor abnormity detection method and system based on computer vision |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113379743B (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113763376A (en) * | 2021-09-17 | 2021-12-07 | 深圳市赛为智能股份有限公司 | Conveyor belt deviation detection method and device, computer equipment and storage medium |
| CN113989251A (en) * | 2021-11-02 | 2022-01-28 | 河南中平自动化股份有限公司 | Intelligent control system and method for separating coal and gangue for mine |
| CN114037703A (en) * | 2022-01-10 | 2022-02-11 | 西南交通大学 | Subway valve state detection method based on two-dimensional positioning and three-dimensional attitude calculation |
| CN114359246A (en) * | 2022-01-10 | 2022-04-15 | 南京中远通科技有限公司 | Conveyor belt detection method, device, system, electronic device and medium |
| CN114937028A (en) * | 2022-06-21 | 2022-08-23 | 苏州上舜精密工业科技有限公司 | Intelligent identification-based quality detection method and system for linear sliding table module |
| CN115331157A (en) * | 2022-09-28 | 2022-11-11 | 山东山矿机械有限公司 | Conveyor abnormity detection method and system based on image processing |
| CN117671607A (en) * | 2024-02-01 | 2024-03-08 | 宝鸡杭叉工程机械有限责任公司 | Real-time detection method and system for abnormality of belt conveyor based on computer vision |
| CN117671607B (en) * | 2024-02-01 | 2024-04-26 | 宝鸡杭叉工程机械有限责任公司 | Real-time detection method and system for abnormality of belt conveyor based on computer vision |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2007322909A (en) * | 2006-06-02 | 2007-12-13 | Fuji Xerox Co Ltd | Image forming apparatus |
| CN104760812A (en) * | 2015-02-26 | 2015-07-08 | 三峡大学 | Monocular vision based real-time location system and method for products on conveying belt |
| CN108692636A (en) * | 2018-04-24 | 2018-10-23 | 庄银灿 | A kind of plank rapid size detection device for building monitoring |
| CN108898589A (en) * | 2018-06-19 | 2018-11-27 | 南通大学 | The quick-fried pearl intelligent detecting method of filter stick based on high speed machines vision |
| CN109708607A (en) * | 2017-10-26 | 2019-05-03 | 富鼎电子科技(嘉善)有限公司 | Detection device, detection method and storage equipment |
| CN111062387A (en) * | 2019-12-09 | 2020-04-24 | 熵智科技(深圳)有限公司 | Method for identifying articles on conveyor belt, method for grabbing articles on conveyor belt and related equipment |
| CN111702756A (en) * | 2020-05-26 | 2020-09-25 | 深圳市中纬智能有限公司 | Object grabbing method based on machine vision, robot and storage medium |
| CN111782456A (en) * | 2020-06-30 | 2020-10-16 | 平安国际智慧城市科技股份有限公司 | Anomaly detection method and device, computer equipment and storage medium |
| CN111829434A (en) * | 2019-05-28 | 2020-10-27 | 北京伟景智能科技有限公司 | Material flow metering detection method and system |
| CN112357454A (en) * | 2020-11-30 | 2021-02-12 | 华电重工股份有限公司 | Device and method for detecting overlapping position of rubber belt of tubular belt conveyor |
| CN112363153A (en) * | 2020-10-27 | 2021-02-12 | 河北燕大燕软信息系统有限公司 | Material pile edge detection method and system |
| CN112561887A (en) * | 2020-12-18 | 2021-03-26 | 中国矿业大学 | Belt conveyor coal flow binocular vision measurement method based on deep migration learning |
| CN112607362A (en) * | 2020-12-24 | 2021-04-06 | 中建材信息技术股份有限公司 | Belt deviation detection method based on video |
| CN113192016A (en) * | 2021-04-21 | 2021-07-30 | 深圳力维智联技术有限公司 | Method, device and equipment for detecting abnormal deformation of conveyor belt and storage medium |
-
2021
- 2021-08-12 CN CN202110921779.0A patent/CN113379743B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2007322909A (en) * | 2006-06-02 | 2007-12-13 | Fuji Xerox Co Ltd | Image forming apparatus |
| CN104760812A (en) * | 2015-02-26 | 2015-07-08 | 三峡大学 | Monocular vision based real-time location system and method for products on conveying belt |
| CN109708607A (en) * | 2017-10-26 | 2019-05-03 | 富鼎电子科技(嘉善)有限公司 | Detection device, detection method and storage equipment |
| CN108692636A (en) * | 2018-04-24 | 2018-10-23 | 庄银灿 | A kind of plank rapid size detection device for building monitoring |
| CN108898589A (en) * | 2018-06-19 | 2018-11-27 | 南通大学 | The quick-fried pearl intelligent detecting method of filter stick based on high speed machines vision |
| CN111829434A (en) * | 2019-05-28 | 2020-10-27 | 北京伟景智能科技有限公司 | Material flow metering detection method and system |
| CN111062387A (en) * | 2019-12-09 | 2020-04-24 | 熵智科技(深圳)有限公司 | Method for identifying articles on conveyor belt, method for grabbing articles on conveyor belt and related equipment |
| CN111702756A (en) * | 2020-05-26 | 2020-09-25 | 深圳市中纬智能有限公司 | Object grabbing method based on machine vision, robot and storage medium |
| CN111782456A (en) * | 2020-06-30 | 2020-10-16 | 平安国际智慧城市科技股份有限公司 | Anomaly detection method and device, computer equipment and storage medium |
| CN112363153A (en) * | 2020-10-27 | 2021-02-12 | 河北燕大燕软信息系统有限公司 | Material pile edge detection method and system |
| CN112357454A (en) * | 2020-11-30 | 2021-02-12 | 华电重工股份有限公司 | Device and method for detecting overlapping position of rubber belt of tubular belt conveyor |
| CN112561887A (en) * | 2020-12-18 | 2021-03-26 | 中国矿业大学 | Belt conveyor coal flow binocular vision measurement method based on deep migration learning |
| CN112607362A (en) * | 2020-12-24 | 2021-04-06 | 中建材信息技术股份有限公司 | Belt deviation detection method based on video |
| CN113192016A (en) * | 2021-04-21 | 2021-07-30 | 深圳力维智联技术有限公司 | Method, device and equipment for detecting abnormal deformation of conveyor belt and storage medium |
Non-Patent Citations (5)
| Title |
|---|
| CHARLES R. QI 等: "Frustum PointNets for 3D Object Detection from RGB-D Data", 《ARXIV》 * |
| MENGCHAO ZHANG 等: "A Computer Vision Based Conveyor System Detection Deviation", 《APPLIED SCIENCES》 * |
| XUANYU YIN 等: "3D Object Detection Method Based on YOLO and K-Means for Image and Point Clouds", 《ARXIV》 * |
| 宓逸舟: "基于双目视觉的快递包裹体积计量系统", 《万方数据库》 * |
| 陈洪敏: "3维模型2维投影图序列特征的异常检测", 《三明学院学报》 * |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113763376A (en) * | 2021-09-17 | 2021-12-07 | 深圳市赛为智能股份有限公司 | Conveyor belt deviation detection method and device, computer equipment and storage medium |
| CN113763376B (en) * | 2021-09-17 | 2024-03-01 | 深圳市赛为智能股份有限公司 | Conveyor belt offset detection method, conveyor belt offset detection device, computer equipment and storage medium |
| CN113989251A (en) * | 2021-11-02 | 2022-01-28 | 河南中平自动化股份有限公司 | Intelligent control system and method for separating coal and gangue for mine |
| CN114037703A (en) * | 2022-01-10 | 2022-02-11 | 西南交通大学 | Subway valve state detection method based on two-dimensional positioning and three-dimensional attitude calculation |
| CN114359246A (en) * | 2022-01-10 | 2022-04-15 | 南京中远通科技有限公司 | Conveyor belt detection method, device, system, electronic device and medium |
| CN114359246B (en) * | 2022-01-10 | 2023-03-10 | 南京中远通科技有限公司 | Conveyor belt detection method, device, system, electronic device and medium |
| CN114937028A (en) * | 2022-06-21 | 2022-08-23 | 苏州上舜精密工业科技有限公司 | Intelligent identification-based quality detection method and system for linear sliding table module |
| CN114937028B (en) * | 2022-06-21 | 2023-12-08 | 苏州上舜精密工业科技有限公司 | Intelligent identification and recognition linear sliding table module quality detection method and system |
| CN115331157A (en) * | 2022-09-28 | 2022-11-11 | 山东山矿机械有限公司 | Conveyor abnormity detection method and system based on image processing |
| CN115331157B (en) * | 2022-09-28 | 2023-02-17 | 山东山矿机械有限公司 | Conveyor abnormity detection method and system based on image processing |
| CN117671607A (en) * | 2024-02-01 | 2024-03-08 | 宝鸡杭叉工程机械有限责任公司 | Real-time detection method and system for abnormality of belt conveyor based on computer vision |
| CN117671607B (en) * | 2024-02-01 | 2024-04-26 | 宝鸡杭叉工程机械有限责任公司 | Real-time detection method and system for abnormality of belt conveyor based on computer vision |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113379743B (en) | 2021-10-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113379743B (en) | Conveyor abnormity detection method and system based on computer vision | |
| CN108416789A (en) | Method for detecting image edge and system | |
| CN109632808B (en) | Edge defect detection method and device, electronic equipment and storage medium | |
| CN111652825B (en) | Edge tracking straight line segment rapid detection device and method based on gradient direction constraint | |
| CN115797872B (en) | Packaging defect identification method, system, equipment and medium based on machine vision | |
| WO2017032308A1 (en) | Pcb board detection method and apparatus | |
| RU2012142713A (en) | METHOD AND SYSTEM FOR DETECTING AND DETERMINING THE GEOMETRIC, SPATIAL AND POSITIONAL CHARACTERISTICS OF PRODUCTS TRANSPORTED BY THE CONVEYOR OF CONTINUOUS ACTION, IN PARTICULAR UNIT OF PROCESSED, SMALL, SMOKED CHEESE | |
| CN113077437B (en) | Workpiece quality detection method and system | |
| CN111553950A (en) | Steel coil centering judgment method, system, medium and electronic terminal | |
| CN106780437B (en) | A kind of quick QFN chip plastic packaging image obtains and amplification method | |
| CN115684176A (en) | Online visual inspection system for film surface defects | |
| CN115953373A (en) | Glass defect detection method and device, electronic equipment and storage medium | |
| Ahmad et al. | Deep learning transforming the manufacturing industry: A case study | |
| CN113781511B (en) | Conveyor belt edge wear detection method, conveyor belt edge wear detection device, computer equipment and storage medium | |
| CN107767372B (en) | Chip pin online visual detection system and method for layered parallel computing | |
| CN115526878A (en) | Coal flow detection method and device of belt conveyor | |
| CN115973724A (en) | Automatic attitude adjusting method, device, equipment and medium for aviation luggage | |
| JP6815793B2 (en) | Rectangular area detection method, rectangular area detection device and program | |
| CN114494313A (en) | Conveyor belt deviation detection method based on edge detection | |
| CN114663402A (en) | Cable prolapse detection method based on Hough linear detection and curve fitting | |
| Ma et al. | Depth image denoising and key points extraction for manipulation plane detection | |
| Zheng | Design of intelligent manufacturing product identification and detection system based on machine vision | |
| CN114170318A (en) | Image processing method, apparatus, system, medium, and electronic device | |
| Bohlool et al. | Cost-efficient automated visual inspection system for small manufacturing industries based on SIFT | |
| Qi et al. | On-line detection of Hami big jujubes' size and shape based on machine vision |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: Conveyor anomaly detection method and system based on computer vision Effective date of registration: 20220419 Granted publication date: 20211029 Pledgee: Shandong Wenshang Rural Commercial Bank Co.,Ltd. Pledgor: SHANDONG ZHONGDU MACHINERY CO.,LTD. Registration number: Y2022980004415 |
|
| PE01 | Entry into force of the registration of the contract for pledge of patent right |