CN115187588B - Foreign matter detection method, foreign matter detection device, storage medium, and electronic apparatus - Google Patents

Foreign matter detection method, foreign matter detection device, storage medium, and electronic apparatus Download PDF

Info

Publication number
CN115187588B
CN115187588B CN202211088278.XA CN202211088278A CN115187588B CN 115187588 B CN115187588 B CN 115187588B CN 202211088278 A CN202211088278 A CN 202211088278A CN 115187588 B CN115187588 B CN 115187588B
Authority
CN
China
Prior art keywords
image
foreign matter
color
foreground
area
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202211088278.XA
Other languages
Chinese (zh)
Other versions
CN115187588A (en
Inventor
申远
戴亮亮
刘传峰
胡晋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hefei Gstar Intelligent Control Technical Co Ltd
Original Assignee
Hefei Gstar Intelligent Control Technical Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hefei Gstar Intelligent Control Technical Co Ltd filed Critical Hefei Gstar Intelligent Control Technical Co Ltd
Priority to CN202211088278.XA priority Critical patent/CN115187588B/en
Publication of CN115187588A publication Critical patent/CN115187588A/en
Application granted granted Critical
Publication of CN115187588B publication Critical patent/CN115187588B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30241—Trajectory

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Quality & Reliability (AREA)
  • Image Analysis (AREA)

Abstract

The invention relates to the technical field of foreign matter detection, and discloses a foreign matter detection method, a foreign matter detection device, a storage medium and electronic equipment, wherein the method comprises the following steps: acquiring a video stream of a transmission band area, and preprocessing a plurality of frames of images of the transmission band area in the video stream to obtain a plurality of frames of material foreground images; local features of the material foreground images of the frames are extracted in a blocking mode, abnormal blocks are screened, and foreign matter candidate areas of the material foreground images of the frames are obtained; and obtaining a foreign matter transportation track based on the transportation direction of the conveyor belt and the foreign matter candidate area of each frame of material foreground image, and determining whether the foreign matter is detected according to the foreign matter transportation track. The method has the advantages of low false detection rate, high foreign matter recall rate and strong robustness of the detection algorithm.

Description

Foreign matter detection method, foreign matter detection device, storage medium, and electronic apparatus
Technical Field
The present invention relates to the field of foreign object detection technologies, and in particular, to a method and an apparatus for detecting a foreign object, a storage medium, and an electronic device.
Background
In the industrial field, in the process of material transportation of a belt conveyor, unconventional objects except materials, such as metal or nonmetal objects of steel plates, iron sheets, steel bars and the like, can appear. The foreign matters are doped in the materials, so that the subsequent production process is influenced, the transportation problems of belt conveyor blockage, belt tearing, feed opening blockage and the like can be caused, and the production safety and the production efficiency are greatly influenced.
The related belt foreign matter detection means mainly comprises a method based on sensor numerical analysis, deep learning and image processing. The method based on the sensor numerical analysis cannot detect foreign matters of similar types to materials, cannot perform foreign matter visualization, and is high in deployment difficulty and high in cost. The foreign matter detection algorithm based on deep learning depends on a large number of real foreign matter images of the belt, and is high in actual acquisition difficulty and long in period. The belt foreign matter detection method based on single-frame image processing can only detect specific foreign matters and is easy to generate false detection.
Disclosure of Invention
The present invention is directed to solving, at least to some extent, one of the technical problems in the related art. Therefore, one objective of the present invention is to provide a foreign object detection method, which has the advantages of low false detection rate, high foreign object recall rate, and strong robustness of the detection algorithm.
A second object of the present invention is to provide a foreign object detection apparatus.
A third object of the invention is to propose a computer-readable storage medium.
A fourth object of the invention is to propose an electronic device.
In order to achieve the above object, a method for detecting a foreign object according to a first embodiment of the present invention includes: acquiring a video stream of a transmission band area, and preprocessing a plurality of frames of transmission band area images in the video stream to obtain a plurality of frames of material foreground images; extracting local features of the material foreground images of each frame in a blocking mode, and screening abnormal blocks to obtain foreign matter candidate areas of the material foreground images of each frame; and obtaining a foreign matter transportation track based on the transportation direction of the conveyor belt and the foreign matter candidate area of the material foreground image of each frame, and determining whether the foreign matter is detected according to the foreign matter transportation track.
According to the foreign matter detection method disclosed by the embodiment of the invention, the foreign matter detection is carried out based on the material foreground image, and compared with the existing detection method based on the whole image, the background interference is effectively reduced, and the false detection rate is reduced; the foreign matter characteristics of each image block are analyzed through the difference characteristics, so that the foreign matters can be identified according to the difference information such as colors, textures, edges and shapes between the foreign matters and materials, and the foreign matter detection recall rate is effectively improved; and on the basis of single-frame foreign matter detection, a foreign matter time sequence trajectory analysis method is introduced, so that the false detection of foreign matters can be effectively reduced, and the algorithm robustness is improved.
In addition, the foreign object detection method proposed according to the above embodiment of the present invention may have the following additional technical features:
according to an embodiment of the present invention, preprocessing a plurality of frames of images of a transmission band area in a video stream to obtain a plurality of frames of foreground images of a material, includes: acquiring a plurality of frames of the transmission band area images in the video stream at preset time intervals; marking the transmission band area in the transmission band area image of each frame to obtain a target area image; aiming at a current frame target area image, carrying out gray difference processing on the current frame target area image and a next frame target area image, and carrying out binarization by using a preset threshold value to obtain a motion difference image; filling the motion area in the motion difference image by using morphological closed operation, and calculating the connected domain area of the filled motion area to obtain a material mask; and overlapping the current frame target area image and the material mask with operation to obtain a current frame material foreground image.
According to one embodiment of the invention, the local features of the foreground image of the current frame material of each frame are extracted in a blocking manner, and abnormal block screening is performed, wherein the abnormal block screening comprises the following steps: dividing the current frame material foreground image by using pre-divided blocks with preset sizes to obtain M multiplied by N image blocks with the preset sizes; calculating the color distance characteristic of each image block in an RGB color space and the color variance characteristic of each image block in an HSV color space; and determining an abnormal block in the foreground image of the current frame material according to the color distance characteristic and the color variance characteristic of each image block.
According to one embodiment of the invention, the size of the pre-portioned blocks is determined based on the material granularity
Figure 630215DEST_PATH_IMAGE001
Wherein, in the step (A),
Figure 364952DEST_PATH_IMAGE002
indicates the width of the pre-partition,
Figure 973788DEST_PATH_IMAGE003
indicating the high of the pre-partition,
Figure 881701DEST_PATH_IMAGE004
particle size of larger than material
Figure 830066DEST_PATH_IMAGE005
Wherein, in the process,
Figure 798022DEST_PATH_IMAGE006
a length representing the estimated particle size of the material,
Figure 894154DEST_PATH_IMAGE007
a width representing an estimated material particle size;
the image block
Figure 543441DEST_PATH_IMAGE008
The expression of (c) is:
Figure 674208DEST_PATH_IMAGE009
wherein, the first and the second end of the pipe are connected with each other,
Figure 813065DEST_PATH_IMAGE010
m =1,2, 3., M denotes the image block horizontal index, N =1,2, 3., N denotes the image block vertical index, x, y denote the horizontal and vertical coordinates of the vertex at the upper left corner of the image block, respectively, and w, h denote the width and height of the image block, respectively.
According to one embodiment of the present invention, the expression of the color distance characteristic is:
Figure 130914DEST_PATH_IMAGE011
wherein, the first and the second end of the pipe are connected with each other,
Figure 583892DEST_PATH_IMAGE012
the color distance characteristic is represented by a color distance,
Figure 569166DEST_PATH_IMAGE013
is a function of the inverse cosine of the,
Figure 878924DEST_PATH_IMAGE014
representing the image block
Figure 621753DEST_PATH_IMAGE015
The color mean vector in the RGB color space,
Figure 675159DEST_PATH_IMAGE016
representing the color mean vector of the material foreground image in RGB color space, wherein the value range of the color distance characteristic is
Figure 514939DEST_PATH_IMAGE017
；
The expression of the color variance characteristic is as follows:
Figure 933282DEST_PATH_IMAGE018
wherein the content of the first and second substances,
Figure 491303DEST_PATH_IMAGE019
a color variance characteristic is represented by a color variance of the color image,
Figure 82821DEST_PATH_IMAGE020
for the image block
Figure 42687DEST_PATH_IMAGE015
The color variance of the chrominance component H component in the HSV color space,
Figure 631931DEST_PATH_IMAGE021
and the integral color variance of the chromaticity component H component of the material foreground image in the HSV color space.
According to an embodiment of the present invention, the determining the abnormal block in the foreground map of the current frame material according to the color distance characteristic and the color variance characteristic of each image block includes:
judging each image block in the current frame material foreground image
Figure 411668DEST_PATH_IMAGE015
Said color distance characteristic of
Figure 806877DEST_PATH_IMAGE012
Whether greater than a color distance threshold
Figure 561862DEST_PATH_IMAGE022
The color variance characteristic
Figure 384325DEST_PATH_IMAGE019
Whether greater than a color variance threshold
Figure 651358DEST_PATH_IMAGE023
；
When in use
Figure 787941DEST_PATH_IMAGE012
≥
Figure 456820DEST_PATH_IMAGE022
And is and
Figure 450184DEST_PATH_IMAGE019
≥
Figure 204513DEST_PATH_IMAGE023
determining the image block
Figure 144788DEST_PATH_IMAGE015
Is the exception block and is noted
Figure 668173DEST_PATH_IMAGE024
Wherein K =1,2, 3., K denotes an index of the foreign object candidate region, and t denotes a foreground map of the current frame material.
According to one embodiment of the invention, the foreign matter transportation track is obtained based on the transportation direction of the conveyor belt and the foreign matter candidate area of each frame of material foreground image, and the method comprises the following steps: obtaining a foreign matter candidate area in the current frame material foreground image
Figure 832438DEST_PATH_IMAGE024
Position information of each of the abnormal blocks; determining a preset position according to the transportation direction and the position information, and judging whether the preset position of the next frame of material foreground image is a foreign matter candidate area
Figure 11746DEST_PATH_IMAGE025
Wherein, in the process,
Figure 552449DEST_PATH_IMAGE025
representing a foreign matter candidate area of the next frame material foreground image, and t +1 representing the next frame material foreground image; if so, recording the track of the foreign matter candidate area, and continuing to perform track association of the foreign matter candidate area of the subsequent frame; if not, the foreign matter candidate area is cleared, and the track association of the foreign matter candidate area is not carried out.
In order to achieve the above object, a second aspect of the present invention provides a foreign object detection apparatus, including: the acquisition module is used for acquiring a video stream of a transmission belt area, and preprocessing a plurality of frames of images of the transmission belt area in the video stream to obtain a plurality of frames of material foreground images; the screening module is used for extracting local characteristics of the material foreground images of the frames in a blocking mode and screening abnormal blocks to obtain foreign matter candidate areas of the material foreground images of the frames; and the detection module is used for obtaining a foreign matter transportation track based on the transportation direction of the conveyor belt and the foreign matter candidate area of each frame of material foreground image, and determining whether the foreign matter is detected according to the foreign matter transportation track.
To achieve the above object, a third aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored, the computer program, when being executed by a processor, implementing the foreign object detection method according to the first aspect of the present invention.
In order to achieve the above object, a fourth aspect of the present invention provides an electronic device, including a memory and a processor, where the memory stores a computer program, and the computer program is executed by the processor to implement the foreign object detection method according to the first aspect of the present invention.
Drawings
FIG. 1 is a flow chart of a foreign object detection method according to an embodiment of the present invention;
FIG. 2 is a flow diagram of pre-processing a transport band region image according to one embodiment of the invention;
FIG. 3 is a schematic diagram of obtaining a map of a target area in accordance with one embodiment of the present invention;
FIG. 4 is a schematic flow chart of obtaining a material foreground map according to an embodiment of the present invention;
FIG. 5 is a flow chart of a method of detecting foreign objects in accordance with one embodiment of the present invention;
FIG. 6 is a schematic diagram of a candidate area for a foreign object in a foreground view of a material according to an embodiment of the present invention;
fig. 7 is a schematic view of a foreign object detection apparatus according to an embodiment of the present invention.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the accompanying drawings are illustrative and intended to explain the present invention and should not be construed as limiting the present invention.
The foreign object detection method, apparatus, storage medium, and electronic device according to the embodiments of the present invention will be described in detail with reference to fig. 1 to 7 and specific embodiments of the present invention.
Fig. 1 is a flowchart of a foreign object detection method according to an embodiment of the present invention. As shown in fig. 1, the foreign object detection method may include:
s1, acquiring a video stream of a transmission band area, and preprocessing multiple frames of images of the transmission band area in the video stream to obtain multiple frames of material foreground images;
according to the embodiment of the invention, when the foreign matter in the material on the transmission belt is detected, the video stream of the transmission belt area is obtained, the multi-frame transmission belt area image is obtained from the video stream, and the foreign matter detection is carried out on the single-frame transmission belt area image. In order to reduce background interference, the single-frame transmission band area image is preprocessed to obtain a single-frame material foreground image only containing a material area.
For better effect, the camera equipment for collecting the video stream in the area of the transmission belt is installed over the transmission belt, so that the camera equipment can clearly shoot the transportation condition of materials on the transmission belt, the quality of the obtained images in the area of the transmission belt is improved, and the influence of the image quality on the detection of foreign matters is reduced.
In an embodiment of the present invention, as shown in fig. 2, preprocessing a plurality of frames of images of a transmission band region in a video stream to obtain a plurality of frames of foreground images of a material may include:
and S11, acquiring multi-frame transmission band area images in the video stream at preset time intervals.
In order to avoid missing partial materials on a transmission belt when a multi-frame transmission belt area image is acquired from a video stream at a preset time interval i, the preset time interval i can be set according to the running speed of the transmission belt. And acquiring multi-frame transmission band area images from the video stream according to the set time interval i, preprocessing the single-frame transmission band area images, and detecting foreign matters in the preprocessed images.
In the embodiment of the present invention, since the running speed of the transmission belt is relatively uniform and does not change suddenly, i can take 10s, that is, every 10s, one transmission belt region image is obtained from the video stream.
And S12, labeling the transmission band region in the transmission band region image of each frame to obtain a target region image.
Specifically, as shown in FIG. 3, the four vertices of the transmission band region in the single frame transmission band region image are marked clockwise, denoted A, B, C, and D in that order. The quadrilateral ABCD contains a transmission band region. And intercepting the external matrix area marked with the quadrangle ABCD from the image of the transmission belt area, and recording the external matrix area as a matrix A 'B' C 'D'. Pixels other than the transfer band in the matrix a 'B' C 'D' are 0-filled, and a partial image, i.e., a target area map, with respect to only the transfer band is acquired.
And S13, carrying out gray difference processing on the current frame target area image and the next frame target area image, and carrying out binarization by using a preset threshold value to obtain a motion difference image.
Specifically, for the current frame target area map, i.e. the tth frame target area map
Figure 930341DEST_PATH_IMAGE026
And its next frame target area map, i.e. t +1 th frame target area map
Figure 468770DEST_PATH_IMAGE027
Performing gray level processing on the target area image of the transmission belt area image acquired at the t + i second, calculating the gray level value of each pixel of the current frame target area image and the next frame target area image, and subtracting the gray level value of the corresponding pixel point to obtain the gray level difference value of each pixel point, comparing the gray level difference value with a preset threshold value, wherein the gray level difference value is within the range of the preset threshold value, and the gray level value of the difference image at the pixel point is regarded as 0; if the gray difference is outside the preset threshold range, the gray value of the difference image at the pixel point is 255, and a motion difference image is obtained, as shown in fig. 4.
In the embodiment of the present invention, when i =10, the difference processing is performed every 10 s. Wherein, i is 10, and the algorithm speed can be increased.
And S14, filling the motion area in the motion difference image by using morphological closed operation, and calculating the connected domain area of the filled motion area to obtain the material mask.
Specifically, as shown in fig. 4, in the morphological processing algorithm, the operation of performing the dilation process first and then performing the erosion process may be referred to as a closed operation, and the closed operation may be used for filling fine holes in the object, connecting neighboring objects, and smoothing the boundary thereof. And performing morphological closed operation processing on the motion difference image after the binarization processing to fill a motion area in the motion difference image, and connecting images formed by high-numerical-value pixel points with relatively close positions together to obtain a plurality of connected domains. And respectively counting the number of pixel points contained in each connected domain in the motion difference image after filling processing, wherein the connected domain with the largest number of pixel points is a material mask.
In the embodiment of the present invention, the number of times of the expansion processing and the etching processing in the closed-loop operation is not limited, and the optimal processing effect is achieved.
And S15, overlapping the current frame target area image and the material mask with operation to obtain a current frame material foreground image.
Will be whenTarget area map of previous frame, i.e. target area map of t-th frame
Figure 932112DEST_PATH_IMAGE026
And overlapping with the obtained material mask to obtain a current frame material foreground image
Figure 276506DEST_PATH_IMAGE028
See fig. 4. And carrying out the processing on the images of the transmission belt areas of each frame to obtain a material foreground image of the images of the transmission belt areas of each frame, and carrying out foreign matter detection on the material foreground image of the images of the transmission belt areas of each frame.
And S2, extracting local features of the material foreground images of the frames in a blocking manner, and screening abnormal blocks to obtain foreign matter candidate areas of the material foreground images of the frames.
Specifically, local features in the material foreground image are extracted in a blocking mode by means of preset blocks, abnormal block screening is conducted on the local features of each image block, and abnormal blocks of the material foreground image are detected, wherein each abnormal block in the material foreground image is a foreign matter candidate area of the material foreground image.
In an embodiment of the present invention, as shown in fig. 5, extracting local features of a foreground image of a current frame material in each frame in a block manner, and performing abnormal block screening may include:
segmenting a current frame material foreground image by utilizing pre-segmentation blocks with preset sizes to obtain M multiplied by N image blocks with preset sizes;
calculating the color distance characteristic of each image block in an RGB color space and the color variance characteristic of each image block in an HSV color space;
and determining an abnormal block in the foreground image of the current frame material according to the color distance characteristic and the color variance characteristic of each image block.
Referring to fig. 6, a foreground image of a current frame material is segmented by using pre-partitioned blocks of a preset size to obtain M × N image blocks of the preset size, where the size of the pre-partitioned blocks may be determined according to the material granularity. Because in the industrial production process, the conveying belt generally transports the same materials, the colors of the materials are generally close, and the main difference is the material granularity and the light color. The foreign matters usually include misplaced equipment, aged and fallen mechanical parts, belt fences and other objects, and have larger differences from the colors and forms of the materials. Therefore, the color distance characteristic in the RGB color space and the color variance characteristic in the HSV color space of each image block can be calculated. And detecting abnormal blocks in the foreground image of the current frame material according to the color distance characteristic and the color variance characteristic of each image block.
In an embodiment of the invention, the size of the pre-partitions is determined based on the material granularity
Figure 508904DEST_PATH_IMAGE001
Wherein, in the process,
Figure 218234DEST_PATH_IMAGE002
indicates the width of the pre-partition,
Figure 168872DEST_PATH_IMAGE003
indicating the high of the pre-partition,
Figure 51378DEST_PATH_IMAGE004
greater than estimated material particle size
Figure 341545DEST_PATH_IMAGE005
Wherein, in the step (A),
Figure 284093DEST_PATH_IMAGE006
indicates the length of the estimated particle size of the material,
Figure 722028DEST_PATH_IMAGE007
a width representing an estimated material particle size;
image block
Figure 345907DEST_PATH_IMAGE029
The expression is given as:
Figure 552897DEST_PATH_IMAGE009
wherein, the first and the second end of the pipe are connected with each other,
Figure 666347DEST_PATH_IMAGE010
m =1,2, 3., M denotes an image block horizontal index, N =1,2, 3., N denotes an image block vertical index, x, y denote the horizontal and vertical coordinates of the vertex at the upper left corner of the image block, respectively, and w, h denote the width and height of the image block, respectively.
It should be noted that m and n are positive integers, and the value range is determined according to the horizontal number and the vertical number of the image blocks in the material foreground image.
In an embodiment of the invention, the pre-tiles are squares, wherein,
Figure 526331DEST_PATH_IMAGE010
，
Figure 750639DEST_PATH_IMAGE030
representing the side length of a preset pre-partition block.
In an embodiment of the present invention, the expression of the color distance characteristic is:
Figure 77715DEST_PATH_IMAGE011
wherein the content of the first and second substances,
Figure 96487DEST_PATH_IMAGE012
a color distance characteristic is represented by a color distance characteristic,
Figure 446697DEST_PATH_IMAGE013
is a function of the inverse cosine of the,
Figure 474695DEST_PATH_IMAGE031
representing image blocks
Figure 656278DEST_PATH_IMAGE032
A color mean vector in the RGB color space,
Figure 49213DEST_PATH_IMAGE016
representing colors of material foreground images in RGB color spaceMean vector, color distance characteristics of value range of
Figure 683457DEST_PATH_IMAGE033
；
The expression for the color variance feature is:
Figure 249568DEST_PATH_IMAGE018
wherein, the first and the second end of the pipe are connected with each other,
Figure 488919DEST_PATH_IMAGE019
the color variance characteristic is represented by a color variance characteristic,
Figure 115072DEST_PATH_IMAGE020
is an image block
Figure 236612DEST_PATH_IMAGE015
The color variance of the chrominance component H component in the HSV color space,
Figure 544097DEST_PATH_IMAGE021
and the integral color variance of the chromaticity component H component of the material foreground image in the HSV color space.
Specifically, the color distance characteristic of each image block in the RGB color space is calculated using the above expression of the color distance characteristic and the expression of the color variance characteristic
Figure 700272DEST_PATH_IMAGE012
And color variance characteristics in HSV color space
Figure 497326DEST_PATH_IMAGE019
. According to the color distance threshold
Figure 43845DEST_PATH_IMAGE022
And color variance threshold
Figure 951758DEST_PATH_IMAGE034
Color distance for each image blockCharacteristic of
Figure 962440DEST_PATH_IMAGE012
And color variance characteristics
Figure 930396DEST_PATH_IMAGE019
And (5) judging to determine an abnormal block in the current frame material foreground image.
In an embodiment of the present invention, as shown in fig. 5, determining an abnormal block in the foreground map of the current frame material according to the color distance characteristic and the color variance characteristic of each image block may include:
judging each image block in the foreground image of the current frame material
Figure 964211DEST_PATH_IMAGE035
Color distance characteristic of
Figure 675815DEST_PATH_IMAGE012
Whether greater than a color distance threshold
Figure 541003DEST_PATH_IMAGE022
Color variance characteristics
Figure 617543DEST_PATH_IMAGE019
Whether greater than a color variance threshold
Figure 200971DEST_PATH_IMAGE023
；
When the temperature is higher than the set temperature
Figure 716266DEST_PATH_IMAGE012
≥
Figure 388292DEST_PATH_IMAGE022
And is and
Figure 698051DEST_PATH_IMAGE019
≥
Figure 768775DEST_PATH_IMAGE023
then, image blocks are determined
Figure 822181DEST_PATH_IMAGE015
Is an abnormal block and is recorded as
Figure 599645DEST_PATH_IMAGE024
Wherein K =1,2, 3., K denotes an index of the foreign object candidate region, and t denotes a foreground map of the current frame material.
Specifically, each image block
Figure 80305DEST_PATH_IMAGE035
Color distance characteristic of
Figure 638325DEST_PATH_IMAGE012
Distance from color threshold
Figure 167526DEST_PATH_IMAGE022
Comparing, color variance characterization
Figure 861813DEST_PATH_IMAGE019
And color variance threshold
Figure 778953DEST_PATH_IMAGE034
A comparison is made. Judging image blocks
Figure 496373DEST_PATH_IMAGE035
Color distance characteristic of
Figure 891583DEST_PATH_IMAGE012
Whether greater than a color distance threshold
Figure 705955DEST_PATH_IMAGE022
Color variance characteristics
Figure 466101DEST_PATH_IMAGE019
Whether greater than a color variance threshold
Figure 733134DEST_PATH_IMAGE023
. Wherein the color distance characteristic
Figure 932034DEST_PATH_IMAGE012
Greater than a color distance threshold
Figure 600913DEST_PATH_IMAGE022
And color variance characteristics
Figure 531960DEST_PATH_IMAGE019
Is also greater than the color variance threshold
Figure 286289DEST_PATH_IMAGE023
Image block of
Figure 23301DEST_PATH_IMAGE035
Is an exception block.
In addition, according to the image blocks
Figure 484369DEST_PATH_IMAGE035
M and n of (c), the row and column where the image block that is the abnormal block is located, i.e., the location of the abnormal block, can be determined.
And S3, obtaining a foreign matter transportation track based on the transportation direction of the conveyor belt and the foreign matter candidate area of each frame of material foreground image, and determining whether the foreign matter is detected according to the foreign matter transportation track.
In an embodiment of the present invention, as shown in fig. 5, obtaining a foreign object transportation track based on a transportation direction of a conveyor and a foreign object candidate region of a foreground map of each frame of material may include:
obtaining a foreign matter candidate area in a current frame material foreground image
Figure 914214DEST_PATH_IMAGE024
Position information of each abnormal block;
determining a preset position according to the transportation direction and the position information, and judging whether the preset position of the next frame of material foreground image is a foreign matter candidate area
Figure 155839DEST_PATH_IMAGE025
Wherein, in the step (A),
Figure 368646DEST_PATH_IMAGE025
representing a foreign matter candidate area of the next frame material foreground image, and t +1 representing the next frame material foreground image;
if so, recording the foreign matter candidate area track, and continuing to perform track association of the foreign matter candidate area of the subsequent frame;
if not, the foreign matter candidate area is removed, and the track association of the foreign matter candidate area is not carried out.
Specifically, a foreign matter candidate area in a current frame material foreground image is obtained
Figure 12117DEST_PATH_IMAGE024
Position information of each abnormal block
Figure 282036DEST_PATH_IMAGE036
Determining the preset position of the foreign object block corresponding to the foreground image of the next frame of material according to the conveying direction of the conveying belt, the conveying speed and the preset time interval as well as the foreground image of the current frame of material, namely the position information of the foreign object block in the foreground image of the tth frame of material
Figure 10958DEST_PATH_IMAGE037
In order to prevent the occurrence of other problems such as jamming of the conveyor belt, the predetermined position may further include
Figure 89772DEST_PATH_IMAGE038
And/or
Figure 525433DEST_PATH_IMAGE039
To prevent missed detection. After the preset position is determined, judging whether the preset position of the next frame of material foreground image is a foreign matter candidate area or not
Figure 297080DEST_PATH_IMAGE025
. If the preset position in the next frame of material foreground image is a foreign matter candidate area
Figure 247718DEST_PATH_IMAGE025
Recording the foreign object candidate area track and continuingAnd performing track association of foreign matter candidate areas of the subsequent frames. If the preset position of the next frame of material foreground image is not the foreign matter candidate area
Figure 67907DEST_PATH_IMAGE025
And removing the foreign matter candidate area, and not performing the track correlation of the foreign matter candidate area so as to improve the accuracy of foreign matter detection and reduce the false detection rate.
Wherein the candidate foreign body track obtained by correlation is recorded as
Figure 420391DEST_PATH_IMAGE040
Wherein id represents the index corresponding to the track.
Figure 362939DEST_PATH_IMAGE040
The following conditions need to be satisfied: the starting position of the track is the starting position of material transportation in the image; the track end position is a material transportation end position in the image.
Compared with the existing detection method based on the whole image, the foreign matter detection method based on the material foreground image effectively reduces background interference and lowers false detection rate; foreign matter characteristics of each image block are analyzed through difference characteristics, so that the foreign matters can be identified according to difference information such as colors, textures, edges and shapes between the foreign matters and materials, and the foreign matter detection recall rate is effectively improved; and on the basis of single-frame foreign matter detection, a foreign matter time sequence trajectory analysis method is introduced, so that the false detection of foreign matters can be effectively reduced, and the algorithm robustness is improved.
The invention also provides a foreign matter detection device.
Fig. 7 is a schematic view of a foreign object detection apparatus according to an embodiment of the present invention. As shown in fig. 7, the foreign object detection apparatus 100 may include an acquisition module 10, a screening module 20, and a detection module 30.
The acquiring module 10 is configured to acquire a video stream of a transmission band region, and pre-process a plurality of frames of images of the transmission band region in the video stream to obtain a plurality of frames of material foreground images; the screening module 20 is configured to extract local features of each frame of material foreground image in a block manner, and perform abnormal block screening to obtain a foreign matter candidate region of each frame of material foreground image; the detection module 30 is configured to obtain a foreign object transportation track based on the transportation direction of the conveyor belt and the foreign object candidate region of each frame of the material foreground map, and determine whether to detect a foreign object according to the foreign object transportation track.
It should be noted that, for other specific embodiments of the foreign object detection apparatus according to the embodiment of the present invention, reference may be made to specific embodiments of the foreign object detection method according to the above-described embodiment of the present invention.
Compared with the existing detection method based on the whole image, the foreign matter detection device provided by the embodiment of the invention effectively reduces background interference and lowers false detection rate; foreign matter characteristics of each image block are analyzed through difference characteristics, so that the foreign matters can be identified according to difference information such as colors, textures, edges and shapes between the foreign matters and materials, and the foreign matter detection recall rate is effectively improved; and on the basis of single-frame foreign matter detection, a foreign matter time sequence trajectory analysis method is introduced, so that the false detection of foreign matters can be effectively reduced, and the algorithm robustness is improved.
The invention also provides a computer readable storage medium.
In this embodiment, a computer-readable storage medium has stored thereon a computer program, which corresponds to the above-mentioned foreign object detection method, and which, when executed by a processor, implements the foreign object detection method as described above.
The invention also provides the electronic equipment.
In this embodiment, the electronic device comprises a processor, a memory and a computer program stored on the memory, which when executed by the processor implements the foreign object detection method as described above.
The storage medium and the electronic device of the embodiment of the invention utilize the foreign matter detection method to realize the foreign matter detection of the transmission belt.
It should be noted that the logic and/or steps shown in the flowcharts or otherwise described herein, such as an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
It should be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
In the description of the present invention, it is to be understood that the terms "central," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," "circumferential," and the like are used in the orientations and positional relationships indicated in the drawings for convenience in describing the invention and to simplify the description, and are not intended to indicate or imply that the referenced devices or elements must have a particular orientation, be constructed and operated in a particular orientation, and are therefore not to be considered limiting of the invention.
Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention, and that variations, modifications, substitutions and alterations can be made to the above embodiments by those of ordinary skill in the art within the scope of the present invention.

Claims (8)

1. A foreign object detection method, characterized in that the method comprises:
acquiring a video stream of a transmission band area, and preprocessing a plurality of frames of images of the transmission band area in the video stream to obtain a plurality of frames of material foreground images;
extracting local features of the material foreground images of each frame in a blocking manner, and screening abnormal blocks to obtain foreign matter candidate areas of the material foreground images of each frame;
obtaining a foreign matter transportation track based on the transportation direction of a conveyor belt and the foreign matter candidate area of each frame of the material foreground image, and determining whether foreign matters are detected according to the foreign matter transportation track;
extracting local features of the material foreground images of each frame in a blocking mode, and screening abnormal blocks, wherein the method comprises the following steps:
segmenting a current frame material foreground image by utilizing pre-segmentation blocks with preset sizes to obtain M multiplied by N image blocks with the preset sizes;
calculating the color distance characteristic of each image block in an RGB color space and the color variance characteristic of each image block in an HSV color space;
determining an abnormal block in the foreground image of the current frame material according to the color distance characteristic and the color variance characteristic of each image block;
the expression of the color distance feature is:
Figure 684077DEST_PATH_IMAGE001
wherein the content of the first and second substances,
Figure 652033DEST_PATH_IMAGE002
a feature representing a distance of said color is provided,
Figure 311947DEST_PATH_IMAGE003
is a function of the inverse cosine of the,
Figure 23551DEST_PATH_IMAGE004
representing image blocks in the m-th row and n-th column
Figure 888739DEST_PATH_IMAGE005
A color mean vector in an RGB color space, wherein M =1,2, 3.., M, M denotes the tile lateral index, N =1,2, 3.., N, N denotes the tile longitudinal index,
Figure 27596DEST_PATH_IMAGE006
representing the color mean vector of the material foreground image in an RGB color space, wherein the value range of the color distance characteristic is
Figure 673341DEST_PATH_IMAGE007
；
The expression of the color variance characteristic is as follows:
Figure 923057DEST_PATH_IMAGE008
wherein the content of the first and second substances,
Figure 908331DEST_PATH_IMAGE009
a feature representing the variance of the color is presented,
Figure 218089DEST_PATH_IMAGE010
for the image block
Figure 288813DEST_PATH_IMAGE011
The color variance of the chrominance component H component in the HSV color space,
Figure 404537DEST_PATH_IMAGE012
and the integral color variance of the chromaticity component H component of the material foreground image in the HSV color space.
2. The foreign object detection method of claim 1, wherein preprocessing the images of the plurality of frames of the transmission band region in the video stream to obtain a plurality of frames of the foreground image of the material comprises:
acquiring multiple frames of the transmission band area images in the video stream at preset time intervals;
marking the transmission band area in the transmission band area image of each frame to obtain a target area image;
aiming at a current frame target area image, carrying out gray level difference processing on the current frame target area image and a next frame target area image, and carrying out binarization by using a preset threshold value to obtain a motion difference image;
filling the motion area in the motion difference image by using morphological closed operation, and calculating the connected domain area of the filled motion area to obtain a material mask;
and overlapping the current frame target area image and the material mask with operation to obtain a current frame material foreground image.
3. The foreign object detection method according to claim 2, wherein the size of the pre-partition is determined based on the particle size of the material
Figure 244317DEST_PATH_IMAGE013
Wherein, in the step (A),
Figure 724977DEST_PATH_IMAGE014
indicates the width of the pre-partition,
Figure 17418DEST_PATH_IMAGE015
indicating the high of the pre-partition,
Figure 936832DEST_PATH_IMAGE013
particle size of larger than material
Figure 631119DEST_PATH_IMAGE016
Wherein, in the step (A),
Figure 282680DEST_PATH_IMAGE017
indicates the length of the estimated particle size of the material,
Figure 327997DEST_PATH_IMAGE018
a width representing an estimated material particle size;
the image block
Figure 723206DEST_PATH_IMAGE019
The expression of (a) is:
Figure 835781DEST_PATH_IMAGE020
wherein the content of the first and second substances,
Figure 923822DEST_PATH_IMAGE021
m =1,2,3,. Multidot., M represents the image block horizontal index, N =1,2,3,. Multidot., N represents the image block longitudinal index, x, y represent the horizontal and vertical coordinates of the vertex at the upper left corner of the image block, respectively, and w, h represent the width and height of the image block, respectively.
4. The method according to claim 3, wherein the determining the abnormal block in the foreground map of the current frame material according to the color distance feature and the color variance feature of each image block comprises:
judging each image block in the current frame material foreground image
Figure 190856DEST_PATH_IMAGE011
Said color distance characteristic of
Figure 389756DEST_PATH_IMAGE002
Whether or not it is greater than or equal to a color distance threshold
Figure 855372DEST_PATH_IMAGE022
The color variance characteristic
Figure 114315DEST_PATH_IMAGE009
Whether or not it is greater than or equal to the color variance threshold
Figure 868645DEST_PATH_IMAGE023
；
When in use
Figure 605656DEST_PATH_IMAGE002
≥
Figure 129042DEST_PATH_IMAGE022
And is and
Figure 621203DEST_PATH_IMAGE009
≥
Figure 597249DEST_PATH_IMAGE023
then, the image block is determined
Figure 137952DEST_PATH_IMAGE011
Is the exception block and is noted
Figure 781423DEST_PATH_IMAGE024
Wherein K =1,2, 3., K denotes an index of the foreign object candidate region, and t denotes a current frame material foreground map.
5. The foreign matter detection method according to claim 4, wherein obtaining the foreign matter transport trajectory based on the transport direction of the transport belt and the foreign matter candidate region of the material foreground map of each frame comprises:
obtaining a foreign matter candidate area in the foreground image of the current frame material
Figure 178906DEST_PATH_IMAGE024
Position information of each of the abnormal blocks;
determining a preset position according to the transportation direction and the position information, and judging whether the preset position of the next frame of material foreground image is a foreign matter candidate area
Figure 907828DEST_PATH_IMAGE025
Wherein, in the step (A),
Figure 986642DEST_PATH_IMAGE025
representing a foreign matter candidate area of the next frame material foreground image, and t +1 representing the next frame material foreground image;
if so, recording the track of the foreign matter candidate area, and continuing to perform track association of the foreign matter candidate area of the subsequent frame;
if not, the foreign matter candidate area is cleared, and the track association of the foreign matter candidate area is not carried out.
6. A foreign object detection apparatus, characterized in that the apparatus comprises:
the acquisition module is used for acquiring a video stream of a transmission belt area, and preprocessing a plurality of frames of images of the transmission belt area in the video stream to obtain a plurality of frames of material foreground images;
the screening module is used for extracting local characteristics of the material foreground images of each frame in a blocking mode and screening abnormal blocks to obtain foreign matter candidate areas of the material foreground images of each frame;
the detection module is used for obtaining a foreign matter transportation track based on the transportation direction of the conveyor belt and the foreign matter candidate area of each frame of material foreground image, and determining whether the foreign matter is detected according to the foreign matter transportation track;
the screening module is specifically used for segmenting a current frame material foreground image by utilizing pre-segmentation blocks with preset sizes to obtain M multiplied by N image blocks with the preset sizes;
calculating the color distance characteristic of each image block in an RGB color space and the color variance characteristic of each image block in an HSV color space;
determining an abnormal block in the foreground image of the current frame material according to the color distance characteristic and the color variance characteristic of each image block;
the expression of the color distance characteristic is as follows:
Figure 484620DEST_PATH_IMAGE001
wherein the content of the first and second substances,
Figure 990687DEST_PATH_IMAGE002
the color distance characteristic is represented by a color distance,
Figure 770687DEST_PATH_IMAGE003
in the form of an inverse cosine function,
Figure 653192DEST_PATH_IMAGE004
image block for representing m row and n column
Figure 5676DEST_PATH_IMAGE005
A color mean vector in an RGB color space, wherein M =1,2, 3.., M, M denotes the tile lateral index, N =1,2, 3.., N, N denotes the tile longitudinal index,
Figure 948224DEST_PATH_IMAGE006
representing the color mean vector of the material foreground image in an RGB color space, wherein the value range of the color distance characteristic is
Figure 448476DEST_PATH_IMAGE007
；
The expression of the color variance characteristic is as follows:
Figure 869093DEST_PATH_IMAGE008
wherein the content of the first and second substances,
Figure 341662DEST_PATH_IMAGE009
a color variance characteristic is represented by a color variance of the color image,
Figure 189533DEST_PATH_IMAGE010
for the image block
Figure 114763DEST_PATH_IMAGE011
The color variance of the chrominance component H component in the HSV color space,
Figure 401388DEST_PATH_IMAGE012
and the integral color variance of the chromaticity component H component of the material foreground image in the HSV color space.
7. A computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements a foreign object detection method according to any one of claims 1 to 5.
8. An electronic device comprising a memory and a processor, the memory having stored thereon a computer program, wherein the computer program, when executed by the processor, implements the foreign object detection method of any of claims 1-5.
CN202211088278.XA 2022-09-07 2022-09-07 Foreign matter detection method, foreign matter detection device, storage medium, and electronic apparatus Active CN115187588B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211088278.XA CN115187588B (en) 2022-09-07 2022-09-07 Foreign matter detection method, foreign matter detection device, storage medium, and electronic apparatus

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211088278.XA CN115187588B (en) 2022-09-07 2022-09-07 Foreign matter detection method, foreign matter detection device, storage medium, and electronic apparatus

Publications (2)

Publication Number Publication Date
CN115187588A CN115187588A (en) 2022-10-14
CN115187588B true CN115187588B (en) 2022-12-09

Family

ID=83523803

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211088278.XA Active CN115187588B (en) 2022-09-07 2022-09-07 Foreign matter detection method, foreign matter detection device, storage medium, and electronic apparatus

Country Status (1)

Country Link
CN (1) CN115187588B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115947066B (en) * 2023-03-14 2023-06-20 合肥金星智控科技股份有限公司 Belt tearing detection method, device and system

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104700430A (en) * 2014-10-05 2015-06-10 安徽工程大学 Method for detecting movement of airborne displays
CN106778595B (en) * 2016-12-12 2020-04-07 河北工业大学 Method for detecting abnormal behaviors in crowd based on Gaussian mixture model
CN110555868A (en) * 2019-05-31 2019-12-10 南京航空航天大学 method for detecting small moving target under complex ground background
CN113486714B (en) * 2021-06-03 2022-09-02 荣耀终端有限公司 Image processing method and electronic equipment

Also Published As

Publication number Publication date
CN115187588A (en) 2022-10-14

Similar Documents

Publication Publication Date Title
CN114937055B (en) Image self-adaptive segmentation method and system based on artificial intelligence
US9946941B2 (en) Lane detection
TWI409718B (en) Method of locating license plate of moving vehicle
CN112184744B (en) Display screen edge defect detection method and device
US20070110309A1 (en) Shadow detection in images
CN109146860B (en) Full-automatic mechanical equipment installation leakage detection method and device
CN113221861B (en) Multi-lane line detection method, device and detection equipment
CN115187588B (en) Foreign matter detection method, foreign matter detection device, storage medium, and electronic apparatus
JPH0957201A (en) Specific color region extracting system and specific color region removing system
US8000535B2 (en) Methods and systems for refining text segmentation results
CN107610132B (en) Method for removing stains from ancient book document image
JP2022551366A (en) Method, computer program product and computer readable medium for generating masks for camera streams
CN114998317A (en) Lens occlusion detection method and device, camera device and storage medium
US10115028B2 (en) Method and device for classifying an object in an image
CN114037730A (en) Traffic flow statistical method, device, equipment and storage medium
Vargas et al. A shadow removal algorithm for vehicle detection based on reflectance ratio and edge density
US20140285662A1 (en) Image processing apparatus, and method
US20150117769A1 (en) Blob-Encoding
Manipriya et al. Performance analysis of spatial color information for object detection using background subtraction
CN114581447B (en) Conveying belt deviation identification method and device based on machine vision
CN106951831B (en) Pedestrian detection tracking method based on depth camera
CN115661194A (en) Moving object extraction method, system, electronic device and medium
CN110223299B (en) Abrasive particle segmentation method based on deposition process
CN113516608A (en) Tire defect detection method and device, and tire detection device
JPH08139995A (en) Method for detecting scene change point of moving image

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant