CN112560743A - Public area foreign matter detection method, device, equipment and storage medium - Google Patents

Public area foreign matter detection method, device, equipment and storage medium Download PDF

Info

Publication number
CN112560743A
CN112560743A CN202011538797.2A CN202011538797A CN112560743A CN 112560743 A CN112560743 A CN 112560743A CN 202011538797 A CN202011538797 A CN 202011538797A CN 112560743 A CN112560743 A CN 112560743A
Authority
CN
China
Prior art keywords
foreign matter
pedestrian
feature
inputting
monitoring image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202011538797.2A
Other languages
Chinese (zh)
Inventor
梁俊杰
赖众程
洪叁亮
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An Bank Co Ltd
Original Assignee
Ping An Bank Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An Bank Co Ltd filed Critical Ping An Bank Co Ltd
Priority to CN202011538797.2A priority Critical patent/CN112560743A/en
Publication of CN112560743A publication Critical patent/CN112560743A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/07—Target detection

Abstract

The invention relates to the field of artificial intelligence and discloses a public area foreign matter detection method, a public area foreign matter detection device, public area foreign matter detection equipment and a public area foreign matter detection storage medium. The method comprises the following steps: collecting a first monitoring image and filtering noise and light to obtain a second monitoring image; calibrating a pedestrian detection area and a foreign matter detection area in the second monitoring image; inputting the second monitoring image and the pedestrian detection area into a pedestrian detection model for pedestrian identification; if the pedestrian identification result is that a pedestrian exists, inputting the second monitoring image and the foreign matter detection area into a foreign matter positioning model for foreign matter positioning; and if the foreign matter positioning result is not empty, inputting the positioned foreign matter external rectangular image into a foreign matter classification model for article classification, outputting probability values of the foreign matters belonging to different article names, taking the article name corresponding to the maximum probability value as the foreign matter name and carrying out foreign matter early warning. The invention can recognize foreign matters in an anti-interference manner, improve the recognition precision of the foreign matters, and simultaneously can automatically recognize the types of the foreign matters and carry out early warning.

Description

Public area foreign matter detection method, device, equipment and storage medium
Technical Field
The invention relates to the technical field of image recognition, in particular to a public area foreign matter detection method, device, equipment and storage medium.
Background
In some service industries, a comfortable, spacious, clean and quiet service environment is provided for customers, and the service experience or safety of the customers is ensured, for example, in a bank hall, foreign matter inspection is usually required in a public area. The business personnel often observe the environment of each business area through the camera at regular intervals, whether the desktop is neat, clean and the like. The inspection is usually performed by pure manual observation, which not only is inefficient and inaccurate in observation time, but also wastes labor cost.
Although the existing mode of automatically identifying the foreign matters based on the camera can be automatically completed by a machine, the robustness of the foreign matters identification process in a complex environment is poor, the foreign matters are easily interfered by the environment, for example, the foreign matters are mistakenly detected due to other interferences such as shadow generated by light, the detection accuracy is required to be improved, and the anti-interference capability is insufficient.
Disclosure of Invention
The invention mainly aims to solve the technical problem of improving the foreign matter detection accuracy rate in a complex public scene.
The invention provides a public area foreign matter detection method, which comprises the following steps:
acquiring a first monitoring image of a target monitoring area, and filtering noise and light of the first monitoring image to obtain a second monitoring image after interference is removed;
calibrating a pedestrian detection area and a foreign matter detection area in the second monitoring image based on a preset coordinate calibration frame;
inputting the second monitoring image and the pedestrian detection area into a preset pedestrian detection model for pedestrian identification, and outputting a pedestrian identification result;
if the pedestrian identification result is that a pedestrian exists, inputting the second monitoring image and the foreign matter detection area into a preset foreign matter positioning model for foreign matter positioning, and outputting a foreign matter positioning result;
if the foreign matter positioning result is not empty, inputting the positioned foreign matter circumscribed rectangular image into a preset foreign matter classification model for article classification, and outputting probability values of the target foreign matter corresponding to different article names respectively;
and selecting the article name corresponding to the maximum probability value as the article name of the target foreign matter, and carrying out foreign matter early warning on the target foreign matter.
Optionally, in a first implementation manner of the first aspect of the present invention, the performing noise and light filtering on the first monitoring image to obtain a second monitoring image after interference elimination includes:
transforming the first monitoring image from a spatial domain to a frequency domain using a fast fourier transform;
filtering low-frequency components and high-frequency components in a frequency domain corresponding to the first monitoring image by using a ButterWorth band-pass filter;
and restoring the filtered first monitoring image from a frequency domain to a space domain by adopting inverse fast Fourier transform to obtain a second monitoring image after interference removal.
Optionally, in a second implementation manner of the first aspect of the present invention, before the acquiring a first monitoring image of a target monitoring area, and performing noise and light filtering on the first monitoring image to obtain a second monitoring image after interference removal, the method further includes:
shooting a plurality of sample images of a target monitoring area, wherein the sample images contain pedestrians and placed foreign matters;
sequentially carrying out pedestrian detection and foreign matter detection on each sample image, and outputting a pedestrian detection frame and a foreign matter detection frame;
combining the pedestrian detection frame and the foreign matter detection frame corresponding to each sample image;
and taking the external rectangular coordinates corresponding to the combined pedestrian detection frame as a coordinate calibration frame corresponding to the pedestrian detection area, and taking the external rectangular coordinates corresponding to the combined foreign matter detection frame as a coordinate calibration frame corresponding to the foreign matter detection area.
Optionally, in a third implementation manner of the first aspect of the present invention, the pedestrian detection model sequentially includes: fast SE-Resnet18 network, multilayer convolution layer, pooling layer, full connection layer and SoftMax layer, the inputting of the second monitoring image and the pedestrian detection area into a preset pedestrian detection model for pedestrian recognition, and the outputting of the result of pedestrian recognition comprises:
inputting the second monitoring image and the pedestrian detection area into a Fast SE-Resnet18 network in the pedestrian detection model for feature extraction, and outputting a plurality of first feature maps of the second monitoring image in the pedestrian detection area;
inputting each first feature map into a multilayer convolution layer in the pedestrian detection model to perform multi-round convolution operation to obtain a plurality of first feature matrixes corresponding to each first feature map;
inputting each first feature matrix into a pooling layer in the pedestrian detection model for down-sampling and feature compression, and outputting a plurality of second feature matrices;
inputting each second feature matrix into a full-connection layer in the pedestrian detection model for feature combination to obtain a plurality of third feature matrices;
and inputting each third feature matrix into a SoftMax layer in the pedestrian detection model for feature classification, and outputting a pedestrian recognition result.
Optionally, in a fourth implementation manner of the first aspect of the present invention, the foreign object localization model includes: an encoder consisting of a Fast SE-Resnet18 network and a decoder comprising in sequence: pooling layer, multilayer anti-convolution layer, SoftMax layer, if pedestrian's identification result is for there being the pedestrian, then will the second monitoring image with foreign matter location model is preset in the foreign matter detection area input carries out the foreign matter location, and output foreign matter location result includes:
if the pedestrian recognition result is that a pedestrian exists, inputting the second monitoring image and the foreign object detection area into a Fast SE-Resnet18 network in the foreign object positioning model for feature extraction, and outputting a plurality of first segmentation feature maps of the second monitoring image in the foreign object detection area;
inputting each first segmentation feature map into a pooling layer in the foreign matter positioning model for up-sampling and feature compression, and outputting a plurality of second segmentation feature maps;
inputting each second segmentation characteristic diagram into a multilayer deconvolution layer in the foreign matter positioning model to perform deconvolution operation, so as to obtain a plurality of segmentation characteristic matrixes corresponding to each second segmentation characteristic diagram;
and inputting each segmentation characteristic matrix into a SoftMax layer in the foreign matter positioning model for characteristic classification, outputting probability values of each segmentation image in the second monitoring image as a background, positioning the segmentation image where the foreign matter is located according to the probability values, and outputting the segmentation image as a foreign matter positioning result.
Optionally, in a fifth implementation manner of the first aspect of the present invention, the foreign object classification model sequentially includes: if the foreign matter positioning result is not empty, inputting a positioned foreign matter circumscribed rectangle image into a preset foreign matter classification model for article classification, and outputting probability values of target foreign matters corresponding to different article names respectively comprises the following steps:
if the foreign matter positioning result is not empty, inputting the positioned foreign matter circumscribed rectangular image into the SE-ResNet34 network for feature extraction, and outputting a plurality of second feature maps of the foreign matter circumscribed rectangular image;
inputting each second feature map into a multilayer convolution layer in the foreign matter classification model to carry out convolution operation, and obtaining a plurality of fourth feature matrixes corresponding to each second feature map;
inputting the fourth feature matrixes into a pooling layer in the foreign matter classification model for down-sampling and feature compression, and outputting a plurality of fifth feature matrixes;
inputting each fifth feature matrix into a full-connection layer in the foreign matter classification model for feature combination to obtain a plurality of sixth feature matrices;
and inputting the sixth feature matrixes into a SoftMax layer in the foreign matter classification model for feature classification, and outputting probability values of the target foreign matters corresponding to different article names respectively.
A second aspect of the present invention provides a public area foreign matter detection apparatus including:
the interference removing module is used for acquiring a first monitoring image of a target monitoring area and filtering noise and light of the first monitoring image to obtain a second monitoring image after interference removal;
the area calibration module is used for calibrating a pedestrian detection area and a foreign matter detection area in the second monitoring image based on a preset coordinate calibration frame;
the pedestrian recognition module is used for inputting the second monitoring image and the pedestrian detection area into a preset pedestrian detection model for pedestrian recognition and outputting a pedestrian recognition result;
the foreign matter positioning module is used for inputting the second monitoring image and the foreign matter detection area into a preset foreign matter positioning model for foreign matter positioning and outputting a foreign matter positioning result if the pedestrian identification result indicates that a pedestrian exists;
the article classification module is used for inputting the positioned external rectangular image of the foreign matter into a preset foreign matter classification model for article classification if the foreign matter positioning result is not empty, and outputting probability values of the target foreign matter corresponding to different article names respectively;
and the foreign matter determining module is used for selecting the article name corresponding to the maximum probability value as the article name of the target foreign matter and carrying out foreign matter early warning on the target foreign matter.
Optionally, in a first implementation manner of the second aspect of the present invention, the common area foreign object detection apparatus further includes:
the calibration frame selection module is used for shooting a plurality of sample images of a target monitoring area, wherein the sample images contain pedestrians and placed foreign matters; sequentially carrying out pedestrian detection and foreign matter detection on each sample image, and outputting a pedestrian detection frame and a foreign matter detection frame; combining the pedestrian detection frame and the foreign matter detection frame corresponding to each sample image; and taking the external rectangular coordinates corresponding to the combined pedestrian detection frame as a coordinate calibration frame corresponding to the pedestrian detection area, and taking the external rectangular coordinates corresponding to the combined foreign matter detection frame as a coordinate calibration frame corresponding to the foreign matter detection area.
Optionally, in a second implementation manner of the second aspect of the present invention, the interference removing module includes:
the acquisition unit is used for acquiring a first monitoring image of a target monitoring area;
an interference removing unit, configured to convert the first monitoring image from a spatial domain to a frequency domain by using a fast fourier transform; filtering low-frequency components and high-frequency components in a frequency domain corresponding to the first monitoring image by using a ButterWorth band-pass filter; and restoring the filtered first monitoring image from a frequency domain to a space domain by adopting inverse fast Fourier transform to obtain a second monitoring image after interference removal.
Optionally, in a third implementation manner of the second aspect of the present invention, the pedestrian detection model sequentially includes: fast SE-Resnet18 network, multilayer convolutional layer, pooling layer, full connectivity layer, and SoftMax layer, the pedestrian identification module is specifically configured to:
inputting the second monitoring image and the pedestrian detection area into a Fast SE-Resnet18 network in the pedestrian detection model for feature extraction, and outputting a plurality of first feature maps of the second monitoring image in the pedestrian detection area; inputting each first feature map into a multilayer convolution layer in the pedestrian detection model to perform multi-round convolution operation to obtain a plurality of first feature matrixes corresponding to each first feature map; inputting each first feature matrix into a pooling layer in the pedestrian detection model for down-sampling and feature compression, and outputting a plurality of second feature matrices; inputting each second feature matrix into a full-connection layer in the pedestrian detection model for feature combination to obtain a plurality of third feature matrices; and inputting each third feature matrix into a SoftMax layer in the pedestrian detection model for feature classification, and outputting a pedestrian recognition result.
Optionally, in a fourth implementation manner of the second aspect of the present invention, the foreign object positioning model includes: an encoder consisting of a Fast SE-Resnet18 network and a decoder comprising in sequence: pooling layer, multilayer deconvolution layer, SoftMax layer, foreign matter orientation module specifically is used for:
if the pedestrian recognition result is that a pedestrian exists, inputting the second monitoring image and the foreign object detection area into a Fast SE-Resnet18 network in the foreign object positioning model for feature extraction, and outputting a plurality of first segmentation feature maps of the second monitoring image in the foreign object detection area; inputting each first segmentation feature map into a pooling layer in the foreign matter positioning model for up-sampling and feature compression, and outputting a plurality of second segmentation feature maps; inputting each second segmentation characteristic diagram into a multilayer deconvolution layer in the foreign matter positioning model to perform deconvolution operation, so as to obtain a plurality of segmentation characteristic matrixes corresponding to each second segmentation characteristic diagram; and inputting each segmentation characteristic matrix into a SoftMax layer in the foreign matter positioning model for characteristic classification, outputting probability values of each segmentation image in the second monitoring image as a background, positioning the segmentation image where the foreign matter is located according to the probability values, and outputting the segmentation image as a foreign matter positioning result.
Optionally, in a fourth implementation manner of the second aspect of the present invention, the foreign object classification model sequentially includes: a SE-ResNet34 network, a multilayer convolutional layer, a pooling layer, a fully-connected layer, and a SoftMax layer, the item classification module being specifically configured to:
if the foreign matter positioning result is not empty, inputting the positioned foreign matter circumscribed rectangular image into the SE-ResNet34 network for feature extraction, and outputting a plurality of second feature maps of the foreign matter circumscribed rectangular image; inputting each second feature map into a multilayer convolution layer in the foreign matter classification model to carry out convolution operation, and obtaining a plurality of fourth feature matrixes corresponding to each second feature map; inputting the fourth feature matrixes into a pooling layer in the foreign matter classification model for down-sampling and feature compression, and outputting a plurality of fifth feature matrixes; inputting each fifth feature matrix into a full-connection layer in the foreign matter classification model for feature combination to obtain a plurality of sixth feature matrices; and inputting the sixth feature matrixes into a SoftMax layer in the foreign matter classification model for feature classification, and outputting probability values of the target foreign matters corresponding to different article names respectively.
A third aspect of the present invention provides a public area foreign matter detection apparatus comprising: a memory and at least one processor, the memory having instructions stored therein; the at least one processor invokes the instructions in the memory to cause the common area foreign object detection apparatus to perform the common area foreign object detection method described above.
A fourth aspect of the present invention provides a computer-readable storage medium having stored therein instructions, which when run on a computer, cause the computer to execute the common area foreign matter detection method described above.
According to the technical scheme, after an original monitoring image is collected, the original image is transferred to a frequency domain, a band-pass filter is used for filtering high-frequency noise and low-frequency light interference, then the filtered frequency domain image is transferred to a space domain, a new image with interference removed is obtained, pedestrian identification is carried out, when a pedestrian is determined to exist, foreign matter positioning is carried out on the new image, foreign matter different from a background image is obtained through a segmentation network, and finally foreign matter classification is carried out through a classification network to obtain the class of an article to which the foreign matter belongs. The invention can recognize foreign matters in an anti-interference way, improves the recognition precision of the foreign matters, simultaneously automatically recognizes the category of the foreign matters and is convenient for workers to quickly process.
Drawings
FIG. 1 is a schematic diagram of a public area foreign object detection method according to a first embodiment of the present invention;
FIG. 2 is a diagram of a method for detecting a foreign object in a public area according to a second embodiment of the present invention;
FIG. 3 is a schematic view of a first embodiment of a public area foreign matter detection apparatus according to an embodiment of the present invention;
FIG. 4 is a schematic view of a second embodiment of a public area foreign object detection apparatus according to an embodiment of the present invention;
fig. 5 is a schematic diagram of an embodiment of the foreign object detection apparatus in the public area in the embodiment of the present invention.
Detailed Description
The embodiment of the invention provides a method, a device and equipment for detecting foreign matters in a public area and a storage medium. The terms "first," "second," "third," "fourth," and the like in the description and in the claims, as well as in the drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be practiced otherwise than as specifically illustrated or described herein. Furthermore, the terms "comprises," "comprising," or "having," and any variations thereof, are intended to cover non-exclusive inclusions, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
For convenience of understanding, a specific flow of an embodiment of the present invention is described below, and referring to fig. 1, a first embodiment of a method for detecting a foreign object in a public area according to an embodiment of the present invention includes:
101. acquiring a first monitoring image of a target monitoring area, and filtering noise and light of the first monitoring image to obtain a second monitoring image after interference is removed;
it is to be understood that the implementation subject of the present invention may be a public area foreign object detection apparatus, and may also be a terminal or a server, which is not limited herein. The embodiment of the present invention is described by taking a server as an execution subject.
In this embodiment, the camera is used for shooting the target monitoring area to form a monitoring image, and only one frame of image needs to be collected for detection when detection is performed, so that the target monitoring area is not limited, and can be a business hall or a rest room. The foreign object in this embodiment refers to an article that does not belong to the current monitored area environment originally, and specifically refers to an article forgotten by others in a specific application scenario, such as an umbrella, a cup, a backpack, and a handbag.
Optionally, in an embodiment, the noise and light filtering is performed as follows:
s1, converting the first monitoring image from a space domain to a frequency domain by adopting fast Fourier transform;
s2, filtering low-frequency components and high-frequency components in a frequency domain corresponding to the first monitoring image by using a ButterWorth band-pass filter;
and S3, restoring the filtered first monitoring image from the frequency domain to the space domain by adopting inverse fast Fourier transform to obtain a second monitoring image after interference removal.
In this optional embodiment, the original image is first converted from the spatial domain to the frequency domain, and specifically, the spatial domain is converted to the frequency domain by using the fast fourier transform. The light interference such as the ambient light is analyzed to belong to low-frequency components, and the noise belongs to high-frequency components, so that the low-frequency components and the high-frequency components are filtered by using Butterworth band-pass filtering, and the interference such as the noise, the light and the like in the image can be filtered by using a band-pass filter. And finally, recovering the frequency domain image with the ambient light filtered out to a space domain from the frequency domain through inverse fast Fourier transform to obtain a new RGB image.
102. Calibrating a pedestrian detection area and a foreign matter detection area in the second monitoring image based on a preset coordinate calibration frame;
in this embodiment, for realizing pedestrian detection and foreign matter detection fast, reduce detection achievement volume, promote detection efficiency and rate of accuracy, consequently, before detecting, mark earlier and wait to detect the region, specifically include:
pedestrian detection area
The pedestrian detection is added to avoid the occurrence of erroneous determination. The article can be judged as the foreign matter only under the condition that the article is not watched by people, if a client is in the target detection area, the client possibly carries out business or waits, and the article is watched by people at the moment, so that the article is in a normal state and the foreign matter early warning cannot be generated.
(II) foreign matter detection region
Usually, the camera is fixed to be set up in high altitude position, and the field of vision that it was shot covers whole monitoring area, and may still have other article (non-foreign matter) in the monitoring area, if carry out the foreign matter detection to all monitoring areas and must discern multiple article and still need carry out the foreign matter to the article that discern and distinguish, this not only has increased the complexity of algorithm, has also reduced detection efficiency and foreign matter discernment degree of accuracy simultaneously, consequently, needs further to narrow detection range, also marks the foreign matter detection area promptly.
Optionally, in an embodiment, the coordinate calibration frame corresponding to the pedestrian detection area and the foreign object detection area is obtained in the following manner:
s1, shooting a plurality of sample images of the target monitoring area, wherein the sample images contain pedestrians and placed foreign matters;
s2, sequentially carrying out pedestrian detection and foreign matter detection on each sample image, and outputting a pedestrian detection frame and a foreign matter detection frame;
s3, combining the pedestrian detection frame and the foreign matter detection frame corresponding to each sample image;
and S4, taking the external rectangular coordinates corresponding to the combined pedestrian detection frame as a coordinate calibration frame corresponding to the pedestrian detection area, and taking the external rectangular coordinates corresponding to the combined foreign matter detection frame as a coordinate calibration frame corresponding to the foreign matter detection area.
In this optional embodiment, since the camera is fixed in position and the shooting angle of view is also fixed, and the position of the fixed object in the monitoring area does not change, the position coordinates of the fixed object in any picture shot by the same camera are the same, and therefore, the coordinate calibration frame obtained in the above manner can be used for calibrating the pedestrian detection area and the foreign object detection area.
103. Inputting the second monitoring image and the pedestrian detection area into a preset pedestrian detection model for pedestrian identification, and outputting a pedestrian identification result;
in the present embodiment, a pedestrian detection model is generated by training in advance, and the model can detect a pedestrian in a specified region in an input image, and may be for a walking person or a sitting or standing person.
A conventional SSD model (Single Shot multi box Detector) uses VGG16 infrastructure to extract features of an image. The VGG16 base network is time consuming in extracting features and lacks the feature of attention mechanism, so that the neural network can pay attention to the unimportant part, thereby generating errors. The present embodiment is therefore trained using a custom convolutional neural network model that preferably uses the Fast SE-Resnet18 network instead of the SSD's underlying network VGG 16.
The embodiment only identifies the pedestrian of the image in the calibrated pedestrian detection area in the monitoring image (namely the second monitoring image), thereby reducing the calculation amount and improving the speed and the detection accuracy of the pedestrian identification.
104. If the pedestrian identification result is that a pedestrian exists, inputting the second monitoring image and the foreign matter detection area into a preset foreign matter positioning model for foreign matter positioning, and outputting a foreign matter positioning result;
in this embodiment, if the pedestrian detection model identifies that a pedestrian exists in the current target monitoring area, the object existing in the current environment may belong to all of the pedestrian and not to a foreign object, so that foreign object detection is not required, and the next frame of monitoring image is directly processed. And if no pedestrian exists in the current target monitoring area, the foreign matter detection can be carried out.
The network structure of the traditional image segmentation model SegNet model is complex, and the recognition speed and the recognition accuracy are not ideal, so the embodiment creates the defects of the SegNet model, uses the self-defined convolutional neural network model for training, and preferably uses the self-defined Fast SE-Resnet18 network to replace the basic network of SegNet.
In this embodiment, the foreign object location model further outputs the position of the foreign object in the monitoring image through the circumscribed rectangle frame while identifying the foreign object in the image. According to the embodiment, the foreign matter identification is only carried out on the image in the foreign matter detection area calibrated in the monitoring image (namely the second monitoring image), the calculated amount is reduced, and the foreign matter identification speed and the detection accuracy are improved.
105. If the foreign matter positioning result is not empty, inputting the positioned foreign matter circumscribed rectangular image into a preset foreign matter classification model for article classification, and outputting probability values of the target foreign matter corresponding to different article names respectively;
in this embodiment, if the foreign object location result is not empty, that is, when a foreign object in the monitored image is located, the located foreign object circumscribed rectangular image is further input to the preset foreign object classification model to perform object classification, and a probability value of the foreign object belonging to various objects is given. And if the foreign matter positioning result is empty, the next frame of monitoring image is continuously processed.
106. And selecting the article name corresponding to the maximum probability value as the article name of the target foreign matter, and carrying out foreign matter early warning on the target foreign matter.
In this embodiment, after the foreign object in the target monitoring area is identified, the foreign object early warning is further performed on the relevant monitoring personnel, for example, the target monitoring area is reminded of some object, so that the worker can timely handle the foreign object, for example, the object may be left by the customer and fall down, and the worker can timely remind the customer.
In this embodiment, after the monitoring original image is collected, the original image is first transferred to a frequency domain, a band-pass filter is used to filter high-frequency noise and low-frequency light interference, then the filtered frequency domain image is converted to a spatial domain to obtain a new image after interference removal, then pedestrian identification is performed, when it is determined that a pedestrian exists, the new image is subjected to foreign object positioning, a segmentation network is used to obtain a foreign object different from a background image, and finally, a classification network is performed on the foreign object to obtain an object type to which the foreign object belongs. This embodiment can anti-interference discernment foreign matter, has promoted the foreign matter discernment precision, and automatic identification foreign matter belongs to the classification simultaneously, and the staff of being convenient for handles fast.
Referring to fig. 2, a second embodiment of the method for detecting a foreign object in a public area according to the embodiment of the present invention includes:
201. acquiring a first monitoring image of a target monitoring area, and filtering noise and light of the first monitoring image to obtain a second monitoring image after interference is removed;
202. calibrating a pedestrian detection area and a foreign matter detection area in the second monitoring image based on a preset coordinate calibration frame;
203. inputting the second monitoring image and the pedestrian detection area into a Fast SE-Resnet18 network in a pedestrian detection model for feature extraction, and outputting a plurality of first feature maps of the second monitoring image in the pedestrian detection area;
204. inputting each first feature map into a multilayer convolution layer in the pedestrian detection model to perform multi-round convolution operation to obtain a plurality of first feature matrixes corresponding to each first feature map;
205. inputting each first feature matrix into a pooling layer in the pedestrian detection model for down-sampling and feature compression, and outputting a plurality of second feature matrices;
206. inputting each second feature matrix into a full-connection layer in the pedestrian detection model for feature combination to obtain a plurality of third feature matrices;
207. inputting each third feature matrix into a SoftMax layer in the pedestrian detection model for feature classification, and outputting a pedestrian recognition result;
in this embodiment, the pedestrian detection model is composed of a Fast SE-Resnet18 network, a multilayer convolutional layer, a pooling layer, a full link layer, and a SoftMax layer. The present embodiment is trained using a custom convolutional neural network model that preferably uses a Fast SE-Resnet18 network instead of the SSD's underlying network VGG 16.
The Fast SE-Resnet18 network of the present embodiment is mainly formed by sequentially connecting the following 4 parts up and down, each part includes two identical convolution kernels superimposed up and down, and the specific structure is as follows:
Figure BDA0002853970340000111
Figure BDA0002853970340000112
each part in the Fast SE-Resnet18 network is used for carrying out feature superposition, and the superposition mechanism specifically comprises the following steps: the overlap is once in the lower subsection of each section and the upper subsection adds a SE _ Block structure, thus allowing the Fast SE-respet 18 network to automatically focus on fine critical information. And the Fast SE-Resnet18 network adopts a dense connection strategy to perform feature fusion, and dense connection is that connection points on the upper layer are overlapped and fused with connection points of each chain on the lower layer.
In the embodiment, a Fast SE-Resnet18 network is used as a basic network of a pedestrian detection model to perform initial feature extraction, and a plurality of convolutional layers, a pooling layer, a full connection layer and a SoftMax layer are further added, wherein the feature extraction speed can be increased by using the Fast SE-Resnet18 network to perform feature extraction, the feature information amount is enriched, the identification accuracy is improved, a plurality of feature maps of a monitoring image in a pedestrian detection area are output, then the feature maps output by the Fast SE-Resnet18 network are further input into the plurality of convolutional layers to perform multi-round convolution operation to obtain a feature matrix corresponding to the feature maps, then the feature matrix is input into the pooling layer to perform down sampling and feature compression to increase the feature information amount of the feature matrix, finally the processed feature matrix is input into the full connection layer to perform feature combination, and the combined feature matrix is input into the SoftMax layer to perform feature classification, thereby obtaining a pedestrian recognition result.
208. If the pedestrian recognition result is that a pedestrian exists, inputting the second monitoring image and the foreign object detection area into a Fast SE-Resnet18 network in the foreign object positioning model for feature extraction, and outputting a plurality of first segmentation feature maps of the second monitoring image in the foreign object detection area;
209. inputting each first segmentation feature map into a pooling layer in the foreign matter positioning model for up-sampling and feature compression, and outputting a plurality of second segmentation feature maps;
210. inputting each second segmentation characteristic diagram into a multilayer deconvolution layer in the foreign matter positioning model to perform deconvolution operation, so as to obtain a plurality of segmentation characteristic matrixes corresponding to each second segmentation characteristic diagram;
211. inputting each segmentation feature matrix into a SoftMax layer in the foreign matter positioning model for feature classification, outputting probability values of each segmentation image in the second monitoring image as a background, positioning a segmentation image where a foreign matter is located according to the probability values, and outputting the segmentation image as a foreign matter positioning result;
in this embodiment, the foreign object positioning model includes in order: an encoder and a decoder, the encoder being formed by a Fast SE-Resnet18 network, the decoder comprising in sequence: the embodiment provides a pooling layer, a multilayer deconvolution layer and a SoftMax layer, which are innovative for the defects of a SegNet model, and a user-defined convolutional neural network model is used for training, and the user-defined Fast SE-Resnet18 network is preferably used for replacing a basic network of SegNet in the convolutional neural network model so as to improve the speed and accuracy of the segmentation model. Next, foreign matter detection is performed using an image of 224 × 224.
Firstly, inputting 224 × 224 images into a Fast SE-Resnet18 network, wherein the size of a feature map in a convolution layer conv5_ x is 7 × 7, then performing two convolutions, performing up-sampling to expand the feature map to 14 × 14, performing deconvolution operation through a plurality of deconvolution layers to further expand the size of the feature map to 56 × 56, and finally directly inputting a SoftMax layer for feature classification to give probability values of each segmented image as a background, for example, black is a background, white is a foreground, and a white area is a foreign matter positioned from a monitored image.
In this embodiment, the Fast SE-Resnet18 network is used as an encoder of the foreign object localization model to perform preliminary feature extraction, and a decoder is further provided to process the extracted features, which specifically includes: one pooling layer, multiple deconvolution layers and one SoftMax layer, wherein feature extraction speed can be increased by using Fast SE-Resnet18 network for feature extraction, feature information amount is enriched, identification accuracy is improved, and multiple segmentation feature maps of the monitored image in the foreign matter detection area are output, then, the segmentation characteristic diagram output by the Fast SE-Resnet18 network is further input into a pooling layer for up-sampling and characteristic compression, the characteristic information quantity of the segmentation characteristic diagram is improved, finally, the processed segmentation characteristic diagram is input into a multilayer deconvolution layer for deconvolution operation to obtain a corresponding segmentation characteristic matrix, finally, the segmentation characteristic matrix is input into a SoftMax layer for characteristic classification, the probability value of each segmentation image in the monitoring image as the background is output, and positioning the segmentation image where the foreign matter is located according to the probability values and outputting the segmentation image as a foreign matter positioning result.
212. If the foreign matter positioning result is not empty, inputting the positioned foreign matter circumscribed rectangular image into a preset foreign matter classification model for article classification, and outputting probability values of the target foreign matter corresponding to different article names respectively;
in this embodiment, the foreign object classification model includes in order: SE-ResNet34 network, multilayer convolutional layer, pooling layer, full connectivity layer, and SoftMax layer. According to the actual conditions of the monitoring environment and the monitoring scene, the category of the foreign matters is set to be common categories in advance, such as books, bottles, mobile phones, computers, small fans, purses, masks, pens, tissues, umbrellas, keys, bags, clothes and the like. The foreign object classification model of the embodiment is used for identifying the probability that the foreign object in the monitored image is the article.
The embodiment improves the ResNet34 network, proposes that a SE-ResNet34 network is used as a basic network of a foreign matter classification model for feature extraction, and mainly comprises the following 4 parts which are sequentially connected up and down, wherein each part comprises a plurality of same convolution kernels which are superposed up and down, and the specific structure is as follows:
Figure BDA0002853970340000131
Figure BDA0002853970340000132
each part in the SE-ResNet34 network is used for carrying out feature superposition, and the superposition mechanism specifically comprises the following steps: superimposed once in the lower subsection of each section and the upper subsection adds a SE _ Block structure, thus enabling the SE-ResNet34 network to automatically focus on fine key information. And the SE-ResNet34 network also adopts a dense connection strategy to perform feature fusion, wherein the dense connection is the superposition fusion of the connection points of the upper layer and each link connection point of the lower layer.
Optionally, in an embodiment, the processing procedure of article classification is as follows:
if the foreign matter positioning result is not empty, inputting the positioned foreign matter circumscribed rectangular image into the SE-ResNet34 network for feature extraction, and outputting a plurality of second feature maps of the foreign matter circumscribed rectangular image;
inputting each second feature map into a multilayer convolution layer in the foreign matter classification model to carry out convolution operation, and obtaining a plurality of fourth feature matrixes corresponding to each second feature map;
inputting the fourth feature matrixes into a pooling layer in the foreign matter classification model for down-sampling and feature compression, and outputting a plurality of fifth feature matrixes;
inputting each fifth feature matrix into a full-connection layer in the foreign matter classification model for feature combination to obtain a plurality of sixth feature matrices;
and inputting the sixth feature matrixes into a SoftMax layer in the foreign matter classification model for feature classification, and outputting probability values of the target foreign matters corresponding to different article names respectively.
In this optional embodiment, the located foreign matter external rectangular image is scaled to a specified size, for example, 224 × 224, and then input to an SE-ResNet34 network for feature extraction to obtain a feature map of the foreign matter external rectangular image, then the extracted feature map is input to a multilayer convolution layer for convolution operation to obtain a feature matrix corresponding to the feature map, then the feature matrix is input to a pooling layer for down-sampling and feature compression to improve feature information amount of the feature matrix, finally the processed feature matrix is input to a full-connection layer for feature combination, and the combined feature matrix is input to a softlayer max for feature classification, so that probability that the foreign matter is a specified article category is obtained.
213. And selecting the article name corresponding to the maximum probability value as the article name of the target foreign matter, and carrying out foreign matter early warning on the target foreign matter.
The embodiment improves the existing target recognition and image segmentation model, greatly improves the image feature extraction speed, enriches the feature information quantity, improves the recognition accuracy, can also recognize foreign matters in an anti-interference manner and automatically recognize the belonged category of the foreign matters, and is convenient for workers to quickly process.
In the above description of the method for detecting the foreign matters in the public area in the embodiment of the present invention, referring to fig. 3, a device for detecting the foreign matters in the public area in the embodiment of the present invention is described as follows, where the first embodiment of the device for detecting the foreign matters in the public area in the embodiment of the present invention includes:
the interference removing module 301 is configured to collect a first monitoring image of a target monitoring area, and perform noise and light filtering on the first monitoring image to obtain a second monitoring image after interference removal;
the region calibration module 302 is configured to calibrate a pedestrian detection region and a foreign object detection region in the second monitoring image based on a preset coordinate calibration frame;
the pedestrian recognition module 303 is configured to input the second monitoring image and the pedestrian detection area into a preset pedestrian detection model to perform pedestrian recognition, and output a pedestrian recognition result;
a foreign object positioning module 304, configured to input the second monitoring image and the foreign object detection area into a preset foreign object positioning model for foreign object positioning if the pedestrian identification result indicates that a pedestrian exists, and output a foreign object positioning result;
an article classification module 305, configured to, if the foreign object location result is not empty, input a located external rectangular image of the foreign object into a preset foreign object classification model for article classification, and output probability values that target foreign objects correspond to different article names respectively;
and the foreign matter determining module 306 is configured to select an article name corresponding to the maximum probability value as the article name of the target foreign matter, and perform foreign matter early warning on the target foreign matter.
In this embodiment, after the monitoring original image is collected, the original image is first transferred to a frequency domain, a band-pass filter is used to filter high-frequency noise and low-frequency light interference, then the filtered frequency domain image is converted to a spatial domain to obtain a new image after interference removal, then pedestrian identification is performed, when it is determined that a pedestrian exists, the new image is subjected to foreign object positioning, a segmentation network is used to obtain a foreign object different from a background image, and finally, a classification network is performed on the foreign object to obtain an object type to which the foreign object belongs. This embodiment can anti-interference discernment foreign matter, has promoted the foreign matter discernment precision, and automatic identification foreign matter belongs to the classification simultaneously, and the staff of being convenient for handles fast.
Referring to fig. 4, a second embodiment of the apparatus for detecting foreign objects in a public area according to the present invention includes:
the interference removing module 301 is configured to collect a first monitoring image of a target monitoring area, and perform noise and light filtering on the first monitoring image to obtain a second monitoring image after interference removal;
the region calibration module 302 is configured to calibrate a pedestrian detection region and a foreign object detection region in the second monitoring image based on a preset coordinate calibration frame;
the pedestrian recognition module 303 is configured to input the second monitoring image and the pedestrian detection area into a preset pedestrian detection model to perform pedestrian recognition, and output a pedestrian recognition result;
a foreign object positioning module 304, configured to input the second monitoring image and the foreign object detection area into a preset foreign object positioning model for foreign object positioning if the pedestrian identification result indicates that a pedestrian exists, and output a foreign object positioning result;
an article classification module 305, configured to, if the foreign object location result is not empty, input a located external rectangular image of the foreign object into a preset foreign object classification model for article classification, and output probability values that target foreign objects correspond to different article names respectively;
the foreign matter determining module 306 is configured to select an article name corresponding to the maximum probability value as the article name of the target foreign matter, and perform foreign matter early warning on the target foreign matter;
a calibration frame selecting module 307, configured to capture multiple sample images of a target monitoring area, where the sample images include pedestrians and placed foreign objects; sequentially carrying out pedestrian detection and foreign matter detection on each sample image, and outputting a pedestrian detection frame and a foreign matter detection frame; combining the pedestrian detection frame and the foreign matter detection frame corresponding to each sample image; and taking the external rectangular coordinates corresponding to the combined pedestrian detection frame as a coordinate calibration frame corresponding to the pedestrian detection area, and taking the external rectangular coordinates corresponding to the combined foreign matter detection frame as a coordinate calibration frame corresponding to the foreign matter detection area.
Optionally, in an embodiment, the interference removing module 301 includes:
the acquisition unit is used for acquiring a first monitoring image of a target monitoring area;
an interference removing unit, configured to convert the first monitoring image from a spatial domain to a frequency domain by using a fast fourier transform; filtering low-frequency components and high-frequency components in a frequency domain corresponding to the first monitoring image by using a ButterWorth band-pass filter; and restoring the filtered first monitoring image from a frequency domain to a space domain by adopting inverse fast Fourier transform to obtain a second monitoring image after interference removal.
Optionally, in an embodiment, the pedestrian detection model sequentially includes: a Fast SE-Resnet18 network, a multilayer convolutional layer, a pooling layer, a full link layer, and a SoftMax layer, where the pedestrian identification module 303 is specifically configured to:
inputting the second monitoring image and the pedestrian detection area into a Fast SE-Resnet18 network in the pedestrian detection model for feature extraction, and outputting a plurality of first feature maps of the second monitoring image in the pedestrian detection area; inputting each first feature map into a multilayer convolution layer in the pedestrian detection model to perform multi-round convolution operation to obtain a plurality of first feature matrixes corresponding to each first feature map; inputting each first feature matrix into a pooling layer in the pedestrian detection model for down-sampling and feature compression, and outputting a plurality of second feature matrices; inputting each second feature matrix into a full-connection layer in the pedestrian detection model for feature combination to obtain a plurality of third feature matrices; and inputting each third feature matrix into a SoftMax layer in the pedestrian detection model for feature classification, and outputting a pedestrian recognition result.
Optionally, in an embodiment, the foreign object localization model includes: an encoder consisting of a Fast SE-Resnet18 network and a decoder comprising in sequence: pooling layer, multilayer deconvolution layer, SoftMax layer, foreign matter location module 304 is specifically used for:
if the pedestrian recognition result is that a pedestrian exists, inputting the second monitoring image and the foreign object detection area into a Fast SE-Resnet18 network in the foreign object positioning model for feature extraction, and outputting a plurality of first segmentation feature maps of the second monitoring image in the foreign object detection area; inputting each first segmentation feature map into a pooling layer in the foreign matter positioning model for up-sampling and feature compression, and outputting a plurality of second segmentation feature maps; inputting each second segmentation characteristic diagram into a multilayer deconvolution layer in the foreign matter positioning model to perform deconvolution operation, so as to obtain a plurality of segmentation characteristic matrixes corresponding to each second segmentation characteristic diagram; and inputting each segmentation characteristic matrix into a SoftMax layer in the foreign matter positioning model for characteristic classification, outputting probability values of each segmentation image in the second monitoring image as a background, positioning the segmentation image where the foreign matter is located according to the probability values, and outputting the segmentation image as a foreign matter positioning result.
Optionally, in an embodiment, the foreign object classification model sequentially includes: SE-ResNet34 network, multilayer convolutional layer, pooling layer, full-link layer, and SoftMax layer, the item classification module 305 is specifically configured to:
if the foreign matter positioning result is not empty, inputting the positioned foreign matter circumscribed rectangular image into the SE-ResNet34 network for feature extraction, and outputting a plurality of second feature maps of the foreign matter circumscribed rectangular image; inputting each second feature map into a multilayer convolution layer in the foreign matter classification model to carry out convolution operation, and obtaining a plurality of fourth feature matrixes corresponding to each second feature map; inputting the fourth feature matrixes into a pooling layer in the foreign matter classification model for down-sampling and feature compression, and outputting a plurality of fifth feature matrixes; inputting each fifth feature matrix into a full-connection layer in the foreign matter classification model for feature combination to obtain a plurality of sixth feature matrices; and inputting the sixth feature matrixes into a SoftMax layer in the foreign matter classification model for feature classification, and outputting probability values of the target foreign matters corresponding to different article names respectively.
The embodiment improves the existing target recognition and image segmentation model, greatly improves the image feature extraction speed, enriches the feature information quantity, improves the recognition accuracy, can also recognize foreign matters in an anti-interference manner and automatically recognize the belonged category of the foreign matters, and is convenient for workers to quickly process.
Fig. 3 and 4 describe the common area foreign matter detection apparatus in the embodiment of the present invention in detail from the perspective of the modular functional entity, and the common area foreign matter detection apparatus in the embodiment of the present invention is described in detail from the perspective of hardware processing.
Fig. 5 is a schematic structural diagram of a public area foreign object detection apparatus according to an embodiment of the present invention, where the public area foreign object detection apparatus 500 may have a relatively large difference due to different configurations or performances, and may include one or more processors (CPUs) 510 (e.g., one or more processors) and a memory 520, and one or more storage media 530 (e.g., one or more mass storage devices) storing applications 533 or data 532. Memory 520 and storage media 530 may be, among other things, transient or persistent storage. The program stored on the storage medium 530 may include one or more modules (not shown), each of which may include a series of instruction operations for the common area foreign object detection apparatus 500. Still further, the processor 510 may be configured to communicate with the storage medium 530 to perform a series of instruction operations in the storage medium 530 on the public area foreign object detection apparatus 500.
Public area foreign object detection device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input-output interfaces 560, and/or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that the common area foreign object detection apparatus configuration shown in fig. 5 does not constitute a limitation of the common area foreign object detection apparatus, and may include more or fewer components than those shown, or some components in combination, or a different arrangement of components.
The invention also provides public area foreign matter detection equipment, which comprises a memory and a processor, wherein computer readable instructions are stored in the memory, and when being executed by the processor, the computer readable instructions cause the processor to execute the steps of the public area foreign matter detection method in the embodiments.
The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium, and which may also be a volatile computer-readable storage medium, having stored therein instructions that, when run on a computer, cause the computer to perform the steps of the public area foreign matter detection method.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims (10)

1. A common area foreign matter detection method, characterized by comprising:
acquiring a first monitoring image of a target monitoring area, and filtering noise and light of the first monitoring image to obtain a second monitoring image after interference is removed;
calibrating a pedestrian detection area and a foreign matter detection area in the second monitoring image based on a preset coordinate calibration frame;
inputting the second monitoring image and the pedestrian detection area into a preset pedestrian detection model for pedestrian identification, and outputting a pedestrian identification result;
if the pedestrian identification result is that a pedestrian exists, inputting the second monitoring image and the foreign matter detection area into a preset foreign matter positioning model for foreign matter positioning, and outputting a foreign matter positioning result;
if the foreign matter positioning result is not empty, inputting the positioned foreign matter circumscribed rectangular image into a preset foreign matter classification model for article classification, and outputting probability values of the target foreign matter corresponding to different article names respectively;
and selecting the article name corresponding to the maximum probability value as the article name of the target foreign matter, and carrying out foreign matter early warning on the target foreign matter.
2. The method for detecting foreign matter in a public area according to claim 1, wherein the filtering the first monitoring image with noise and light to obtain the second monitoring image after interference elimination comprises:
transforming the first monitoring image from a spatial domain to a frequency domain using a fast fourier transform;
filtering low-frequency components and high-frequency components in a frequency domain corresponding to the first monitoring image by using a ButterWorth band-pass filter;
and restoring the filtered first monitoring image from a frequency domain to a space domain by adopting inverse fast Fourier transform to obtain a second monitoring image after interference removal.
3. The method for detecting foreign matters in a public area according to claim 1, wherein before the step of acquiring the first monitoring image of the target monitoring area, and performing noise and light filtering on the first monitoring image to obtain the second monitoring image after interference elimination, the method further comprises the following steps:
shooting a plurality of sample images of a target monitoring area, wherein the sample images contain pedestrians and placed foreign matters;
sequentially carrying out pedestrian detection and foreign matter detection on each sample image, and outputting a pedestrian detection frame and a foreign matter detection frame;
combining the pedestrian detection frame and the foreign matter detection frame corresponding to each sample image;
and taking the external rectangular coordinates corresponding to the combined pedestrian detection frame as a coordinate calibration frame corresponding to the pedestrian detection area, and taking the external rectangular coordinates corresponding to the combined foreign matter detection frame as a coordinate calibration frame corresponding to the foreign matter detection area.
4. The public area foreign matter detection method according to any one of claims 1 to 3, wherein the pedestrian detection model sequentially includes: fast SE-Resnet18 network, multilayer convolution layer, pooling layer, full connection layer and SoftMax layer, the inputting of the second monitoring image and the pedestrian detection area into a preset pedestrian detection model for pedestrian recognition, and the outputting of the result of pedestrian recognition comprises:
inputting the second monitoring image and the pedestrian detection area into a Fast SE-Resnet18 network in the pedestrian detection model for feature extraction, and outputting a plurality of first feature maps of the second monitoring image in the pedestrian detection area;
inputting each first feature map into a multilayer convolution layer in the pedestrian detection model to perform multi-round convolution operation to obtain a plurality of first feature matrixes corresponding to each first feature map;
inputting each first feature matrix into a pooling layer in the pedestrian detection model for down-sampling and feature compression, and outputting a plurality of second feature matrices;
inputting each second feature matrix into a full-connection layer in the pedestrian detection model for feature combination to obtain a plurality of third feature matrices;
and inputting each third feature matrix into a SoftMax layer in the pedestrian detection model for feature classification, and outputting a pedestrian recognition result.
5. The public area foreign matter detection method according to any one of claims 1 to 3, wherein the foreign matter localization model includes: an encoder consisting of a Fast SE-Resnet18 network and a decoder comprising in sequence: pooling layer, multilayer anti-convolution layer, SoftMax layer, if pedestrian's identification result is for there being the pedestrian, then will the second monitoring image with foreign matter location model is preset in the foreign matter detection area input carries out the foreign matter location, and output foreign matter location result includes:
if the pedestrian recognition result is that a pedestrian exists, inputting the second monitoring image and the foreign object detection area into a Fast SE-Resnet18 network in the foreign object positioning model for feature extraction, and outputting a plurality of first segmentation feature maps of the second monitoring image in the foreign object detection area;
inputting each first segmentation feature map into a pooling layer in the foreign matter positioning model for up-sampling and feature compression, and outputting a plurality of second segmentation feature maps;
inputting each second segmentation characteristic diagram into a multilayer deconvolution layer in the foreign matter positioning model to perform deconvolution operation, so as to obtain a plurality of segmentation characteristic matrixes corresponding to each second segmentation characteristic diagram;
and inputting each segmentation characteristic matrix into a SoftMax layer in the foreign matter positioning model for characteristic classification, outputting probability values of each segmentation image in the second monitoring image as a background, positioning the segmentation image where the foreign matter is located according to the probability values, and outputting the segmentation image as a foreign matter positioning result.
6. The public area foreign matter detection method according to any one of claims 1 to 3, wherein the foreign matter classification model sequentially includes: if the foreign matter positioning result is not empty, inputting a positioned foreign matter circumscribed rectangle image into a preset foreign matter classification model for article classification, and outputting probability values of target foreign matters corresponding to different article names respectively comprises the following steps:
if the foreign matter positioning result is not empty, inputting the positioned foreign matter circumscribed rectangular image into the SE-ResNet34 network for feature extraction, and outputting a plurality of second feature maps of the foreign matter circumscribed rectangular image;
inputting each second feature map into a multilayer convolution layer in the foreign matter classification model to carry out convolution operation, and obtaining a plurality of fourth feature matrixes corresponding to each second feature map;
inputting the fourth feature matrixes into a pooling layer in the foreign matter classification model for down-sampling and feature compression, and outputting a plurality of fifth feature matrixes;
inputting each fifth feature matrix into a full-connection layer in the foreign matter classification model for feature combination to obtain a plurality of sixth feature matrices;
and inputting the sixth feature matrixes into a SoftMax layer in the foreign matter classification model for feature classification, and outputting probability values of the target foreign matters corresponding to different article names respectively.
7. A public area foreign matter detection apparatus, characterized by comprising:
the interference removing module is used for acquiring a first monitoring image of a target monitoring area and filtering noise and light of the first monitoring image to obtain a second monitoring image after interference removal;
the area calibration module is used for calibrating a pedestrian detection area and a foreign matter detection area in the second monitoring image based on a preset coordinate calibration frame;
the pedestrian recognition module is used for inputting the second monitoring image and the pedestrian detection area into a preset pedestrian detection model for pedestrian recognition and outputting a pedestrian recognition result;
the foreign matter positioning module is used for inputting the second monitoring image and the foreign matter detection area into a preset foreign matter positioning model for foreign matter positioning and outputting a foreign matter positioning result if the pedestrian identification result indicates that a pedestrian exists;
the article classification module is used for inputting the positioned external rectangular image of the foreign matter into a preset foreign matter classification model for article classification if the foreign matter positioning result is not empty, and outputting probability values of the target foreign matter corresponding to different article names respectively;
and the foreign matter determining module is used for selecting the article name corresponding to the maximum probability value as the article name of the target foreign matter and carrying out foreign matter early warning on the target foreign matter.
8. The common zone foreign matter detection device according to claim 7, further comprising:
the calibration frame selection module is used for shooting a plurality of sample images of a target monitoring area, wherein the sample images contain pedestrians and placed foreign matters; sequentially carrying out pedestrian detection and foreign matter detection on each sample image, and outputting a pedestrian detection frame and a foreign matter detection frame; combining the pedestrian detection frame and the foreign matter detection frame corresponding to each sample image; and taking the external rectangular coordinates corresponding to the combined pedestrian detection frame as a coordinate calibration frame corresponding to the pedestrian detection area, and taking the external rectangular coordinates corresponding to the combined foreign matter detection frame as a coordinate calibration frame corresponding to the foreign matter detection area.
9. A public area foreign matter detection apparatus, characterized by comprising: a memory and at least one processor, the memory having instructions stored therein; the at least one processor invokes the instructions in the memory to cause the public area foreign object detection apparatus to perform the public area foreign object detection method of any one of claims 1-6.
10. A computer readable storage medium having instructions stored thereon, wherein the instructions, when executed by a processor, implement the public area foreign object detection method according to any one of claims 1 to 6.
CN202011538797.2A 2020-12-23 2020-12-23 Public area foreign matter detection method, device, equipment and storage medium Pending CN112560743A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011538797.2A CN112560743A (en) 2020-12-23 2020-12-23 Public area foreign matter detection method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011538797.2A CN112560743A (en) 2020-12-23 2020-12-23 Public area foreign matter detection method, device, equipment and storage medium

Publications (1)

Publication Number Publication Date
CN112560743A true CN112560743A (en) 2021-03-26

Family

ID=75032247

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011538797.2A Pending CN112560743A (en) 2020-12-23 2020-12-23 Public area foreign matter detection method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN112560743A (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108985169A (en) * 2018-06-15 2018-12-11 浙江工业大学 Across the door operation detection method in shop based on deep learning target detection and dynamic background modeling
CN109784186A (en) * 2018-12-18 2019-05-21 深圳云天励飞技术有限公司 A kind of pedestrian recognition methods, device, electronic equipment and computer readable storage medium again
CN109977782A (en) * 2019-02-27 2019-07-05 浙江工业大学 Across shop operation detection method based on target position information reasoning
WO2020062088A1 (en) * 2018-09-28 2020-04-02 安徽继远软件有限公司 Image identification method and device, storage medium, and processor

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108985169A (en) * 2018-06-15 2018-12-11 浙江工业大学 Across the door operation detection method in shop based on deep learning target detection and dynamic background modeling
WO2020062088A1 (en) * 2018-09-28 2020-04-02 安徽继远软件有限公司 Image identification method and device, storage medium, and processor
CN109784186A (en) * 2018-12-18 2019-05-21 深圳云天励飞技术有限公司 A kind of pedestrian recognition methods, device, electronic equipment and computer readable storage medium again
CN109977782A (en) * 2019-02-27 2019-07-05 浙江工业大学 Across shop operation detection method based on target position information reasoning

Similar Documents

Publication Publication Date Title
JP5546317B2 (en) Visual inspection device, visual inspection discriminator generation device, visual inspection discriminator generation method, and visual inspection discriminator generation computer program
US9767604B2 (en) Image analysis method by analyzing point cloud using hierarchical search tree
CN109977997B (en) Image target detection and segmentation method based on convolutional neural network rapid robustness
CN111553422A (en) Automatic identification and recovery method and system for surgical instruments
KR101933856B1 (en) System for image processing using the convolutional neural network and method using the same
CN108427959A (en) Board state collection method based on image recognition and system
CN114821102A (en) Intensive citrus quantity detection method, equipment, storage medium and device
CN108960260A (en) A kind of method of generating classification model, medical image image classification method and device
CN111161295A (en) Background stripping method for dish image
CN113988271A (en) Method, device and equipment for detecting high-resolution remote sensing image change
KR101874968B1 (en) Visibility measuring system base on image information and method for using the same
CN110443179B (en) Off-post detection method and device and storage medium
CN114723724A (en) Dangerous goods identification method, device, equipment and storage medium based on artificial intelligence
CN114612399A (en) Picture identification system and method for mobile phone appearance mark
Tribak et al. QR code patterns localization based on Hu Invariant Moments
CN108021914B (en) Method for extracting character area of printed matter based on convolutional neural network
CN113052234A (en) Jade classification method based on image features and deep learning technology
CN116503733B (en) Remote sensing image target detection method, device and storage medium
CN112435214A (en) Pollen detection method and device based on prior frame linear scaling and electronic equipment
CN112560743A (en) Public area foreign matter detection method, device, equipment and storage medium
CN117315578A (en) Monitoring method and system for rust area expansion by combining classification network
CN116563306A (en) Self-adaptive fire trace spectrum image segmentation method and system
CN110992361A (en) Engine fastener detection system and detection method based on cost balance
CN116778473A (en) Improved YOLOV 5-based mushroom offline real-time identification method and system
Tan et al. Automatic extraction of built-up area based on deep convolution neural network

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination