CN111539441A - Hidden object detection method and system based on millimeter wave security inspection image - Google Patents
Hidden object detection method and system based on millimeter wave security inspection image Download PDFInfo
- Publication number
- CN111539441A CN111539441A CN201911116590.3A CN201911116590A CN111539441A CN 111539441 A CN111539441 A CN 111539441A CN 201911116590 A CN201911116590 A CN 201911116590A CN 111539441 A CN111539441 A CN 111539441A
- Authority
- CN
- China
- Prior art keywords
- millimeter wave
- neural network
- convolutional neural
- hidden
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Multimedia (AREA)
- Molecular Biology (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Probability & Statistics with Applications (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Image Analysis (AREA)
- Radar Systems Or Details Thereof (AREA)
Abstract
The invention provides a hidden object detection method and system based on a millimeter wave security check image, wherein the method comprises the following steps: training random initialization parameters in a convolutional neural network by using image data which does not carry hidden objects and carries positive and negative samples of the hidden objects; carrying out feature extraction on newly sampled millimeter wave security check graphic data by using the trained convolutional neural network, and continuously training the convolutional neural network or the classifier by using the extracted features; and enabling the millimeter wave image to be detected to pass through the trained convolutional neural network to obtain a hidden object detection result. The invention can improve the recognition rate of dangerous hidden objects in millimeter wave security inspection imaging.
Description
Technical Field
The invention relates to the field of millimeter wave imaging, in particular to a hidden object detection method based on a millimeter wave security check image and a hidden object detection system based on the millimeter wave security check image.
Background
In recent years, anti-terrorism situation in China and China is getting more severe, safety problems become a problem of general attention of society in all countries, and especially, security inspection pressure of personnel in important occasions is huge, such as stations, airports, ports, customs gates in all places, and security inspection pressure of personnel needing to pass rapidly is also faced by large-scale venue assemblies.
In the past, conventional security inspection is directed at security inspection of a single cooperative target person, mainly used for inspecting small articles carried by people, such as lighters, knives, liquids and the like, and is typically used for security inspection of entrances of airports and railway stations. After years of practical operation, the security inspection is very important, and the occurrence of an event that an individual carries dangerous goods into an important place is effectively prevented.
The traditional security inspection means such as metal detection, infrared rays, X rays and manual hand touch cannot meet the requirements of new security situations. The traditional metal detector can only detect a short-distance small-range target; various rays such as X-ray can cause ionizing radiation damage to a detected human body; the infrared ray is based on the temperature imaging of the surface of an object, and the imaging cannot be clearly carried out under the condition that a fabric is shielded. The millimeter wave security inspection imaging system can detect not only hidden metal objects under the fabric, but also nonmetal dangerous goods such as plastic guns, ceramic cutters, explosives and the like, and can obtain visual security inspection images.
In the millimeter wave security inspection imaging system in the prior art, the millimeter wave security inspection imager market is dominated by L-3 in the United kingdom, Smith corporation and Rohde & Schwarz in Germany and represents the highest level of the millimeter wave security inspection imaging technology, but the three types of imagers are security inspection imaging systems facing close-range cooperative targets. The imaging resolution of an L3 millimeter wave imager product (shown in figure 1) is less than 1cm, the scanning time is about 2s, the imaging method is mainly characterized by rotary mechanical scanning, the imaging algorithm is a back scattering holographic imaging algorithm based on a dense array, and when an object to be detected is static, the imaging image artifact sidelobe is low, and the imaging quality is high. The germany RS millimeter wave security imaging system (as shown in fig. 2) adopts a fully electronic sparse front structure without mechanical scanning, and the basic principle of the imaging algorithm is similar to the holographic imaging algorithm based on back scattering of the company L3. For the millimeter wave imagers of L3 and RS, the operating bandwidth does not exceed 10GHz, and therefore the longitudinal distance resolution does not exceed 1.5cm, and in practical applications, the longitudinal resolution is not high, so that the image resolution is seriously affected when the examined passenger wears a plurality of pieces of clothes or thick clothes. The imaging principle and basic algorithm of the millimeter wave imager of the Smiths Detection company are similar to those of the two aforementioned companies.
However, compared with the existing automatic target recognition algorithm for imaging close-range static cooperative target characters, the automatic target recognition algorithm for non-cooperative target characters in a moving state has higher difficulty in recognizing hidden objects in an image due to image blurring or distortion caused by movement. How to improve the identification rate of hazardous articles carried by human bodies in security inspection imaging is a problem to be urgently solved in the prior art.
Disclosure of Invention
The technical problem to be solved by the technical scheme of the invention is how to improve the recognition rate of dangerous hidden objects in millimeter wave security inspection imaging.
In order to solve the technical problem, the technical scheme of the invention provides a hidden object detection method based on a millimeter wave security check image, which comprises the following steps:
training random initialization parameters in a convolutional neural network by using image data which does not carry hidden objects and carries positive and negative samples of the hidden objects;
carrying out feature extraction on newly sampled millimeter wave security check graphic data by using the trained convolutional neural network, and continuously training the convolutional neural network or the classifier by using the extracted features;
and enabling the millimeter wave image to be detected to pass through the trained convolutional neural network to obtain a hidden object detection result.
Optionally, the training of the random initialization parameter in the convolutional neural network by using the image data of the positive and negative samples carrying no hidden object and the hidden object includes:
the image data is subjected to positive and negative sample selection through a preset sliding window based on IoU indexes to train a convolutional neural network, and the IoU indexes are suitable for screening the image data and training feature differences among the image data.
Optionally, the IoU index refers to a criterion for measuring the accuracy of detection of concealed objects in the image data.
Optionally, the performing feature extraction on the newly sampled millimeter wave security check graphic data includes:
enabling newly sampled millimeter wave security check graphic data to pass through the trained convolutional neural network to extract multi-level features;
the continuing training of the convolutional neural network or classifier using the extracted features comprises:
automatically classifying the extracted multi-level features, and marking out suspicious regions according to the similarity;
synthesizing all suspicious regions to obtain a suspicious target heat map, and further determining suspicious target positions of hidden objects in the newly sampled millimeter wave security inspection graphic data;
continuing to train the convolutional neural network or classifier based on the suspected target location of the concealed object.
Optionally, the performing feature extraction on the newly sampled millimeter wave security check graphic data includes:
extracting FAST characteristics of the newly sampled millimeter wave security inspection graphic data;
continuing to train the convolutional neural network or classifier using the extracted features comprises:
carrying out preliminary judgment on hidden areas of the millimeter wave security check graphic data based on the FAST characteristics of the extracted millimeter wave security check graphic data;
clustering is carried out on the basis of the preliminarily judged hidden area, and a search area is determined;
performing exhaustive search near a search area by using a preset sliding window, and detecting and identifying a window area image by using the trained convolutional neural network to distinguish whether hidden objects exist or not;
carrying out probability statistics on the detection and identification results to obtain the position where the hidden object is most likely to exist;
continuing to train the convolutional neural network or classifier based on the locations of the concealed objects.
Optionally, the preset sliding window size is 60 × 44.
Optionally, the convolutional neural network has 7 layers, starting from the input layer, the size of the first convolutional layer is 5 × 5, 40 56 × 40 feature maps are output, the number of the feature maps after the first pooling layer is unchanged, and the size extraction is 28 × 20; the second convolution layer uses a convolution kernel of 5 multiplied by 5 to output 30 characteristic graphs of 24 multiplied by 16, the quantity of the characteristic graphs is unchanged after the second pooling layer, and the size extraction is 12 multiplied by 8; the third convolutional layer uses a convolution kernel of 5 multiplied by 5, 30 feature maps of 8 multiplied by 4 are output after convolution, the output of the third convolutional layer is connected to the full connection layer, and the final judgment result is obtained through output of Softmax.
In order to solve the above technical problem, the present invention provides another concealed object detection system based on a millimeter wave security check image, including:
the initial training unit is suitable for training the random initialization parameters in the convolutional neural network by using the image data which does not carry the concealed objects and carries the concealed objects, namely, the positive and negative samples;
the continuous training unit is suitable for extracting the characteristics of the newly sampled millimeter wave security check graphic data by using the trained convolutional neural network and continuously training the convolutional neural network or the classifier by using the extracted characteristics;
and the detection unit is suitable for enabling the millimeter wave image to be detected to pass through the trained convolutional neural network so as to obtain a hidden object detection result.
Optionally, the training continuation unit includes:
the multilevel extraction subunit is suitable for enabling the newly sampled millimeter wave security check graphic data to pass through the trained convolutional neural network so as to extract multilevel characteristics;
the marking subunit is suitable for automatically classifying the extracted multi-level features and marking out suspicious regions according to the similarity;
the suspicious detection subunit is suitable for synthesizing all the suspicious regions to be superposed to obtain a suspicious target heat map so as to determine the suspicious target position of the hidden object in the newly sampled millimeter wave security inspection graphic data;
a first training subunit adapted to continue training the convolutional neural network or classifier based on the suspected target location of the concealer.
Optionally, the training continuation unit includes:
the FAST feature extraction subunit is suitable for extracting the FAST features of the newly sampled millimeter wave security inspection graphic data;
the preliminary judgment subunit is suitable for carrying out preliminary judgment on the hidden areas of the millimeter wave security check graphic data based on the FAST characteristics of the extracted millimeter wave security check graphic data;
the clustering subunit is suitable for clustering based on the preliminarily judged hidden area to determine a search area;
the identification subunit is suitable for performing exhaustive search on the vicinity of a search area by using a preset sliding window, and detecting and identifying the window area image by using the trained convolutional neural network to distinguish whether hidden objects exist or not;
the statistic subunit is suitable for carrying out probability statistics on the detection and identification results to obtain the position where the hidden object is most likely to exist;
a second training subunit adapted to continue training the convolutional neural network or classifier based on the location of the concealer.
The technical scheme of the invention at least comprises the following beneficial effects:
according to the technical scheme, the automatic target identification of the dangerous goods carried by the generated image can be performed aiming at the security inspection imaging image of the non-cooperative target person in the walking state, and the identification accuracy of the hidden goods is improved. Compared with the traditional method, the technical scheme provided by the invention can obviously improve two key indexes of the detection rate and the false alarm rate, and accords with the actual effect. According to the technical scheme, the relationship between the model and each physical quantity in an actual scene is tried to be deduced based on the deep learning neural network model, so that the accuracy of identifying the hidden object target of the image can be improved.
Drawings
Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments with reference to the following drawings:
FIG. 1 is a schematic diagram illustrating an application of a millimeter wave security inspection imaging system in the prior art;
FIG. 2 is a schematic diagram of another millimeter wave security inspection imaging system in the prior art;
FIG. 3 is a schematic structural diagram of an active millimeter wave imaging security inspection instrument for cooperative human body security inspection;
FIG. 4 is a schematic structural diagram of a three-dimensional dense array imaging system;
FIG. 5 is a schematic diagram illustrating the existence of hidden objects in the millimeter wave image according to the present invention;
FIG. 6 is a block diagram illustrating a method for detecting hidden objects in millimeter-wave images according to an embodiment of the present invention;
fig. 7 is a schematic diagram illustrating a result of preliminary determination of a foreign object region using FAST characteristics for a millimeter wave image according to the present invention;
fig. 8 is a schematic diagram of a CNN network structure used in the technical solution of the present invention;
FIG. 9 is a schematic diagram illustrating the results of testing 5 kinds of hidden objects according to the present invention;
fig. 10 is a schematic diagram of a detection result obtained after counting 100 pictures according to the technical solution of the present invention;
fig. 11 is a schematic flow chart of a hidden object detection method based on a millimeter wave security check image according to the present invention;
fig. 12 is a schematic structural diagram of a concealed object detection system based on a millimeter wave security inspection image according to the present invention;
fig. 13 is a schematic structural diagram of another concealed object detection system based on a millimeter wave security inspection image according to the present invention.
Detailed Description
In order to better and clearly show the technical scheme of the invention, the invention is further described with reference to the attached drawings.
A millimeter wave security inspection imaging system, namely an active millimeter wave imaging security inspection instrument for cooperative human body security inspection, is mainly structured as shown in figure 3, and the system combines electronic array scanning and mechanical scanning, and has a core of 2 x 80 array elements of millimeter wave receiving and transmitting imaging front-end linear arrays, the scanning is carried out by switching the imaging front-end linear arrays through switches in the horizontal direction, the scanning is carried out by vertical mechanical motion of the imaging front-end linear arrays in the vertical direction, and multi-frequency point scanning is carried out by combining each horizontal scanning, so that the space-frequency three-dimensional electromagnetic echo data of an imaging target area is obtained. From the point of view of the imaging algorithm, the imaging system may be understood as a three-dimensional dense array imaging system as shown in fig. 4.
Referring to fig. 4, assuming that the target ("target") is a volumetric target ("target point") with a strong reflection point coordinate of (x, y, Z), the transceiver-filled scan plane ("scanned" aperture) achieved by electronic switching and vertical mechanical scanning still lies at Z = Z1The position ("transceiver position") of a transceiver on the scan plane is (x ', y', Z)1）。
When the system works in a wide frequency band instead of a frequency point, the receiving response of a certain transceiver at the frequency point ω = kc can be obtained by integrating the reflection characteristic function f (x, y, z) of the target, and k is a wave function.
The basic algorithm principle formula of the system can be obtained through transformation as follows:
according to an algorithm formula, the echo complex signals of each scanning point, namely the amplitude and the phase of the echo, are required to be acquired in active millimeter wave security inspection imaging, and an imaging scene image can be recovered.
Based on the millimeter wave security inspection imaging system and the imaging principle thereof, aiming at the security inspection imaging of non-cooperative target people in a walking state, the research of an automatic target identification algorithm carrying dangerous goods on generated images is needed. Compared with the existing automatic target recognition algorithm for imaging close-distance static cooperative target characters, the automatic target recognition algorithm for non-cooperative target characters in a motion state has higher difficulty due to image blurring or distortion caused by motion. In the research of the technical scheme of the invention, the identification rate of the dangerous goods carried by the human body needs to be improved, and the false alarm rate needs to be reduced as much as possible.
Compared with a cooperative millimeter wave terahertz human body security inspection system which mainly takes a static image as a data source for automatic target detection, the non-cooperative type security inspection system can cause great difference of the size, position, form and gray level information of a formed image due to the fact that the posture, movement and position of a detected person are not limited, a plurality of human bodies and articles can also appear, and shielding can also appear among the human bodies and the articles, so that the data source for automatic target detection adopts a millimeter wave video stream and a synchronously recorded optical video stream, the two data sources provide different information of the detected object, the information can be reasonably utilized to make up the working mode of the non-cooperative type security inspection system, and the difficulty of target detection is overcome.
The technical scheme of the invention considers that a multi-mode information fusion model is adopted to realize the automatic detection of the target, the two video streams reflect the temporal-spatial characteristic change of the target to be detected in the process of moving along with the detected person, and a temporal-spatial unified deep learning neural network model is adopted to process so as to fully utilize the temporal-spatial correlation of the target to be detected.
The technical scheme of the invention researches network training and semi-supervised training of the network under the condition of small data samples so as to reduce various expenses on network learning and requirements on actual use.
In the technical scheme of the invention, a reconfigurable millimeter wave security inspection imaging system is designed and developed according to the system, and then the imaging algorithm experiment and automatic dangerous goods target identification algorithm verification are carried out by utilizing the system. In the experimental verification, people with different sexes and different body types are arranged as experimental objects. During the experiment, the imaging algorithm and the automatic target recognition algorithm are debugged and perfected through the conditions of stillness, walking, carrying dangerous goods, not carrying dangerous goods and the like.
After the millimeter wave image and the video image of the non-cooperative pedestrian target are obtained, the millimeter wave image is subjected to corresponding image enhancement by using the video image, then noise reduction is carried out, background noise is removed, the integrity of the human body image is kept to the maximum extent, then image features are extracted, and deep learning and recognition are carried out on different imaging situations through a classifier.
(a) Multimodal information fusion
Compared with a data source of a cooperative millimeter wave terahertz human body security inspection system which mainly takes a static image as a target for automatic detection, the non-cooperative type security inspection system can cause great difference of the size, position, form and gray information of a formed image due to the fact that the posture, motion and position of a detected person are not limited, and a plurality of human bodies and articles can appear and can be shielded mutually. Therefore, the data source for automatically detecting the target adopts the millimeter wave video stream and the optical video stream, the two data sources provide different information of the detected object, and the information can be reasonably utilized to make up for the target detection difficulty caused by the change of the working mode of the non-cooperative security inspection system. The invention considers the multi-mode information fusion strategy to realize the automatic detection of the target, one method is to adopt two deep learning networks (a convolution neural network or a recursion neural network aiming at the video information processing) to respectively process millimeter wave video stream and optical video stream, then to use the conventional data fusion method to carry out the information fusion under the constraint conditions of space correspondence and time correspondence, and then to carry out the target identification according to the fused information; and the other method completely adopts a unified deep learning network framework, the millimeter wave video stream and the optical video stream are respectively used as two major parts of multi-source input, and then high-level features are uniformly extracted from a high-level abstract part for classification and identification.
(b) Deep reinforcement learning
According to the technical scheme, if the convolutional neural network is adopted to automatically detect the dangerous goods, whether enough massive training data sets exist or not is also the problem that whether the data are marked or not, which is particularly remarkable in a non-cooperative millimeter wave terahertz security inspection system, because the data used for training are changed into videos from images, each frame of image in the videos is difficult to be manually marked (position and type), and due to shielding caused by position angles, the targets do not always appear in the whole detected time period. The present invention proposes to solve this problem in the form of a combination of partially labeled data and partially unlabeled data, i.e., changing supervised training to semi-supervised training. One solution for realizing the idea is to combine the convolutional neural network and reinforcement learning to fully utilize the perception capability of the convolutional neural network and the decision capability of the convolutional neural network. Selecting a plurality of video frames containing targets with definite positions in a video stream for marking, and using the video frames for supervised training of a convolutional neural network; and the video frames adjacent to the target detection area are made into a reinforcement learning punishment rule according to the possibility that the target is detected and the corresponding position and size, and the reinforcement learning punishment rule is used for reinforcement learning of the convolutional neural network, so that the identification rate of target detection can be improved.
(c) Deep migration learning
Transfer learning refers to a machine learning method for solving problems in different but related fields by using learned knowledge. The technical scheme of the invention has a problem that enough data of the cooperative millimeter wave terahertz security inspection system can not be acquired and marked in enough time to form a training library of mass data, or the acquired data can not cover various situations of hidden dangerous goods, so that the developed experimental system has poor automatic detection effect of the dangerous goods due to insufficient training. If the convolutional neural network and the transfer learning are combined, the problem may be solved, and the transfer learning of the convolutional neural network is formed, and the flow thereof is as follows:
1) the method comprises the steps that image data of a positive sample and a negative sample which do not carry a concealed object and carry the concealed object are shot by an existing cooperation type millimeter wave terahertz security inspection system in a posture with different specified near field positions, the concealed object is designed according to a target to be detected by the non-cooperation type millimeter wave terahertz security inspection system, and the image data of real inspection standard postures of the practical cooperation type millimeter wave terahertz security inspection system are added to train random initialization parameters in a convolutional neural network;
2) carrying out feature extraction on new sampling data of a trial-made non-cooperative millimeter wave terahertz security check system by using a trained convolutional neural network;
3) and further training a convolutional neural network or a classifier according to the data of the trial-made non-cooperative millimeter wave terahertz security inspection system by using the extracted features.
The study of such convolutional neural networks in conjunction with migratory learning may have the following significance: 1) the problem that training samples of a convolutional neural network are insufficient under the condition of a small data set is solved; 2) the existing achievements of the cooperative millimeter wave terahertz security inspection system are fully utilized, so that the cost required when the non-cooperative millimeter wave terahertz security inspection system is retrained to be matched with a network and the corresponding software upgrading cost when the hardware of a subsequent system is upgraded are greatly reduced; 3) for new dangerous goods which appear newly or new modes of hidden dangerous goods, the detection performance of the convolutional neural network can be improved in a similar progressive mode.
(d) Spatiotemporal target detection
The data source for the automatic target detection provided by the non-cooperative security check system is millimeter wave video stream and optical video stream shot synchronously, and the motion of human body may endow the target with strong space-time variation characteristics, thus increasing the difficulty of automatic detection. The invention considers the experience and achievement of research in the attention selection information processing of the video stream, not only decouples the video stream into an image set and forms the extracted target information into a time sequence for detection and analysis, but also considers uniformly forming two video streams into a high-dimensional data space, popularizes the original two-dimensional convolution depth neural network into an honest high-dimensional convolution neural network, constructs a new high-dimensional convolution kernel, fully utilizes the high correlation of the target space-time information, and improves the recognition rate of target detection.
The technical scheme of the invention provides a CNN network-based automatic detection method for human hidden objects, which comprises the following three steps:
firstly, selecting positive and negative samples of a millimeter wave image through a sliding window with the size determined by priori knowledge based on an IoU (interference over ion) index, so as to overcome the problem that the current millimeter wave image database training sample set is insufficient, and basically meet the requirement of the designed CNN network deep learning on the size of the training data set;
meanwhile, an IoG (interaction over ground-route) index is newly provided, and the candidate positive and negative samples are further screened according to the index, so that the characteristic difference between the positive and negative samples for training is enlarged, and the generalization capability of the CNN model obtained by training is enhanced.
Then, the millimeter wave image to be detected is subjected to multi-level feature extraction through the trained CNN by the image block intercepted by each sliding window, automatic classification is carried out according to the multi-level feature extraction, and the suspicious region is marked according to the similarity degree.
And finally, synthesizing all the suspicious regions to obtain a suspicious target heat map, and further determining the position of the suspicious target.
Fig. 5 is a diagram showing the existence of hidden objects in the millimeter wave image according to the present invention.
Fig. 6 is a block diagram of a method for detecting a hidden object in a millimeter wave image according to the present invention.
The preliminary test result of the detection method on the data set provided by the millimeter wave/terahertz human body security inspection imaging system shows that the detection rate and the false alarm rate are obviously improved compared with the traditional method, the detection rate is improved to 88% from no more than 50%, the false alarm rate is reduced to 13.5% from the maximum 85% in the traditional method, and the practical requirement is basically met.
The technical scheme of the invention also analyzes and researches 700 millimeter wave static security inspection images of the SimImage system provided from the outside, and provides a calculation method for detecting and positioning foreign matters in the millimeter wave images. The basic idea and flow of the method are as follows:
1) considering that the joints between the foreign body and the body part generally have texture changes, the fast (feature from accessed Segment test) feature of the image is sensitive to the change area, so that the foreign body can be located and identified by detecting the image feature. The result of preliminary determination of foreign matter regions using FAST features for millimeter wave images is shown in fig. 7, where the position marked by the "+" point in fig. 7 is a region where foreign matter may exist.
2) And clustering the areas where foreign matters possibly exist, and determining the search area.
3) And performing exhaustive search on the vicinity of the characteristic region by using a sliding window, wherein the size of the search window is 60 multiplied by 44, and detecting and identifying a window region image by using a Convolutional Neural Network (CNN) (volumetric Neural network) to distinguish whether foreign objects exist or not. The CNN network structure is shown in fig. 8. Namely:
the convolutional neural network used is 7 layers, starting from the input layer, the size of the first convolutional layer is 5 multiplied by 5, 40 characteristic maps of 56 multiplied by 40 are output, the number of the characteristic maps is unchanged after the first pooling layer, and the size extraction is 28 multiplied by 20; the second convolution layer uses a convolution kernel of 5 multiplied by 5 to output 30 characteristic graphs of 24 multiplied by 16, the quantity of the characteristic graphs is unchanged after the second pooling layer, and the size extraction is 12 multiplied by 8; the third convolutional layer uses a convolution kernel of 5 multiplied by 5, 30 feature maps of 8 multiplied by 4 are output after convolution, the output of the third convolutional layer is connected to the full connection layer, and the final judgment result is obtained through output of Softmax.
4) And carrying out probability statistics on the CNN identification result to obtain the most possible foreign body position. The 700 images provided from the outside include 5 kinds of foreign matters including a knife, a gun, a mobile phone, a hammer and a wrench. 600 of the images (572 containing a single alien material and 28 free) were segmented and used to train the above CNN. After training was completed, the test was performed using 100 images, 88 images containing a single foreign object and 12 images containing no foreign object. The results of the 5 foreign body tests are shown in fig. 9, and include: a) hammer (hammer), b) wrench (wrench), c) mobile phone (mobile phone), d) knife (knife), 5) pistol (roll). The dotted frame in the figure is groudtruth, and the solid frame is the result of positioning. The results of counting 100 pictures are shown in the detection statistical data table shown in fig. 10, and the overall recognition rate in the test set can reach 95%.
In the prior art, a deep neural network is adopted for image target identification, mostly, black box type operation in the aspect of pure image frequency spectrum is carried out, and the relation between each physical parameter of an actual system and a neural network input element is ignored, so that a neural network model cannot be associated with an actual imaging system and an imaging target, and a physical parameter model cannot be provided. In the technical scheme of the invention, for a moving target, compared with the existing static target, the scattering models and the abundant Doppler information of different parts of the moving target can be obtained when people walk, so that a large amount of theoretical research and experimental verification are needed to research the deep learning neural network model for identifying the image target of the pedestrian non-cooperative target, and the relation between the model and each physical quantity in an actual scene is tried to be deduced, so that the accuracy of identifying the hidden object target in the image can be improved.
Based on the above, the technical solution of the present invention provides a method for detecting a hidden object based on a millimeter wave security check image, as shown in fig. 11, the method includes:
s100, training random initialization parameters in a convolutional neural network by using image data which does not carry hidden objects and carries positive and negative samples of the hidden objects;
step S101, utilizing the trained convolutional neural network to extract characteristics of newly sampled millimeter wave security check pattern data, and utilizing the extracted characteristics to continuously train the convolutional neural network or the classifier;
and S102, enabling the millimeter wave image to be detected to pass through the trained convolutional neural network to obtain a hidden object detection result.
In step S100, the training of the random initialization parameter in the convolutional neural network using the image data of the positive and negative samples carrying no hidden object and the hidden object includes:
the image data is subjected to positive and negative sample selection through a preset sliding window based on IoU indexes to train a convolutional neural network, and the IoU indexes are suitable for screening the image data and training feature differences among the image data. The IoU index refers to a criterion that measures the accuracy of detection of concealed objects in the image data.
According to step S101, in an example, the performing feature extraction on the newly sampled millimeter wave security check graphic data includes: enabling newly sampled millimeter wave security check graphic data to pass through the trained convolutional neural network to extract multi-level features;
the continuing training of the convolutional neural network or classifier using the extracted features comprises: automatically classifying the extracted multi-level features, and marking out suspicious regions according to the similarity; and synthesizing all suspicious regions to obtain a suspicious target heat map, and further determining suspicious target positions of hidden objects in the newly sampled millimeter wave security check graphic data; continuing to train the convolutional neural network or classifier based on the suspected target location of the concealed object.
According to step S101, in another example, the performing feature extraction on the newly sampled millimeter wave security check graphic data includes: extracting FAST characteristics of the newly sampled millimeter wave security inspection graphic data;
the continuing training of the convolutional neural network or classifier using the extracted features comprises:
carrying out preliminary judgment on hidden areas of the millimeter wave security check graphic data based on the FAST characteristics of the extracted millimeter wave security check graphic data;
clustering is carried out on the basis of the preliminarily judged hidden area, and a search area is determined;
performing exhaustive search near a search area by using a preset sliding window, and detecting and identifying a window area image by using the trained convolutional neural network to distinguish whether hidden objects exist or not;
carrying out probability statistics on the detection and identification results to obtain the position where the hidden object is most likely to exist;
continuing to train the convolutional neural network or classifier based on the locations of the concealed objects.
The preset sliding window size and the structure of the convolutional neural network can refer to the above contents.
According to step S102, in fact, the implementation of step S102 may refer to the implementation of step S101, steps S102 and S101 may be executed in parallel, or steps S101 and S102 may be both the same step in other embodiments. In other embodiments, the millimeter wave image to be detected may be understood as a newly sampled millimeter wave security check graphic data, and therefore the execution of step S102 may be performed with reference to step S101.
When the trained convolutional neural network is accurate enough, the detection result can be obtained by directly extracting the trained convolutional neural network according to a general image feature extraction principle.
In one implementation of step S102, the method includes:
enabling newly sampled millimeter wave security check graphic data to pass through the trained convolutional neural network to extract multi-level features;
automatically classifying the extracted multi-level features, and marking out suspicious regions according to the similarity; and the number of the first and second groups,
and synthesizing all suspicious regions to obtain a suspicious target heat map, and further determining the suspicious target position of the hidden object in the newly sampled millimeter wave security inspection graphic data.
In another implementation of step S102, the method includes:
extracting FAST characteristics of the newly sampled millimeter wave security inspection graphic data;
carrying out preliminary judgment on hidden areas of the millimeter wave security check graphic data based on the FAST characteristics of the extracted millimeter wave security check graphic data;
clustering is carried out on the basis of the preliminarily judged hidden area, and a search area is determined;
performing exhaustive search near a search area by using a preset sliding window, and detecting and identifying a window area image by using the trained convolutional neural network to distinguish whether hidden objects exist or not;
and carrying out probability statistics on the detection and identification results to obtain the position where the hidden object is most likely to exist.
The technical solution of the present invention is based on the above mentioned hidden object detection method, and further provides a hidden object detection system based on a millimeter wave security check image, as shown in fig. 12, including:
the initial training unit 1 is suitable for training random initialization parameters in a convolutional neural network by using image data which does not carry hidden objects and carries positive and negative samples of the hidden objects;
the continuous training unit 2 is suitable for extracting features of newly sampled millimeter wave security check graphic data by using the trained convolutional neural network, and continuously training the convolutional neural network or the classifier by using the extracted features;
and the detection unit 3 is suitable for enabling the millimeter wave image to be detected to pass through the trained convolutional neural network so as to obtain a hidden object detection result.
In another modification, as shown in fig. 13, a concealed object detection system based on a millimeter wave security check image includes:
the initial training unit 1' is suitable for training random initialization parameters in a convolutional neural network by using image data which does not carry hidden objects and carries positive and negative samples of the hidden objects;
the detection unit 2' is suitable for extracting the characteristics of the millimeter wave image to be detected through the trained convolutional neural network by using the trained convolutional neural network so as to obtain a hidden object detection result, and continuously training the convolutional neural network or the classifier by using the extracted characteristics.
The foregoing description of specific embodiments of the present invention has been presented. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes and modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention.
Claims (10)
1. A hidden object detection method based on a millimeter wave security check image is characterized by comprising the following steps:
training random initialization parameters in a convolutional neural network by using image data which does not carry hidden objects and carries positive and negative samples of the hidden objects;
carrying out feature extraction on newly sampled millimeter wave security check graphic data by using the trained convolutional neural network, and continuously training the convolutional neural network or the classifier by using the extracted features;
and enabling the millimeter wave image to be detected to pass through the trained convolutional neural network to obtain a hidden object detection result.
2. The method of claim 1 wherein training random initialization parameters in a convolutional neural network using image data that carries no concealer and both positive and negative samples of concealer comprises:
the image data is subjected to positive and negative sample selection through a preset sliding window based on IoU indexes to train a convolutional neural network, and the IoU indexes are suitable for screening the image data and training feature differences among the image data.
3. The method according to claim 2, wherein the IoU index is a criterion for measuring accuracy of detection of a concealed object in the image data.
4. The concealed object detection method according to claim 1, wherein said performing feature extraction with respect to newly sampled millimeter wave security check graphic data comprises:
enabling newly sampled millimeter wave security check graphic data to pass through the trained convolutional neural network to extract multi-level features;
the continuing training of the convolutional neural network or classifier using the extracted features comprises:
automatically classifying the extracted multi-level features, and marking out suspicious regions according to the similarity;
synthesizing all suspicious regions to obtain a suspicious target heat map, and further determining suspicious target positions of hidden objects in the newly sampled millimeter wave security inspection graphic data;
continuing to train the convolutional neural network or classifier based on the suspected target location of the concealed object.
5. The concealed object detection method according to claim 1, wherein said performing feature extraction with respect to newly sampled millimeter wave security check graphic data comprises:
extracting FAST characteristics of the newly sampled millimeter wave security inspection graphic data;
continuing to train the convolutional neural network or classifier using the extracted features comprises:
carrying out preliminary judgment on hidden areas of the millimeter wave security check graphic data based on the FAST characteristics of the extracted millimeter wave security check graphic data;
clustering is carried out on the basis of the preliminarily judged hidden area, and a search area is determined;
performing exhaustive search near a search area by using a preset sliding window, and detecting and identifying a window area image by using the trained convolutional neural network to distinguish whether hidden objects exist or not;
carrying out probability statistics on the detection and identification results to obtain the position where the hidden object is most likely to exist;
continuing to train the convolutional neural network or classifier based on the locations of the concealed objects.
6. The method of claim 5 wherein the predetermined sliding window size is 60 x 44.
7. The concealment object detection method according to claim 5, wherein said convolutional neural network has 7 layers, starting from the input layer, the first convolutional layer has a size of 5 x 5, 40 56 x 40 feature maps are output, the number of feature maps is unchanged after the first pooling layer, and the size extraction is 28 x 20; the second convolution layer uses a convolution kernel of 5 multiplied by 5 to output 30 characteristic graphs of 24 multiplied by 16, the quantity of the characteristic graphs is unchanged after the second pooling layer, and the size extraction is 12 multiplied by 8; the third convolutional layer uses a convolution kernel of 5 multiplied by 5, 30 feature maps of 8 multiplied by 4 are output after convolution, the output of the third convolutional layer is connected to the full connection layer, and the final judgment result is obtained through output of Softmax.
8. A concealed object detection system based on a millimeter wave security check image is characterized by comprising:
the initial training unit is suitable for training the random initialization parameters in the convolutional neural network by using the image data which does not carry the concealed objects and carries the concealed objects, namely, the positive and negative samples;
the continuous training unit is suitable for extracting the characteristics of the newly sampled millimeter wave security check graphic data by using the trained convolutional neural network and continuously training the convolutional neural network or the classifier by using the extracted characteristics;
and the detection unit is suitable for enabling the millimeter wave image to be detected to pass through the trained convolutional neural network so as to obtain a hidden object detection result.
9. The concealer detection system according to claim 8, wherein the continuous training unit comprises:
the multilevel extraction subunit is suitable for enabling the newly sampled millimeter wave security check graphic data to pass through the trained convolutional neural network so as to extract multilevel characteristics;
the marking subunit is suitable for automatically classifying the extracted multi-level features and marking out suspicious regions according to the similarity;
the suspicious detection subunit is suitable for synthesizing all the suspicious regions to be superposed to obtain a suspicious target heat map so as to determine the suspicious target position of the hidden object in the newly sampled millimeter wave security inspection graphic data;
a first training subunit adapted to continue training the convolutional neural network or classifier based on the suspected target location of the concealer.
10. The concealer detection system according to claim 8, wherein the continuous training unit comprises:
the FAST feature extraction subunit is suitable for extracting the FAST features of the newly sampled millimeter wave security inspection graphic data;
the preliminary judgment subunit is suitable for carrying out preliminary judgment on the hidden areas of the millimeter wave security check graphic data based on the FAST characteristics of the extracted millimeter wave security check graphic data;
the clustering subunit is suitable for clustering based on the preliminarily judged hidden area to determine a search area;
the identification subunit is suitable for performing exhaustive search on the vicinity of a search area by using a preset sliding window, and detecting and identifying the window area image by using the trained convolutional neural network to distinguish whether hidden objects exist or not;
the statistic subunit is suitable for carrying out probability statistics on the detection and identification results to obtain the position where the hidden object is most likely to exist;
a second training subunit adapted to continue training the convolutional neural network or classifier based on the location of the concealer.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910090454.5A CN109886319A (en) | 2019-01-30 | 2019-01-30 | A kind of concealment object detecting method and system based on millimeter wave safety check image |
| CN2019100904545 | 2019-01-30 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111539441A true CN111539441A (en) | 2020-08-14 |
Family
ID=66927359
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910090454.5A Pending CN109886319A (en) | 2019-01-30 | 2019-01-30 | A kind of concealment object detecting method and system based on millimeter wave safety check image |
| CN201911116590.3A Pending CN111539441A (en) | 2019-01-30 | 2019-11-15 | Hidden object detection method and system based on millimeter wave security inspection image |
Family Applications Before (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910090454.5A Pending CN109886319A (en) | 2019-01-30 | 2019-01-30 | A kind of concealment object detecting method and system based on millimeter wave safety check image |
Country Status (1)
| Country | Link |
|---|---|
| CN (2) | CN109886319A (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110533025A (en) * | 2019-07-15 | 2019-12-03 | 西安电子科技大学 | The millimeter wave human body image detection method of network is extracted based on candidate region |
| CN111008676B (en) * | 2019-12-27 | 2022-07-12 | 哈尔滨工业大学 | Security inspection method and security inspection system |
| CN110956225B (en) * | 2020-02-25 | 2020-05-29 | 浙江啄云智能科技有限公司 | Contraband detection method and system, computing device and storage medium |
| CN111738290B (en) * | 2020-05-14 | 2024-04-09 | 北京沃东天骏信息技术有限公司 | Image detection method, model construction and training method, device, equipment and medium |
| CN112183583A (en) * | 2020-09-08 | 2021-01-05 | 昆明理工大学 | Small target detection method in attention-oriented terahertz image |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102708372A (en) * | 2012-02-29 | 2012-10-03 | 北京无线电计量测试研究所 | Automatic detection and identification method for hidden articles |
| WO2018059125A1 (en) * | 2016-09-27 | 2018-04-05 | 华讯方舟科技有限公司 | Millimeter wave image based human body foreign object detection method and system |
-
2019
- 2019-01-30 CN CN201910090454.5A patent/CN109886319A/en active Pending
- 2019-11-15 CN CN201911116590.3A patent/CN111539441A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102708372A (en) * | 2012-02-29 | 2012-10-03 | 北京无线电计量测试研究所 | Automatic detection and identification method for hidden articles |
| WO2018059125A1 (en) * | 2016-09-27 | 2018-04-05 | 华讯方舟科技有限公司 | Millimeter wave image based human body foreign object detection method and system |
Non-Patent Citations (2)
| Title |
|---|
| 姚家雄;杨明辉;朱玉琨;吴亮;孙晓玮;: "利用卷积神经网络进行毫米波图像违禁物体定位" * |
| 骆尚;吴晓峰;杨明辉;王斌;孙晓玮;: "基于卷积神经网络的毫米波图像人体隐匿物检测" * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109886319A (en) | 2019-06-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111539441A (en) | Hidden object detection method and system based on millimeter wave security inspection image | |
| US7365672B2 (en) | Detection of a concealed object | |
| EP2148217B1 (en) | Detection of a concealed object | |
| Frigui et al. | Detection and discrimination of land mines in ground-penetrating radar based on edge histogram descriptors and a possibilistic $ k $-nearest neighbor classifier | |
| Yang et al. | CNN with spatio-temporal information for fast suspicious object detection and recognition in THz security images | |
| CN104408482B (en) | A kind of High Resolution SAR Images object detection method | |
| Moalla et al. | Application of convolutional and recurrent neural networks for buried threat detection using ground penetrating radar data | |
| Noreen et al. | Using pattern recognition with HOG to automatically detect reflection hyperbolas in ground penetrating radar data | |
| CN110245675A (en) | A kind of dangerous objects detection method based on millimeter-wave image human body contextual information | |
| Wang et al. | GPRI2Net: A deep-neural-network-based ground penetrating radar data inversion and object identification framework for consecutive and long survey lines | |
| Zheng et al. | Human posture reconstruction for through-the-wall radar imaging using convolutional neural networks | |
| Xu et al. | YOLO-MSFG: toward real-time detection of concealed objects in passive terahertz images | |
| Xu et al. | Shadow-background-noise 3D spatial decomposition using sparse low-rank Gaussian properties for video-SAR moving target shadow enhancement | |
| Hu et al. | An infrared target intrusion detection method based on feature fusion and enhancement | |
| Yang et al. | A novel deformable body partition model for MMW suspicious object detection and dynamic tracking | |
| Sage et al. | Security applications of computer vision | |
| Chen et al. | Space-based infrared aerial target detection method via interframe registration and spatial local contrast | |
| de Oliveira et al. | Generating synthetic short-range fmcw range-doppler maps using generative adversarial networks and deep convolutional autoencoders | |
| EP1678521B1 (en) | Detection of a concealed object | |
| Danso et al. | An optimal defect recognition security-based terahertz low resolution image system using deep learning network | |
| Liu et al. | Advances in automatic identification of road subsurface distress using ground penetrating radar: State of the art and future trends | |
| Liu et al. | Concealed weapon detection: A data fusion perspective | |
| Zhong et al. | Context guided reverse attention network with multiscale aggregation for infrared small target detection | |
| Kaul et al. | Automated vehicle detection and classification methods | |
| Lin et al. | Radar Spectral Maps Classification Based on Deep Learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |