CN110751079A - Article detection method, apparatus, system and computer readable storage medium - Google Patents
Article detection method, apparatus, system and computer readable storage medium Download PDFInfo
- Publication number
- CN110751079A CN110751079A CN201910981844.1A CN201910981844A CN110751079A CN 110751079 A CN110751079 A CN 110751079A CN 201910981844 A CN201910981844 A CN 201910981844A CN 110751079 A CN110751079 A CN 110751079A
- Authority
- CN
- China
- Prior art keywords
- image
- article
- identified
- detected
- gray
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 155
- 238000000034 method Methods 0.000 claims abstract description 28
- 238000007689 inspection Methods 0.000 claims abstract description 25
- 238000012549 training Methods 0.000 claims description 65
- 238000000605 extraction Methods 0.000 claims description 21
- 238000012545 processing Methods 0.000 claims description 18
- 238000004590 computer program Methods 0.000 claims description 9
- 238000002372 labelling Methods 0.000 claims description 9
- 238000013507 mapping Methods 0.000 claims description 8
- 238000003709 image segmentation Methods 0.000 claims description 7
- 238000003062 neural network model Methods 0.000 claims description 6
- 238000010586 diagram Methods 0.000 description 15
- 230000008569 process Effects 0.000 description 8
- 230000006870 function Effects 0.000 description 6
- 238000010801 machine learning Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 239000003086 colorant Substances 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 235000013305 food Nutrition 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 230000000877 morphologic effect Effects 0.000 description 2
- 238000012706 support-vector machine Methods 0.000 description 2
- 238000010521 absorption reaction Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 239000000919 ceramic Substances 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000003628 erosive effect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 239000002184 metal Substances 0.000 description 1
- 238000012821 model calculation Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000011368 organic material Substances 0.000 description 1
- 230000000149 penetrating effect Effects 0.000 description 1
- 230000003595 spectral effect Effects 0.000 description 1
- 238000013526 transfer learning Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N23/00—Investigating or analysing materials by the use of wave or particle radiation, e.g. X-rays or neutrons, not covered by groups G01N3/00 – G01N17/00, G01N21/00 or G01N22/00
- G01N23/02—Investigating or analysing materials by the use of wave or particle radiation, e.g. X-rays or neutrons, not covered by groups G01N3/00 – G01N17/00, G01N21/00 or G01N22/00 by transmitting the radiation through the material
- G01N23/06—Investigating or analysing materials by the use of wave or particle radiation, e.g. X-rays or neutrons, not covered by groups G01N3/00 – G01N17/00, G01N21/00 or G01N22/00 by transmitting the radiation through the material and measuring the absorption
- G01N23/10—Investigating or analysing materials by the use of wave or particle radiation, e.g. X-rays or neutrons, not covered by groups G01N3/00 – G01N17/00, G01N21/00 or G01N22/00 by transmitting the radiation through the material and measuring the absorption the material being confined in a container, e.g. in a luggage X-ray scanners
-
- G01V5/22—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/56—Extraction of image or video features relating to colour
Abstract
The disclosure relates to an article detection method, an article detection device, an article detection system and a computer readable storage medium, and relates to the technical field of computers. The method of the present disclosure comprises: acquiring an image to be identified generated by scanning an article to be detected by a security inspection machine; converting an image to be identified into a gray-scale image to be identified; and inputting the gray-scale image to be recognized into the article detection model, and determining whether each article to be detected in the gray-scale image to be recognized belongs to the forbidden articles.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a method, an apparatus, a system, and a computer-readable storage medium for detecting an article.
Background
With the economic development, the traffic is developed more and more, and people go out more conveniently and quickly. At the same time, the safety of the personnel is more non-negligible. In order to ensure the safety of travel, security check machines are arranged at the entrances of various rail transit, stations, customs, logistics centers and the like, so that the luggage carrying or consigning objects are subjected to security check, and the dangerous objects are forbidden to enter.
At present, security check machines mainly emit X-rays, process the X-rays into images with different colors according to the absorption degree of an object, and display the images on a screen. The security inspector can judge whether forbidden articles exist or not through experience by checking the X-ray perspective image.
Disclosure of Invention
The inventor finds that: at present, the security inspection of articles is mainly judged by security personnel according to experience. The personnel themselves have uncontrollable factors, such as various factors such as insufficient experience of security inspectors, work lacked, carelessness and the like, which can cause the omission of dangerous goods, and have safety risks.
One technical problem to be solved by the present disclosure is: the accuracy of the safety detection of the article is improved.
According to some embodiments of the present disclosure, there is provided an article detection method including: acquiring an image to be identified generated by scanning an article to be detected by a security inspection machine; converting an image to be identified into a gray-scale image to be identified; and inputting the gray-scale image to be recognized into the article detection model, and determining whether each article to be detected in the gray-scale image to be recognized belongs to the forbidden articles.
In some embodiments, converting the image to be identified into the grayscale map to be identified includes: removing noise in the image to be identified; removing the background in the image to be identified after the noise is removed to obtain a target area image; and converting the target area image into a gray map as a gray map to be identified according to the hue, saturation and brightness HSV characteristics of the target area image.
In some embodiments, removing the background in the image to be recognized after removing the noise to obtain the target area image includes: inputting the image to be recognized after the noise is removed into an image segmentation model, and determining the category to which each pixel point belongs, wherein the categories comprise: a foreground category or a background category; extracting a mask image of the image to be identified after noise removal according to the category to which each pixel point belongs; and performing bitwise AND operation on the mask image and the image to be identified to obtain a target area image.
In some embodiments, converting the target area image into a gray map according to HSV features of the target area image, wherein converting the target area image into the gray map as the gray map to be identified includes: converting red, green and blue RGB values of pixels in the target area image into HSV values; and converting the target area image into a gray scale image according to the tone value of the pixel, wherein the gray scale image is used as the gray scale image to be identified.
In some embodiments, inputting the grayscale map to be recognized into the article detection model, and determining whether each article to be detected in the grayscale map to be recognized belongs to an prohibited article includes: inputting the gray level image to be recognized into a feature extraction network in the article detection model to obtain the image features of the output gray level image to be recognized; wherein, the characteristic extraction network is a lightweight neural network model; inputting the image characteristics into a target detection network in the article detection model to obtain the output class information of each article to be detected; and determining whether the object to be detected belongs to the prohibited object or not according to the category information of each object to be detected.
In some embodiments, the method further comprises: and sending alarm information under the condition that each article to be detected contains forbidden articles.
In some embodiments, the article detection model further outputs position information of each article to be detected in the gray-scale image to be identified; the method further comprises the following steps: and mapping the identification information of whether each article to be detected belongs to the forbidden articles into the image to be identified according to the position information of each article to be detected in the gray-scale image to be identified, and sending the image to be identified with the identification information to a display device for displaying.
In some embodiments, the method further comprises: generating a first training sample image containing forbidden articles and non-forbidden articles, and labeling the category of each article in the first training sample image; training a target detection network in the article detection model by using the first training sample image; wherein, article detection model still includes: the characteristic extraction network determines parameters after pre-training; acquiring a real image generated by a security check machine as a second training sample image, and labeling the category of each article in the second training sample image; and adjusting parameters of a target detection network in the article detection model by using the second training sample image to finish training the article detection model.
According to further embodiments of the present disclosure, there is provided an article detection apparatus including: the acquisition module is used for acquiring an image to be identified generated by scanning an article to be detected by the security inspection machine; the image processing module is used for converting the image to be identified into a gray image to be identified; and the detection module is used for inputting the gray-scale image to be recognized into the article detection model and determining whether each article to be detected in the gray-scale image to be recognized belongs to the prohibited article or not.
In some embodiments, the image processing module is configured to remove noise in the image to be identified; removing the background in the image to be identified after the noise is removed to obtain a target area image; and converting the target area image into a gray map as a gray map to be identified according to the hue, saturation and brightness HSV characteristics of the target area image.
In some embodiments, the image processing module is configured to input the image to be recognized after removing noise into an image segmentation model, and determine a category to which each pixel belongs, where the category includes: a foreground category or a background category; extracting a mask image of the image to be identified after noise removal according to the category to which each pixel point belongs; and performing bitwise AND operation on the mask image and the image to be identified to obtain a target area image.
In some embodiments, the image processing module is configured to convert red, green, and blue RGB values of pixels in the target area image into HSV values; and converting the target area image into a gray scale image according to the tone value of the pixel, wherein the gray scale image is used as the gray scale image to be identified.
In some embodiments, the detection module is configured to input the grayscale image to be identified into a feature extraction network in the article detection model, to obtain an image feature of the output grayscale image to be identified; wherein, the characteristic extraction network is a lightweight neural network model; inputting the image characteristics into a target detection network in the article detection model to obtain the output class information of each article to be detected; and determining whether the object to be detected belongs to the prohibited object or not according to the category information of each object to be detected.
In some embodiments, the apparatus further comprises: and the alarm module is used for sending alarm information under the condition that the articles to be detected contain forbidden articles.
In some embodiments, the article detection model further outputs position information of each article to be detected in the gray-scale image to be identified; the device also includes: and the display module is used for mapping the identification information of whether each article to be detected belongs to the forbidden articles into the image to be identified according to the position information of each article to be detected in the gray-scale image to be identified, and mapping the image to be identified with the identification information.
In some embodiments, the apparatus further comprises: the training module is used for generating a first training sample image containing forbidden articles and non-forbidden articles and labeling the category of each article in the first training sample image; training a target detection network in the article detection model by using the first training sample image; wherein, article detection model still includes: the characteristic extraction network determines parameters after pre-training; acquiring a real image generated by a security check machine as a second training sample image, and labeling the category of each article in the second training sample image; and adjusting parameters of a target detection network in the article detection model by using the second training sample image to finish training the article detection model.
According to still other embodiments of the present disclosure, there is provided an article detection apparatus including: a processor; and a memory coupled to the processor for storing instructions that, when executed by the processor, cause the processor to perform the item detection method of any of the preceding embodiments.
According to still further embodiments of the present disclosure, there is provided a computer-readable non-transitory storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the article detection method of any of the foregoing embodiments.
According to still further embodiments of the present disclosure, there is provided an article detection system including: the article detection device and the security check machine of any embodiment are used for scanning an image to be identified generated by an article to be detected.
In some embodiments, the system further comprises: and the display device is used for receiving the image to be identified with the identification information sent by the article detection device and displaying the image.
In the method, the security check machine scans the to-be-identified image generated by the to-be-detected object to obtain the to-be-identified image, and the to-be-identified image is converted into the gray image to be input into the object detection model, so that whether the to-be-detected object belongs to the prohibited object or not is identified. The method adopts the machine learning method to identify the images to be identified, can accurately identify a large number of images in real time without interruption, and improves the accuracy of the safety detection of articles. In addition, different articles in the image to be identified generated by the security inspection machine have obvious color difference, the image to be identified is converted into a gray-scale image, and under the condition of keeping the difference, the calculation complexity in the article detection process can be reduced, and the efficiency of security detection is improved.
Other features of the present disclosure and advantages thereof will become apparent from the following detailed description of exemplary embodiments thereof, which proceeds with reference to the accompanying drawings.
Drawings
In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 illustrates a flow diagram of an item detection method of some embodiments of the present disclosure.
Fig. 2 shows a flow diagram of an item detection method of further embodiments of the present disclosure.
Fig. 3 shows a schematic structural view of an article detection apparatus of some embodiments of the present disclosure.
Fig. 4 shows a schematic structural view of an article detection apparatus according to further embodiments of the present disclosure.
Fig. 5 shows a schematic structural diagram of an article detection apparatus according to further embodiments of the present disclosure.
Fig. 6 illustrates a schematic structural view of an item detection system of some embodiments of the present disclosure.
Detailed Description
The technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are only a part of the embodiments of the present disclosure, and not all of the embodiments. The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or uses. All other embodiments, which can be derived by a person skilled in the art from the embodiments disclosed herein without making any creative effort, shall fall within the protection scope of the present disclosure.
The scheme is provided for solving the problem that the security inspection of the existing articles is mainly judged by a security officer according to experience and has risks such as missing inspection. Some embodiments of the article detection method of the present disclosure are described below in conjunction with fig. 1.
Fig. 1 is a flow chart of some embodiments of an article detection method of the present disclosure. As shown in fig. 1, the method of this embodiment includes: steps S102 to S106.
In step S102, an image to be recognized generated by scanning an object to be detected by a security inspection machine is acquired.
After the article to be detected is conveyed into the security inspection machine, the security inspection machine can scan the article to be detected by adopting X rays or other forms to form images. The security check machines deployed in the market at present are all matched with display screens, and can synchronously display image information generated by scanning an article to be detected on the screens through signals such as High Definition Multimedia Interface (HDMI) signals or Video Graphics Array (VGA) signals. Can acquire the signal that security check machine generated through this disclosed article detection device's collection module, further convert the signal into the video source, read the image frame in the video source to obtain the image of waiting to discern. For example, a frame of image can be extracted from a video source every preset frame number to serve as an image to be identified, so that all the images to be identified can cover all the articles to be detected passing through the security inspection machine, and the omission is avoided. For example, OpenCV may be used to read a video source to obtain an image to be recognized.
In step S104, the image to be recognized is converted into a grayscale map to be recognized.
The image to be identified is, for example, an X-ray scanning image, and different articles have obvious color differences due to different penetrating power of the X-ray to the different articles, and are distributed in a certain rule. For example, organic materials such as foods show orange color; ceramics, etc. appear green; the metal appears blue. The image to be recognized is converted into the gray-scale image to obtain the gray-scale image to be recognized, the difference of colors of different articles can be reserved, and meanwhile, the data volume of pixel values of pixel points can be reduced, so that the subsequent image recognition efficiency is improved.
Further, the image to be recognized may be converted into a grayscale map according to HSV (hue, saturation, brightness) features of the image to be recognized. The image to be recognized can be converted from an RGB color space to an HSV color space, and the H component in the image can be used for converting the color of the pixel point. The H parameter represents the color information, i.e. the position of the spectral color. The parameter is expressed by an angular measure, red, green and blue being separated by 120 degrees. The gray level image is converted through the H parameter, so that the color difference of different articles can be kept as much as possible, and the accuracy of subsequent identification is improved.
That is, the RGB (red, green, blue) values of the pixels of the image to be recognized may be first converted into HSV values, and the image to be recognized may be converted into a gray-scale image according to the hue values (H values) of the pixels. The H value can be normalized by 0-255 to obtain a mapped gray scale image. For example, the ratio of the H value to 240 is multiplied by 255 to obtain a gray value.
Before converting the image to be recognized into the gray-scale image to be recognized, the noise in the image to be recognized can be removed firstly. For example, the image to be recognized may be preprocessed by gaussian low-pass filtering or morphological operations, so as to remove noise points in the image to be recognized. Morphological operations such as expansion, erosion, opening and closing operations of images, and gaussian low-pass filtering are all prior art and are not described herein.
Further, the background in the image to be recognized after the noise is removed can be removed, so that the target area image is obtained. And converting the target area image into a gray image according to the HSV characteristics of the target area image, and taking the gray image as a gray image to be identified. In some embodiments, the image to be recognized after removing noise is input into an image segmentation model, and the category to which each pixel point belongs is determined, where the category includes: a foreground category or a background category; extracting a mask image of the image to be identified after noise removal according to the category to which each pixel point belongs; and performing bitwise AND operation on the mask image and the image to be identified to obtain a target area image.
The image segmentation model may be, for example, an SVM (support vector machine) model or an existing model such as FCN (full volume connectivity network, full volume and neural network), and will not be described herein again. And separating the foreground and the background of the image to be recognized by using the image segmentation model to obtain a mask image, and then fusing the mask image with the image to be recognized, wherein the obtained target area image only contains all the objects to be detected, the objects and the noise are removed, and the accuracy of subsequent object detection is improved.
The process of converting the target area image into the gray scale image to be recognized may refer to the foregoing embodiments, for example, converting RGB values of pixels in the target area image into HSV values; and converting the target area image into a gray scale image according to the tone value of the pixel, wherein the gray scale image is used as the gray scale image to be identified.
In step S106, the grayscale map to be recognized is input into the article detection model, and it is determined whether each article to be detected in the grayscale map to be recognized belongs to an prohibited article.
In some embodiments, the item detection model may include a feature extraction network and an object detection network. Inputting the gray level image to be recognized into a feature extraction network in the article detection model to obtain the image features of the output gray level image to be recognized; inputting the image characteristics into a target detection network in the article detection model to obtain the output position information and the output category information of each article to be detected, and determining whether the article to be detected belongs to the contraband article or not according to the category information of each article to be detected. Because the color of the gray level image to be recognized is relatively single and the data volume of the pixel value is small, the feature extraction network can adopt a lightweight neural network model, the model calculation amount can be reduced, and the processing speed is increased. For example, the item detection model may be a MobileNet-SSD model, not limited to the illustrated example. The method comprises the following steps that MobileNet (a light weight neural network model facing a mobile terminal) serves as a feature extraction network, and SSD (single-shot multi-frame detector) serves as a target detection network. The article detection model may adopt an existing model, and is not described in detail herein.
The position information of the object to be detected is, for example, coordinate information. The category information may be a category determined when the article detection model is trained, for example, the article to be detected may be divided into an prohibited article and a non-prohibited article, and an actual category of the article to be detected may also be identified, for example, food, a tool, and the like, and then mapped into the prohibited article and the non-prohibited article according to the actual category.
The article detection model can be trained offline in advance, and the trained model can be used for real-time article detection. The training process of the article detection model includes, for example: and generating a first training sample image containing forbidden articles and non-forbidden articles, and labeling the category of each article in the first training sample image. Training a target detection network in the article detection model by using the first training sample image; wherein, article detection model still includes: and (4) a characteristic extraction network, wherein the characteristic extraction network determines parameters after pre-training. And acquiring a real image generated by the security check machine as a second training sample image, and labeling the category of each article in the second training sample image. And adjusting parameters of a target detection network in the article detection model by using the second training sample image to finish training the article detection model.
Because the training model needs a large amount of training sample images, the training sample images can be generated by utilizing the acquired images of various articles under the condition that the images generated by the security inspection machine are not easy to acquire in practical application. For example, images of various contraband and non-contraband may be captured with reference to actual conditions (e.g., various images captured from a network). Further, a first training sample image containing contraband and non-contraband may be generated by an image fusion technique. The images of the articles can be preprocessed by scaling, rotation, blurring, color conversion and the like before fusion, so that the generated first training sample image is closer to a real image.
The feature extraction network in the article detection model may be pre-trained with a public data set (e.g., a COCO data set) to obtain network parameters. Further, parameters of the feature extraction network can be reserved in a transfer learning mode, and the first training sample image is used for training the target detection network in the article detection model. For example, inputting the first training sample image into the article detection model to obtain the category of each article in the output first training sample image, calculating a first loss function according to the output category of each article and the labeled category of each article, adjusting the parameters of the target detection network according to the first loss function, and repeating the above processes until preset conditions are met to obtain the initially trained article detection model. And further, the real image generated by the security inspection machine is used as a second training sample image to perform fine adjustment on the article detection model. Namely, on the basis of the primarily trained article detection model, inputting a second training sample image for retraining again, and finally obtaining the trained article detection model. The first training sample image and the second training sample image may also be subjected to image processing according to the method of the foregoing embodiment to finally obtain a corresponding grayscale, and then used for training the article detection model, which is not described herein again.
According to the training process, effective training of the article detection model can be completed under the condition that the number of real images generated by the security inspection machine is small, and the accuracy of article detection model detection is guaranteed. Under the condition that the real image generated by the security inspection machine is enough, the training of the article detection model can be completed by directly utilizing the real image.
In the method of the embodiment, the security check machine scans the to-be-identified image generated by the to-be-detected object to obtain the to-be-identified image, and the to-be-identified image is converted into the gray scale image to be input into the object detection model, so that whether the to-be-detected object belongs to the contraband object or not is identified. The method of the embodiment adopts a machine learning method to identify the images to be identified, can accurately identify a large number of images in real time without interruption, and improves the accuracy of the security detection of the articles. In addition, different articles in the image to be identified generated by the security inspection machine have obvious color difference, the image to be identified is converted into a gray-scale image, and under the condition of keeping the difference, the calculation complexity in the article detection process can be reduced, and the efficiency of security detection is improved.
Further embodiments of the article detection method of the present disclosure are described below in conjunction with fig. 2.
FIG. 2 is a flow chart of further embodiments of the article detection method of the present disclosure. As shown in fig. 2, the method of this embodiment includes: steps S202 to S210.
In step S202, an image to be recognized generated by scanning an object to be detected by a security inspection machine is acquired.
In step S204, the image to be recognized is converted into a grayscale map to be recognized.
In step S206, the grayscale map to be recognized is input into the object detection model, and it is determined whether each object to be detected in the grayscale map to be recognized belongs to an prohibited object.
Steps S202 to S206 can be explained with reference to the description of the foregoing embodiment.
In step S208, in a case where it is determined that each of the items to be detected contains an prohibited item, alarm information is issued.
The alarm can be automatically given out under the condition of detecting out forbidden articles, so that a security inspector is reminded of processing, and stop information can be sent out to the security inspection machine, so that the forbidden articles are processed.
In step S210, according to the position information of each object to be detected in the grayscale map to be recognized, mapping the identification information of whether each object to be detected belongs to the contraband object to the image to be recognized, and mapping the image to be recognized with the identification information.
Steps S208 and S210 may be executed in parallel, not in sequence.
The positions of the objects to be detected in the gray-scale image to be recognized correspond to the positions of the objects to be recognized in the image to be recognized, identification information of forbidden objects and non-forbidden objects can be directly mapped into the image to be recognized, and the image with the identification information can be further sent to a display device corresponding to a security check machine to be displayed so that a security check worker can check the image conveniently.
In the present disclosure, the method of the above embodiment may be implemented by an article detection device, which may be externally installed on a security inspection system (such as a security inspection machine and a display device), and acquire an X-ray perspective image of an article through an acquisition module, and transmit the X-ray perspective image to an image processing module and a detection module. The detection module detects the image by adopting a machine learning model. When the contraband is detected, an alarm mechanism is triggered to inform the staff to carry out recheck inspection. When the article detection device is integrally used as an embedded module, the article detection device can be connected to any security check machine, the original manual decision work is submitted to algorithm application for judgment, intelligent identification and alarm are carried out, the intelligent transformation of the existing gate is realized, the article detection efficiency and accuracy are improved, and the labor cost is reduced.
The present disclosure also provides an article detection apparatus, described below in conjunction with fig. 3.
FIG. 3 is a block diagram of some embodiments of an article detection device of the present disclosure. As shown in fig. 3, the apparatus 30 of this embodiment includes: an acquisition module 302, an image processing module 304, and a detection module 306.
The acquisition module 302 is configured to acquire an image to be identified, which is generated by scanning an article to be detected by a security inspection machine.
And the image processing module 304 is used for converting the image to be identified into a gray map to be identified.
In some embodiments, the image processing module 304 is used to remove noise in the image to be identified; removing the background in the image to be identified after the noise is removed to obtain a target area image; and converting the target area image into a gray map as a gray map to be identified according to the hue, saturation and brightness HSV characteristics of the target area image.
In some embodiments, the image processing module 304 is configured to input the image to be recognized after removing the noise into an image segmentation model, and determine a category to which each pixel belongs, where the category includes: a foreground category or a background category; extracting a mask image of the image to be identified after noise removal according to the category to which each pixel point belongs; and performing bitwise AND operation on the mask image and the image to be identified to obtain a target area image.
In some embodiments, the image processing module 304 is configured to convert the red, green, and blue RGB values of the pixels in the target area image into HSV values; and converting the target area image into a gray scale image according to the tone value of the pixel, wherein the gray scale image is used as the gray scale image to be identified.
The detection module 306 is configured to input the grayscale image to be recognized into the article detection model, and determine whether each article to be detected in the grayscale image to be recognized belongs to an prohibited article.
In some embodiments, the detection module 306 is configured to input the grayscale image to be recognized into a feature extraction network in the article detection model, so as to obtain an image feature of the output grayscale image to be recognized; wherein, the characteristic extraction network is a lightweight neural network model; inputting the image characteristics into a target detection network in the article detection model to obtain the output class information of each article to be detected; and determining whether the object to be detected belongs to the prohibited object or not according to the category information of each object to be detected.
In some embodiments, the apparatus 30 further comprises: and the alarm module 308 is configured to send alarm information when it is determined that each object to be detected includes an illegal object.
In some embodiments, the article detection model further outputs position information of each article to be detected in the gray-scale image to be identified; the apparatus 30 further comprises: the display module 310 is configured to map, according to the position information of each object to be detected in the grayscale map to be recognized, the identification information of whether each object to be detected belongs to an prohibited object into the image to be recognized, and map the image to be recognized with the identification information.
In some embodiments, the apparatus 30 further comprises: the training module 312 is configured to generate a first training sample image containing prohibited articles and non-prohibited articles, and label categories of the articles in the first training sample image; training a target detection network in the article detection model by using the first training sample image; wherein, article detection model still includes: the characteristic extraction network determines parameters after pre-training; acquiring a real image generated by a security check machine as a second training sample image, and labeling the category of each article in the second training sample image; and adjusting parameters of a target detection network in the article detection model by using the second training sample image to finish training the article detection model.
The article detection apparatus in the embodiments of the present disclosure may each be implemented by various computing devices or computer systems, which are described below in conjunction with fig. 4 and 5.
Fig. 4 is a block diagram of some embodiments of an article detection device of the present disclosure. As shown in fig. 4, the apparatus 40 of this embodiment includes: a memory 410 and a processor 420 coupled to the memory 410, the processor 420 configured to perform a method of item detection in any of the embodiments of the present disclosure based on instructions stored in the memory 410.
Fig. 5 is a block diagram of further embodiments of the article detection apparatus of the present disclosure. As shown in fig. 5, the apparatus 50 of this embodiment includes: memory 510 and processor 520 are similar to memory 410 and processor 420, respectively. An input output interface 530, a network interface 540, a storage interface 550, and the like may also be included. These interfaces 530, 540, 550 and the connections between the memory 510 and the processor 520 may be, for example, via a bus 560. The input/output interface 530 provides a connection interface for input/output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 540 provides a connection interface for various networking devices, such as a database server or a cloud storage server. The storage interface 550 provides a connection interface for external storage devices such as an SD card and a usb disk.
The present disclosure also provides an article detection system, described below in conjunction with fig. 6.
Fig. 6 is a block diagram of some embodiments of the article detection system of the present disclosure. As shown in fig. 6, the system 6 of this embodiment includes: the article detection device 30/40/50 of any of the preceding embodiments, and the security check machine 62.
The security check machine 62 is used to scan the image to be identified generated by the item to be detected.
In some embodiments, the system 6 further comprises: and the display device 64 is used for receiving the image to be identified with the identification information sent by the article detection device and displaying the image.
As will be appreciated by one skilled in the art, embodiments of the present disclosure may be provided as a method, system, or computer program product. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product embodied on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only exemplary of the present disclosure and is not intended to limit the present disclosure, so that any modification, equivalent replacement, or improvement made within the spirit and principle of the present disclosure should be included in the scope of the present disclosure.
Claims (15)
1. An item detection method, comprising:
acquiring an image to be identified generated by scanning an article to be detected by a security inspection machine;
converting the image to be recognized into a gray scale image to be recognized;
inputting the gray-scale image to be recognized into an article detection model, and determining whether each article to be detected in the gray-scale image to be recognized belongs to prohibited articles.
2. The article detection method according to claim 1,
the converting the image to be recognized into the gray-scale image to be recognized comprises the following steps:
removing noise in the image to be identified;
removing the background in the image to be identified after the noise is removed to obtain a target area image;
and converting the target area image into a gray map as a gray map to be identified according to the hue, saturation and brightness HSV characteristics of the target area image.
3. The item detection method according to claim 2,
the step of removing the background in the image to be recognized after the noise is removed to obtain the target area image comprises the following steps:
inputting the image to be recognized after the noise is removed into an image segmentation model, and determining the category to which each pixel point belongs, wherein the categories comprise: a foreground category or a background category;
extracting the mask image of the image to be recognized after the noise is removed according to the category of each pixel point;
and performing pixel value AND operation on the mask image and the image to be identified according to bits to obtain the target area image.
4. The item detection method according to claim 2,
the converting the target area image into a gray scale image according to the HSV characteristics of the target area image, wherein the gray scale image to be identified comprises the following steps:
converting the red, green and blue RGB values of the pixels in the target area image into HSV values;
and converting the target area image into a gray scale image according to the tone value of the pixel, wherein the gray scale image is used as a gray scale image to be identified.
5. The article detection method according to claim 1,
the step of inputting the gray level image to be recognized into an article detection model and determining whether each article to be detected in the gray level image to be recognized belongs to an forbidden article comprises the following steps:
inputting the gray level image to be recognized into a feature extraction network in the article detection model to obtain the image features of the output gray level image to be recognized; wherein, the feature extraction network is a lightweight neural network model;
inputting the image characteristics into a target detection network in the article detection model to obtain output class information of each article to be detected;
and determining whether the to-be-detected articles belong to forbidden articles according to the category information of the to-be-detected articles.
6. The item detection method of claim 1, further comprising:
and sending alarm information under the condition that the articles to be detected contain forbidden articles.
7. The article detection method according to claim 1,
the article detection model also outputs the position information of each article to be detected in the gray level image to be identified;
the method further comprises the following steps:
and mapping the identification information of whether each article to be detected belongs to the forbidden articles into the image to be identified according to the position information of each article to be detected in the gray-scale image to be identified, and sending the image to be identified with the identification information to a display device for displaying.
8. The item detection method of claim 1, further comprising:
generating a first training sample image containing forbidden articles and non-forbidden articles, and labeling the category of each article in the first training sample image;
training a target detection network in the article detection model by using the first training sample image; wherein the item detection model further comprises: a feature extraction network, wherein the feature extraction network is subjected to pre-training to determine parameters;
acquiring a real image generated by a security check machine as a second training sample image, and labeling the category of each article in the second training sample image;
and adjusting parameters of a target detection network in the article detection model by using the second training sample image to finish training the article detection model.
9. An article detection device comprising:
the acquisition module is used for acquiring an image to be identified generated by scanning an article to be detected by the security inspection machine;
the image processing module is used for converting the image to be identified into a gray image to be identified;
and the detection module is used for inputting the gray-scale image to be recognized into an article detection model and determining whether each article to be detected in the gray-scale image to be recognized belongs to an illegal article.
10. The item detection apparatus of claim 9, further comprising:
and the alarm module is used for sending alarm information under the condition that each article to be detected contains forbidden articles.
11. The item detecting device according to claim 9,
the article detection model also outputs the position information of each article to be detected in the gray level image to be identified;
the device further comprises:
and the display module is used for mapping the identification information of whether each article to be detected belongs to the forbidden articles into the image to be identified according to the position information of each article to be detected in the gray-scale image to be identified, and mapping the image to be identified with the identification information.
12. An article detection device comprising:
a processor; and
a memory coupled to the processor for storing instructions that, when executed by the processor, cause the processor to perform the item detection method of any of claims 1-8.
13. A computer-readable non-transitory storage medium having a computer program stored thereon, wherein the program when executed by a processor implements the steps of the method of any one of claims 1-8.
14. An item detection system comprising: the item detection apparatus of any one of claims 9-12; and
and the security inspection machine is used for scanning the image to be identified generated by the article to be detected.
15. The item detection system of claim 14, further comprising:
and the display device is used for receiving the image to be identified with the identification information sent by the article detection device and displaying the image.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910981844.1A CN110751079A (en) | 2019-10-16 | 2019-10-16 | Article detection method, apparatus, system and computer readable storage medium |
| PCT/CN2020/116728 WO2021073370A1 (en) | 2019-10-16 | 2020-09-22 | Article inspection method, apparatus and system and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910981844.1A CN110751079A (en) | 2019-10-16 | 2019-10-16 | Article detection method, apparatus, system and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110751079A true CN110751079A (en) | 2020-02-04 |
Family
ID=69278456
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910981844.1A Pending CN110751079A (en) | 2019-10-16 | 2019-10-16 | Article detection method, apparatus, system and computer readable storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN110751079A (en) |
| WO (1) | WO2021073370A1 (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111401182A (en) * | 2020-03-10 | 2020-07-10 | 北京海益同展信息科技有限公司 | Image detection method and device for feeding fence |
| WO2021073370A1 (en) * | 2019-10-16 | 2021-04-22 | 北京海益同展信息科技有限公司 | Article inspection method, apparatus and system and computer readable storage medium |
| CN113505771A (en) * | 2021-09-13 | 2021-10-15 | 华东交通大学 | Double-stage article detection method and device |
| CN113740356A (en) * | 2020-05-29 | 2021-12-03 | 同方威视技术股份有限公司 | Image acquisition method, image acquisition device and non-volatile computer-readable storage medium |
| CN113822859A (en) * | 2021-08-25 | 2021-12-21 | 日立楼宇技术（广州）有限公司 | Article detection method, system, device and storage medium based on image recognition |
| CN114548230A (en) * | 2022-01-25 | 2022-05-27 | 西安电子科技大学广州研究院 | X-ray contraband detection method based on RGB color separation double-path feature fusion |
| CN115761460A (en) * | 2023-01-10 | 2023-03-07 | 北京市农林科学院智能装备技术研究中心 | Greenhouse risk identification method and device, electronic equipment and storage medium |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113220927A (en) * | 2021-05-08 | 2021-08-06 | 百度在线网络技术（北京）有限公司 | Image detection method, device, equipment and storage medium |
| CN113221737B (en) * | 2021-05-11 | 2023-09-05 | 杭州海康威视数字技术股份有限公司 | Material information determining method, device, equipment and storage medium |
| CN113515248B (en) * | 2021-05-24 | 2024-03-29 | 浙江华视智检科技有限公司 | Display method, control equipment and computer readable storage medium of security inspection instrument |
| CN113506283A (en) * | 2021-07-26 | 2021-10-15 | 浙江大华技术股份有限公司 | Image processing method and device, storage medium and electronic device |
| CN114332543B (en) * | 2022-01-10 | 2023-02-14 | 成都智元汇信息技术股份有限公司 | Multi-template security inspection image identification method, equipment and medium |
| CN115797857B (en) * | 2022-11-07 | 2023-08-01 | 北京声迅电子股份有限公司 | Travel event determining method, security inspection method and event management method |
| CN117272124B (en) * | 2023-11-23 | 2024-03-12 | 湖南苏科智能科技有限公司 | Chemical classification method, system, equipment and storage medium based on energy level |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105021529A (en) * | 2015-06-11 | 2015-11-04 | 浙江水利水电学院 | Spectrum information and image information fusing crop plant disease and insect pest identifying and distinguishing method |
| CN106503719A (en) * | 2016-09-27 | 2017-03-15 | 深圳增强现实技术有限公司 | A kind of object color is extracted and detection method and device |
| CN106932414A (en) * | 2015-12-29 | 2017-07-07 | 同方威视技术股份有限公司 | Inspection and quarantine inspection system and its method |
| CN108198207A (en) * | 2017-12-22 | 2018-06-22 | 湖南源信光电科技股份有限公司 | Multiple mobile object tracking based on improved Vibe models and BP neural network |
| CN109978827A (en) * | 2019-02-25 | 2019-07-05 | 平安科技（深圳）有限公司 | Violated object recognition methods, device, equipment and storage medium based on artificial intelligence |
| CN110020647A (en) * | 2018-01-09 | 2019-07-16 | 杭州海康威视数字技术股份有限公司 | A kind of contraband object detection method, device and computer equipment |
| CN110018524A (en) * | 2019-01-28 | 2019-07-16 | 同济大学 | A kind of X-ray safety check contraband recognition methods of view-based access control model-attribute |
| CN110096960A (en) * | 2019-04-03 | 2019-08-06 | 罗克佳华科技集团股份有限公司 | Object detection method and device |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080152082A1 (en) * | 2006-08-16 | 2008-06-26 | Michel Bouchard | Method and apparatus for use in security screening providing incremental display of threat detection information and security system incorporating same |
| CN110751079A (en) * | 2019-10-16 | 2020-02-04 | 北京海益同展信息科技有限公司 | Article detection method, apparatus, system and computer readable storage medium |
-
2019
- 2019-10-16 CN CN201910981844.1A patent/CN110751079A/en active Pending
-
2020
- 2020-09-22 WO PCT/CN2020/116728 patent/WO2021073370A1/en active Application Filing
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105021529A (en) * | 2015-06-11 | 2015-11-04 | 浙江水利水电学院 | Spectrum information and image information fusing crop plant disease and insect pest identifying and distinguishing method |
| CN106932414A (en) * | 2015-12-29 | 2017-07-07 | 同方威视技术股份有限公司 | Inspection and quarantine inspection system and its method |
| CN106503719A (en) * | 2016-09-27 | 2017-03-15 | 深圳增强现实技术有限公司 | A kind of object color is extracted and detection method and device |
| CN108198207A (en) * | 2017-12-22 | 2018-06-22 | 湖南源信光电科技股份有限公司 | Multiple mobile object tracking based on improved Vibe models and BP neural network |
| CN110020647A (en) * | 2018-01-09 | 2019-07-16 | 杭州海康威视数字技术股份有限公司 | A kind of contraband object detection method, device and computer equipment |
| CN110018524A (en) * | 2019-01-28 | 2019-07-16 | 同济大学 | A kind of X-ray safety check contraband recognition methods of view-based access control model-attribute |
| CN109978827A (en) * | 2019-02-25 | 2019-07-05 | 平安科技（深圳）有限公司 | Violated object recognition methods, device, equipment and storage medium based on artificial intelligence |
| CN110096960A (en) * | 2019-04-03 | 2019-08-06 | 罗克佳华科技集团股份有限公司 | Object detection method and device |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021073370A1 (en) * | 2019-10-16 | 2021-04-22 | 北京海益同展信息科技有限公司 | Article inspection method, apparatus and system and computer readable storage medium |
| CN111401182A (en) * | 2020-03-10 | 2020-07-10 | 北京海益同展信息科技有限公司 | Image detection method and device for feeding fence |
| CN111401182B (en) * | 2020-03-10 | 2023-12-08 | 京东科技信息技术有限公司 | Image detection method and device for feeding rail |
| CN113740356A (en) * | 2020-05-29 | 2021-12-03 | 同方威视技术股份有限公司 | Image acquisition method, image acquisition device and non-volatile computer-readable storage medium |
| CN113822859A (en) * | 2021-08-25 | 2021-12-21 | 日立楼宇技术（广州）有限公司 | Article detection method, system, device and storage medium based on image recognition |
| CN113822859B (en) * | 2021-08-25 | 2024-02-27 | 日立楼宇技术（广州）有限公司 | Article detection method, system, device and storage medium based on image recognition |
| CN113505771A (en) * | 2021-09-13 | 2021-10-15 | 华东交通大学 | Double-stage article detection method and device |
| CN113505771B (en) * | 2021-09-13 | 2021-12-03 | 华东交通大学 | Double-stage article detection method and device |
| CN114548230A (en) * | 2022-01-25 | 2022-05-27 | 西安电子科技大学广州研究院 | X-ray contraband detection method based on RGB color separation double-path feature fusion |
| CN114548230B (en) * | 2022-01-25 | 2024-03-26 | 西安电子科技大学广州研究院 | X-ray contraband detection method based on RGB color separation double-path feature fusion |
| CN115761460A (en) * | 2023-01-10 | 2023-03-07 | 北京市农林科学院智能装备技术研究中心 | Greenhouse risk identification method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2021073370A1 (en) | 2021-04-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110751079A (en) | Article detection method, apparatus, system and computer readable storage medium | |
| CN110543857A (en) | Contraband identification method, device and system based on image analysis and storage medium | |
| KR101969022B1 (en) | Image analysis apparatus and method | |
| CN109977877B (en) | Intelligent auxiliary image judging method, system and system control method for security inspection | |
| CN105701519A (en) | Realogram scene analysis of images: superpixel scene analysis | |
| Sheikh et al. | Traffic sign detection and classification using colour feature and neural network | |
| CN106951898B (en) | Vehicle candidate area recommendation method and system and electronic equipment | |
| CN110488368B (en) | Contraband identification method and device based on dual-energy X-ray security inspection machine | |
| CN109829458B (en) | Method for automatically generating log file for recording system operation behavior in real time | |
| US11182637B2 (en) | X-ray image processing system and method, and program therefor | |
| CN112712093A (en) | Security check image identification method and device, electronic equipment and storage medium | |
| CN112967289A (en) | Security check package matching method, system, equipment and storage medium | |
| CN112016545A (en) | Image generation method and device containing text | |
| Agarwal et al. | An efficient algorithm for automatic car plate detection & recognition | |
| CN113378648A (en) | Artificial intelligence port and wharf monitoring method | |
| CN112069907A (en) | X-ray machine image recognition method, device and system based on example segmentation | |
| KR102032796B1 (en) | Image analysis apparatus and method | |
| CN108596244A (en) | A kind of high spectrum image label noise detecting method based on spectrum angle density peaks | |
| KR20190059083A (en) | Apparatus and method for recognition marine situation based image division | |
| KR101990123B1 (en) | Image analysis apparatus and method | |
| KR102158967B1 (en) | Image analysis apparatus, image analysis method and recording medium | |
| CN110569716A (en) | Goods shelf image copying detection method | |
| CN107463934A (en) | A kind of tunnel slot detection method and device | |
| CN105448095A (en) | Yellow label car detection method and device | |
| CN106845498A (en) | With reference to the single width mountain range remote sensing images landslide detection method of elevation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200204 |
|
| RJ01 | Rejection of invention patent application after publication |