WO2022257892A1

WO2022257892A1 - Information association method and apparatus, security inspection device, and storage medium

Info

Publication number: WO2022257892A1
Application number: PCT/CN2022/097192
Authority: WO
Inventors: 唐政; 张俊力; 高其涛
Original assignee: 杭州海康威视数字技术股份有限公司
Priority date: 2021-06-09
Filing date: 2022-06-06
Publication date: 2022-12-15
Also published as: CN113267828A

Abstract

An information association method and apparatus, a security inspection device, and a storage medium. The method comprises: obtaining a monitoring image of an entrance of a security inspection device (S101); determining a package to be inspected and a target person to which said package belongs in the monitoring image (S102); determining, according to a placement position of said package on a conveyor belt of the security inspection device and the position of an inspection area of the security inspection device, a target X-ray image acquired by the security inspection device, wherein the target X-ray image comprises said package (S103); and establishing a correspondence among said package, the target person, and the target X-ray image (S104). Since the association process is directly performed by the security inspection device on the basis of the acquired monitoring image and X-ray image, the method does not need manual participation, such that a time required by manual association is saved, the possibility of errors in manual association in a scenario with relatively high human traffic or a relatively complex environment is reduced, and the accuracy of information association is improved.

Description

An information association method, device, security inspection equipment and storage medium

This application claims the priority of the Chinese patent application with the application number 202110644127.7 submitted to the China Patent Office on June 9, 2021. incorporated in this application.

technical field

The present application relates to the technical field of the Internet, in particular to an information association method, device, security inspection equipment and storage medium.

Background technique

With the continuous progress of society and the continuous improvement of people's attention to the safety of key places, the application of security inspection equipment is becoming more and more extensive. Such as railway stations, airports and other places are equipped with security equipment. Personnel place the package carried on the conveyor belt of the security inspection equipment, and the security inspection equipment uses the conveyor belt to transmit the package to the X-ray (X-ray) detection area, thereby completing the abnormal detection of the items in the package, such as detecting whether there are prohibited items in the package .

In the above detection process, the detection of the package information on the transmission belt and the personnel monitoring process are independent of each other, which makes the correlation between the detected package information and the monitored personnel information poor. Therefore, when it is detected that the package contains abnormal items, it is necessary to manually determine the target person associated with the abnormal package. For example, when a package is detected to contain prohibited items, the inspectors will determine the owner of the package through manual association, that is, the above-mentioned target person, thereby reducing the potential safety hazard caused by the target person bringing the prohibited items onto the plane or train . However, for scenes with a large flow of people or a more complex environment, the time required for manual association and the difficulty of association will be greatly increased, which will seriously affect the efficiency and accuracy of information association.

Contents of the invention

The purpose of the embodiments of the present application is to provide an information association method, device, security inspection equipment, and storage medium, so as to improve the efficiency and accuracy of information association. The specific technical scheme is as follows:

An embodiment of the present application provides an information association method, which is applied to security inspection equipment, and the method includes:

Acquiring the monitoring image at the entrance of the security inspection equipment;

determining the package to be detected in the surveillance image and the target person to whom the package to be detected belongs;

According to the placement position of the package to be detected on the conveyor belt of the security inspection device and the position of the detection area of the security inspection device, determine the target X-ray image collected by the security inspection device including the package to be detected;

A corresponding relationship between the package to be detected, the target person, and the target X-ray image is established.

Optionally, the conveyor belt of the security inspection equipment includes a pressure sensor;

The step of acquiring the monitoring image at the entrance of the security inspection equipment includes:

When it is detected by the pressure sensor that there is a package placed on the conveyor belt, the monitoring image at the entrance of the security inspection device collected at the current moment is acquired.

Optionally, the step of determining the package to be detected in the surveillance image and the target person to whom the package to be detected belongs includes:

Determining the package on the conveyor belt of the security inspection device in the monitoring image as the package to be detected;

The person closest to the package to be detected in the monitoring image is determined as the target person to whom the package to be detected belongs.

Optionally, according to the placement position of the package to be detected on the conveyor belt of the security inspection device and the position of the detection area of the security inspection device, the target collected by the security inspection device including the package to be detected is determined. Steps for X-ray images, including:

determining the distance between the package to be detected and the detection area according to the placement position of the package to be detected on the conveyor belt of the security inspection device and the position of the detection area of the security inspection device;

Based on the determined distance and the transmission speed of the conveyor belt, determine the time range for the package to be detected to be transmitted to the detection area;

From the X-ray images collected by the security inspection equipment within the time range, determine a target X-ray image collected by the security inspection equipment that includes the package to be inspected.

The method also includes:

Recognizing the monitoring image to obtain the placement position of the package to be detected on the conveyor belt of the security inspection device; and/or

The position of the pressure signal detected by the pressure sensor is determined as the placement position of the package to be detected on the conveyor belt.

Optionally, before determining the target X-ray image collected by the security inspection device including the package to be detected from the X-ray images collected by the security inspection device within the time range, the method further includes:

Acquiring the first package feature of the package to be detected in the monitoring image;

The step of determining, from the X-ray images collected by the security inspection equipment within the time range, the target X-ray images collected by the security inspection equipment that include the package to be detected includes:

Obtain the second package feature of the package in each X-ray image collected by the security inspection equipment within the time range;

The first package feature is matched with the second package feature of the package in each X-ray image to obtain a target X-ray image collected by the security inspection device including the package to be detected.

Optionally, the package features include multi-dimensional features;

The step of matching the characteristics of the first package with the characteristics of the second package in each X-ray image to obtain the target X-ray image collected by the security inspection equipment and including the package to be detected includes:

For each X-ray image, calculate the weighted sum of the matching confidence degrees corresponding to the second package feature of the X-ray image and each feature dimension in the first package feature, and obtain the package in the X-ray image and the The matching confidence between the packages to be detected;

Based on the matching confidence between the package in each X-ray image and the package to be detected, and the target person to whom the package to be detected belongs, perform a logical judgment on each X-ray image to determine the security inspection equipment The collected target X-ray image includes the package to be detected.

Optionally, the package features include one or more of package weight, package size and package shape.

Optionally, the step of establishing the corresponding relationship between the package to be detected, the target person and the target X-ray image includes:

Acquiring the image of the package before the package to be detected enters the detection area and the person image of the target person;

Based on the package image, the person image, and the target X-ray image, a corresponding relationship between the package to be detected, the target person, and the target X-ray image is established.

Optionally, the method also includes:

When it is detected that the package in the X-ray image of the target contains abnormal items, an alarm message is output based on the correspondence between the package to be detected, the person of the target, and the X-ray image of the target.

The embodiment of the present application also provides an information association device, which is applied to security inspection equipment, and the device includes:

The first acquisition module is used to acquire the monitoring image at the entrance of the security inspection equipment;

A first determining module, configured to determine the package to be detected in the surveillance image and the target person to whom the package to be detected belongs;

The second determining module is configured to determine the information collected by the security inspection equipment including the package to be detected according to the placement position of the package to be detected on the conveyor belt of the security inspection device and the position of the detection area of the security inspection device. Target X-ray image;

An establishment module, configured to establish correspondence between the package to be detected, the target person, and the target X-ray image.

The embodiment of the present application also provides a security inspection device, which includes a camera, a conveyor belt, an image analysis module, an X-ray detection module, and an information association module; the conveyor belt includes a pressure sensor:

The camera is used to acquire the monitoring image at the entrance of the security inspection equipment collected at the current moment when the pressure sensor detects that there is a package placed on the conveyor belt;

The conveyor belt is used for conveying parcels;

The image analysis module is used to acquire the monitoring image; determine the package to be detected in the monitoring image;

The X-ray detection module is configured to collect an X-ray image of the package when the package on the conveyor belt is transported to the detection area, and perform abnormal detection on the package based on the X-ray image;

The information association module is configured to determine the target person to whom the package to be detected belongs in the monitoring image; according to the placement position of the package to be detected on the conveyor belt of the security inspection device and the detection area of the security inspection device Determine the location of the target X-ray image collected by the security inspection equipment including the package to be detected; establish a corresponding relationship between the package to be detected, the target person, and the target X-ray image.

Optionally, the security inspection device also includes a pressure analysis module, a face capture module, an information storage module and an information display module;

The pressure analysis module is configured to obtain the pressure signal generated by the pressure sensor when the package to be detected is placed on the conveyor belt, and determine the pressure sensor pressure signal according to the preset correspondence between the pressure signal and the pressure value. The pressure value corresponding to the generated pressure signal is used as the package weight of the package to be detected;

The face capture module is used to capture the face of the person at the entrance of the security inspection device when the pressure sensor detects that a package is placed on the conveyor belt;

The information storage module is used to store the corresponding relationship between the package to be detected, the target person and the target X-ray image;

The information display module is configured to output an alarm based on the correspondence between the package to be detected, the target person, and the target X-ray image when detecting that the package in the target X-ray image contains abnormal items information.

The embodiment of the present application also provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the steps of any one of the information association methods described above are implemented.

The embodiment of the present application also provides a computer program product containing instructions, which, when run on a computer, causes the computer to execute any one of the information association methods described above.

Beneficial effects of the embodiment of the application:

The information association method, device, security inspection equipment, and storage medium provided in the embodiments of the present application can determine the target person to whom the package to be inspected in the surveillance image belongs to by obtaining the monitoring image at the entrance of the security inspection equipment, that is, determine the person who is associated with the package to be inspected. target personnel. In addition, according to the placement position of the package to be detected on the conveyor belt of the security inspection device and the position of the detection area of the security inspection device, the target X-ray image collected by the security inspection device including the package to be detected can be determined, that is, it can be determined that the package to be detected is associated with X-ray image of the target. This enables the security inspection equipment to establish the correspondence between the package to be detected, the target person, and the target X-ray image according to the target person to whom the package to be detected belongs, and the target X-ray image including the package to be detected, that is, the package, person and X-ray image The three are connected. Since the association process is directly carried out by the security inspection equipment based on the collected monitoring images and X-ray images, no manual participation is required, which saves the time required for manual association, and reduces manual association in scenes with a large number of people or complex environments Possibility of errors increases the accuracy of the efficiency of information association.

In addition, by adopting the technical solution provided by the embodiment of the present application, by making minor modifications on the basis of related security inspection equipment, the association of packages, personnel and X-ray images is realized, which makes the modification cost of security inspection equipment lower . Moreover, during the entire information association process, the target personnel only need to place the package on the conveyor belt, and the security inspection equipment can automatically associate the package, personnel and X-ray images, that is, complete the information association process without the personnel being aware of it. This does not affect the personnel security screening process.

Of course, implementing any product or method of the present application does not necessarily need to achieve all the above-mentioned advantages at the same time.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application and the prior art, the following briefly introduces the accompanying drawings required in the embodiments and the prior art. Obviously, the accompanying drawings in the following description are only the present invention. For some embodiments of the application, those skilled in the art can also obtain other drawings based on these drawings without creative work.

FIG. 1 is a schematic flow chart of the first information association method provided by the embodiment of the present application;

Fig. 2 is a schematic structural diagram of the first type of security inspection equipment provided by the embodiment of the present application;

Fig. 3 is a second structural schematic diagram of the security inspection equipment provided by the embodiment of the present application;

FIG. 4 is a schematic diagram of the third structure of the security inspection equipment provided by the embodiment of the present application;

Fig. 5 is a second schematic flow chart of the information association method provided by the embodiment of the present application;

FIG. 6 is a schematic diagram of the transmission of the package to be detected provided by the embodiment of the present application;

FIG. 7 is a schematic flowchart of a third information association method provided by the embodiment of the present application;

FIG. 8 is a schematic diagram of a pressure analysis module provided in an embodiment of the present application;

FIG. 9 is a schematic flow chart of an abnormality warning method provided by an embodiment of the present application;

FIG. 10 is a signaling diagram of the information association process provided by the embodiment of the present application;

FIG. 11 is a schematic structural diagram of an information association module provided by an embodiment of the present application;

FIG. 12 is a schematic structural diagram of an information association device provided by an embodiment of the present application;

FIG. 13 is a schematic diagram of a fourth structure of the security inspection device provided by the embodiment of the present application.

Detailed ways

In order to make the purpose, technical solution, and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and examples. Apparently, the described embodiments are only some of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of this application.

In order to solve the efficiency and accuracy problems caused by manual association in related technologies, an information association method is provided in the embodiment of the present application. The method is applied to security inspection equipment. As shown in FIG. 1 , as shown in FIG. 1 , FIG. 1 is a schematic flow chart of the first information association method provided by the embodiment of the present application. The method is applied to the above-mentioned security inspection equipment, and specifically includes the following steps.

Step S101, acquiring the monitoring image at the entrance of the security inspection equipment.

Step S102, determining the package to be detected in the surveillance image and the target person to whom the package to be detected belongs.

Step S103, according to the placement position of the package to be detected on the conveyor belt of the security inspection device and the position of the detection area of the security inspection device, determine the target X-ray image collected by the security inspection device including the package to be detected.

Step S104, establishing a correspondence between the package to be detected, the target person, and the X-ray image of the target.

The above-mentioned security inspection equipment may be a security inspection machine, wherein the security inspection machine may also be called a security inspection instrument, a security X-ray machine, a luggage security inspection machine, a channel-type X-ray machine, and the like. The security inspection machine is an electronic device that sends the package into the X-ray inspection channel (that is, the detection area) by means of a conveyor belt, thereby completing the detection of the items in the package. In the embodiment of the present application, no specific limitation is imposed on the above-mentioned security inspection equipment.

For ease of understanding, the above security inspection equipment will be described by taking FIG. 2 as an example. Fig. 2 is a schematic structural diagram of the first type of security inspection equipment provided by the embodiment of the present application. The aforementioned security inspection equipment may at least include a transmission belt, a pressure sensor, a binocular camera, an image analysis module, an X-ray detection module and an information correlation module. The above-mentioned conveyor belt is not shown in the security inspection device shown in FIG. 2 .

The above-mentioned pressure sensors are distributed on the above-mentioned conveyor belt. The binocular camera, image analysis module, X-ray detection module and information correlation module shown in Figure 2 can be distributed on the same hardware entity, such as the security inspection machine mentioned above.

In the embodiment shown in FIG. 2 , the cameras in the above-mentioned security inspection equipment are binocular cameras. Since the above-mentioned security inspection equipment is obtained after minor modification on the basis of related security inspection equipment, for example, the above-mentioned information association module is added, etc., therefore, according to the specific application scenarios of the security inspection equipment, the related security inspection equipment The cameras are also different, and the cameras in the security inspection equipment obtained after modification based on the relevant security inspection equipment are also different. For example, the camera in the above security inspection equipment may also be an infrared camera or the like. Here, the cameras included in the above security inspection equipment are not specifically limited. For ease of description, the following only uses a binocular camera as an example for illustration, and does not play any limiting role.

Taking FIG. 3 as an example for illustration, FIG. 3 is a second structural schematic diagram of the security inspection device provided in the embodiment of the present application. In Fig. 3, the security inspection equipment mainly includes a transmission area and a detection area, and the above-mentioned pressure sensors and conveyor belts are distributed in the transmission area. The above-mentioned binocular camera, image analysis module, X-ray detection module and information association module are distributed in the detection area. In Fig. 3, the arrow direction indicates the moving direction of the conveyor belt.

In the embodiment of the present application, a plurality of pressure sensors are distributed on the conveyor belt of the above-mentioned security inspection equipment. Here, the number and distribution method of the pressure sensors on the above-mentioned conveyor belt are not specifically limited.

In addition to the various components shown in FIG. 2 above, the above-mentioned security inspection equipment may also include other components, such as a stress analysis module, a face capture module, an information storage module, and an information display module. Specifically, as shown in FIG. 4 , FIG. 4 is a schematic diagram of a third structure of the security inspection device provided by the embodiment of the present application. For the functions of each component in the above-mentioned security inspection equipment, refer to the description below, and details are not repeated here.

Through the method provided in the embodiment of the present application, by acquiring the monitoring image at the entrance of the security inspection device, the target person to whom the package to be detected in the monitoring image belongs can be determined, that is, the target person associated with the package to be detected can be determined. In addition, according to the placement position of the package to be detected on the conveyor belt of the security inspection device and the position of the detection area of the security inspection device, the target X-ray image collected by the security inspection device including the package to be detected can be determined, that is, it can be determined that the package to be detected is associated with X-ray image of the target. This enables the security inspection equipment to establish the correspondence between the package to be detected, the target person, and the target X-ray image according to the target person to whom the package to be detected belongs, and the target X-ray image including the package to be detected, that is, the package, person and X-ray image The three are connected. Since the association process is directly carried out by the security inspection equipment based on the collected monitoring images and X-ray images, no manual participation is required, which saves the time required for manual association, and reduces manual association in scenes with a large number of people or complex environments Possibility of errors increases the accuracy of the efficiency of information association.

The embodiments of the present application will be described below through specific embodiments.

With regard to the above step S101, the monitoring image at the entrance of the security inspection device is acquired.

In this step, the above-mentioned binocular camera can collect images at the entrance of the security inspection equipment in real time to obtain surveillance video. The surveillance video includes multiple frames of surveillance images. The above image analysis module can obtain monitoring images from the monitoring video collected by the binocular camera.

The entrance of the above-mentioned security inspection equipment can be the area within the preset range where the above-mentioned conveyor belt is located, or it can be the area within the preset range before the detection area of the security inspection equipment.

For ease of understanding, the above security inspection equipment shown in FIG. 3 is taken as an example. The entrance of the above-mentioned security inspection equipment can be: the area within 2 meters on the upper side, right side and left side of the transmission area shown in FIG. 3 . The entrance of the above-mentioned security inspection equipment can also be expressed as: the area within 10 meters in front of the detection area (right side of the detection area) shown in FIG. 3 . Here, the range of the area corresponding to the entrance of the above-mentioned security inspection equipment is not specifically limited.

In an optional embodiment, the acquisition of the monitoring image at the entrance of the security inspection device may be expressed as: the image analysis module acquires each frame of monitoring image of the monitoring video collected by the binocular camera.

In another optional embodiment, in order to improve the analysis efficiency of the image analysis module on the acquired monitoring images and save the computing resources of the image analysis module, the above-mentioned acquisition of the monitoring image at the entrance of the security inspection device can be expressed as: after being detected by the pressure sensor When a package is placed on the conveyor belt, the monitoring image at the entrance of the security inspection equipment collected by the binocular camera at the current moment is obtained.

In the above embodiment, when the pressure sensor detects that the package is placed on the conveyor belt, the monitoring image collected by the binocular camera at the current moment must include the package and at least one person. And there must be a target person to whom the package belongs among the persons included in the surveillance image. Therefore, by obtaining the monitoring image at this moment, the image analysis module can reduce the number of monitoring images that the image analysis module needs to analyze under the premise that there must be a package to be detected in the acquired monitoring image, and improve the image analysis module. The analysis efficiency of the monitoring image saves computing resources of the image analysis module. In addition, since the monitoring image at this moment must include the target person to whom the package to be detected belongs, it is also convenient for the above-mentioned information association module to determine the target person to whom the package to be detected in the monitoring image belongs to later, that is, to determine the correlation information between the package and the person. The determination improves the efficiency and accuracy of determining the associated information between packages and personnel.

The entrance of the above-mentioned security inspection equipment may include the area where the conveyor belt of the above-mentioned security inspection equipment is located, and the area within a preset range around the conveyor belt. Wherein, the preset range around the conveyor belt can be adjusted according to the shooting range of the binocular camera and user needs, and there is no specific limitation on the entrance of the above-mentioned security inspection equipment.

One of the aforementioned binocular cameras is a visible light camera. Another camera can be selected according to the application scenarios of the above-mentioned security inspection equipment and different user needs. For example, another camera in the aforementioned binocular camera may be a thermal imaging camera. For another example, the other camera in the aforementioned binocular camera may also be a visible light camera or a depth camera. Here, there is no specific limitation on the category of the camera other than the visible light camera among the above-mentioned binocular cameras.

In the embodiment of the present application, due to the uncertainty of the category of the binocular camera other than the visible light camera, the monitoring image acquired by the image analysis module may include the visible light image acquired by the visible light camera. In addition, the disparity map collected by the binocular camera can also be included.

For the above step S102, the package to be detected in the surveillance image and the target person to whom the package to be detected belongs are determined.

In this step, the image analysis module in the above-mentioned security inspection equipment can analyze the above-mentioned obtained monitoring image, and determine the package to be detected in the monitoring image. The information association module in the above-mentioned security inspection equipment can determine the target person to whom the package to be inspected belongs based on the surveillance image.

In an optional embodiment, the determination of the package to be detected in the monitoring image in the above step S102 can specifically be expressed as:

The above image analysis module determines the package on the conveyor belt of the security inspection device in the monitoring image as the package to be detected.

For example, the image analysis module can use the deep learning algorithm to identify the package on the conveyor belt in the monitoring image, so as to determine the identified package as the package to be detected. For example, the image analysis module can identify the package to be detected in the monitoring image according to the difference between the pixel value of the conveyor belt in the monitoring image and the pixel value of the package in the monitoring image.

In another optional embodiment, when the environment where the above-mentioned security inspection equipment is located has a large number of people or the environment is relatively complicated, there may be multiple packages on the conveyor belt of the above-mentioned security inspection equipment at the same time, that is, there are multiple packages to be detected in the monitoring image , at this time, in order to facilitate the distinction of each package to be detected, when the image analysis module determines the package to be detected in the monitoring image, it can also integrate the position of the pressure signal detected by the pressure sensor to determine the package to be detected. For example, the image analysis module can make a one-to-one correspondence between the identified position of the package to be detected and the position of the pressure signal detected by the pressure sensor, and determine the package to be detected on the conveyor belt.

In the embodiment of the present application, the package to be detected may be any package in the surveillance image, and here, the number of packages to be detected included in the surveillance image is not specifically limited. For the convenience of description, in the embodiment of the present application, only one package to be detected is used for illustration, which does not serve as any limitation.

In an optional embodiment, the determination in the above step S102 of the target person to whom the package to be detected in the surveillance image belongs can specifically be expressed as:

The above information association module can determine the person closest to the package to be detected in the monitoring image as the target person to whom the package to be detected belongs.

In this step, the above-mentioned image analysis module may send the acquired monitoring image to the above-mentioned information association module. The information association module can determine the distance between each person included in the monitoring image and the determined package to be detected according to the nearest neighbor rule classification (K Nearest Neighbor, KNN) algorithm, thereby determining the nearest person from the package to be detected as the Detect who the package belongs to.

In an optional embodiment, the distance between each person in the surveillance image and the determined package to be detected can be calculated according to the depth information corresponding to each person in the surveillance image and the depth information between the package to be detected. Here, the calculation of the distance is not specifically described.

In an optional embodiment, when the camera in the above security inspection device is the above binocular camera, the above depth information may be calculated according to the parallax between the binocular cameras. Here, the calculation of the depth information will not be specifically described.

In another optional embodiment, when the camera in the above-mentioned security inspection device is another type of camera, such as the above-mentioned infrared camera, the above-mentioned depth information can be obtained by combining the camera with other sensors. For example, by combining an infrared camera with a depth sensor, the depth information corresponding to each person in the surveillance image and the depth information of the package to be detected can be determined.

In the embodiment of the present application, there is no specific limitation on the method of determining the depth information corresponding to each person in the surveillance image and the depth information of the package to be detected.

In an optional embodiment, when the information association module determines the target person to whom the package to be detected belongs, it may establish a correspondence between the package to be detected and the target person to which it belongs. The corresponding relationship includes at least the package image of the package to be detected and the person image of the target person. Regarding the acquisition of the package image and the person image, refer to the description below, and details are not repeated here.

In the above-mentioned embodiment, when the above-mentioned monitoring image is obtained according to the detection result of the pressure sensor, that is, when the pressure sensor detects that there is a package placed on the conveyor belt, since the target person has just placed the package on the conveyor belt at this time, Therefore, it is more likely that the distance between the target person and the package is smaller than the distance between other persons and the package in the surveillance image. This greatly improves the accuracy of the target person determined based on the distance between the package to be detected and the person, thereby improving the accuracy of the associated information between the package and the person.

In an optional embodiment, in order to further improve the accuracy of determining the target person, after determining the person closest to the package to be detected in the monitoring image, the behavior of the person can be tracked based on the monitoring image, so as to determine Whether the person is the target person to whom the package to be detected belongs.

In an optional embodiment, according to the information association method shown in FIG. 1 above, this embodiment of the present application further provides an information association method. As shown in FIG. 5 , FIG. 5 is a schematic flowchart of a second information association method provided by the embodiment of the present application. In the method shown in Figure 5, the above step S103, that is, according to the placement position of the package to be detected on the conveyor belt of the security inspection device, and the position of the detection area of the security inspection device, determine the target X collected by the security inspection device including the package to be detected The light image is refined into the following steps, that is, step S1031-step S1033.

Step S1031, according to the placement position of the package to be detected on the conveyor belt of the security inspection device and the location of the detection area of the security inspection device, determine the distance between the package to be detected and the detection area.

In this step, the above image analysis module can calculate the distance between the location of the package to be detected and the location of the detection area according to the location of the package to be detected on the conveyor belt of the security inspection device and the location of the detection area of the security inspection device , to get the distance between the package to be detected and the detection area. The image analysis module can send the determined distance between the package to be detected and the detection area to the information association module.

In an optional embodiment, the distance between the package to be detected and the detection area may be a specific distance value. For example, the distance between the package to be detected and the detection area may be expressed as: the distance from the center of the location where the package to be detected is placed to the center of the detection area.

In another optional embodiment, the distance between the package to be detected and the detection area may also be a specific distance range. For example, the minimum value of the distance range can be the distance between the side of the package to be detected that first enters the detection area and the area where the detection area is located, and the maximum value of the distance range can be the distance between the side of the package to be detected that leaves the detection area and the detection area. The distance between the regions.

For ease of understanding, it will be described with reference to FIG. 6 as an example. FIG. 6 is a schematic diagram of the transmission of the package to be detected provided by the embodiment of the present application. Among them, position B is the midpoint position of the detection area, side A and side C are the left edge position and right edge position of the detection area respectively, position E is the center position of the package to be detected, and side D and F Sides are the left edge position and the right edge position of the package to be detected respectively. According to the moving direction of the conveyor belt, that is, from right to left, the D side of the package to be detected enters the detection area first, and the F side leaves the detection area last.

The distance between the package to be detected and the detection area determined by the above image analysis module can be expressed as the distance between the center position of the package to be detected (ie position E) and the center position of the detection area (ie position B), namely Figure 6 shows the distance d2. It can also be expressed as the distance between the D side of the package to be detected and the C side of the area to be detected, and the distance corresponding to the distance between the F side of the package to be detected and the A side of the area to be detected, that is, as shown in Figure 6 [d1,d3].

In the embodiment of the present application, there is no specific limitation on the way of expressing the distance between the package to be detected and the detection area.

The location of the detection area of the above-mentioned security inspection equipment is determined according to the location of the above-mentioned X-ray detection module. For example, the X-ray detection module can be arranged at the end position of the conveying direction of the conveyor belt. For another example, the X-ray detection module can be set at a preset distance before the end position of the conveying direction of the conveyor belt, and the preset distance can be set according to user requirements, the size of the package that can be detected by the security inspection equipment, and the like. Here, there is no specific limitation on the location of the detection area.

In an optional embodiment, the placement position of the above-mentioned package to be detected on the conveyor belt can be obtained in the following manner:

Mode 1, the above-mentioned image analysis module can identify the monitoring image, and obtain the placement position of the package to be detected on the conveyor belt of the security inspection device.

For example, according to the depth information of the package to be detected in the surveillance image, the position of the package to be detected on the conveyor belt of the security inspection device is calculated and obtained.

In the second way, the above-mentioned pressure analysis module can determine the position of the pressure signal detected by the pressure sensor as the placement position of the package to be detected on the conveyor belt. The image analysis module obtains the placement position of the package to be detected on the conveyor belt from the pressure analysis module.

Through the above method 1 and method 2, the placement position of the package to be detected on the conveyor belt can be accurately determined, thereby improving the accuracy of the distance between the package to be detected and the detection area determined according to the placement position.

Step S1032, based on the determined distance and the conveying speed of the conveyor belt, determine the time range for the package to be inspected to be conveyed to the inspection area.

In this step, after the package to be detected is placed on the conveyor belt, the conveyor belt will transport the package to be detected to the above detection area at a certain transmission speed for detection. Therefore, the above-mentioned information association module can calculate the time required for the package to be detected to be transmitted to the detection area according to the determined distance between the package to be detected and the detection area, and the transmission speed of the conveyor belt, so as to determine that the package to be detected is transmitted to the detection area. The time range of the region.

The above time range may be expressed as a certain time point, or may be expressed as a certain time interval.

For ease of understanding, the above-mentioned distance between the package to be detected and the detection area is taken as an example for illustration. When the above-mentioned time range is a certain time point, the above-mentioned determined time range for the package to be detected to be delivered to the detection area may be: the quotient of the distance corresponding to the midpoint of the above-mentioned distance range and the conveying speed of the conveyor belt. When the above-mentioned time range is a certain time interval, the minimum value of the above-mentioned determined time range for the package to be detected to be transmitted to the detection area can be: the quotient of the minimum value of the distance range and the transmission speed of the conveyor belt, and the maximum value of the time range can be It is: the quotient of the maximum value of the distance range and the transmission speed of the conveyor belt.

The transmission speed of the above-mentioned conveyor belt can be read directly from the security inspection equipment, or can be calculated according to the change of the distance between the package to be detected and the detection area over time in the surveillance video collected by the binocular camera. Here, the acquisition of the transmission speed of the above-mentioned conveyor belt is not specifically limited.

In the embodiment of the present application, since the conveying speed of the conveyor belt may be affected by the quantity and weight of the parcels conveyed on the conveying belt, the above-mentioned distance between the parcels to be detected and the detection area determined according to the above, and the conveying speed of the conveying belt , after calculating the time required for the package to be detected to be delivered to the detection area, a time range can be selected based on the calculated time, so as to ensure that the X-ray image associated with the package to be detected will be included in the X-ray image acquired later based on this time range image.

For example, the current time is 8:30, after the package to be detected is placed on the conveyor belt, according to the distance between the package to be detected and the detection area, and the transmission speed of the conveyor belt, it is determined that the package to be detected is expected to be delivered to the detection area at 8:31. However, considering that there is a large flow of people at the current moment, that is, there are many packages on the conveyor belt, the information association module can extend the time backward by a certain period of time, such as 30 milliseconds, according to the determined time for the package to be detected to be transmitted to the detection area. At this time, the time when the package to be detected arrives at the detection area may be marked as a time range, that is, a time range of 8:31 to 30 milliseconds after 8:31.

The length of the above-mentioned backward extension can be set according to the flow of people in the specific application scenario, the transmission speed of the conveyor belt, and user needs, and is not specifically limited here.

Step S1033, from the X-ray images collected by the security inspection equipment within the time range, determine the target X-ray images collected by the security inspection equipment including the package to be inspected.

For the determination of the target X-ray image including the package to be detected, please refer to the description below, and no specific description is given here.

In an optional embodiment, if the above-mentioned time range is a certain time interval, the information correlation module acquires any image within the time interval as the target X-ray image including the package to be detected collected by the security inspection equipment. For example, the information association module may obtain the X-ray image collected by the security inspection equipment at the time point corresponding to the midpoint of the above time range as the target X-ray image including the package to be inspected.

In another optional embodiment, if the above-mentioned time range is a certain point in time, when the above-mentioned conveying belt has fewer packages, or the conveyed packages are lighter, the information association module can directly obtain the information calculated in the above-mentioned step S1032 The X-ray image collected by the security inspection equipment at the time point required for the package to be detected to be transported to the detection area is used as the target X-ray image including the package to be detected.

In the embodiment of the present application, when it is determined that the target X-ray image collected by the security inspection equipment includes the package to be detected, the information association module may establish a correspondence between the package to be detected and the target X-ray image. The corresponding relationship includes at least the package image of the package to be detected and the above-mentioned target X-ray image. Regarding the acquisition of the package image, please refer to the description below, which will not be repeated here.

In the above-mentioned embodiment, the number of acquired target X-ray images is 1 as an example for illustration. In addition, the number of target X-ray images acquired by the information association module may also be multiple. Here, the number of the target X-ray images is not specifically limited.

Through the method shown in the above step S1031-step S1033, the information association module can accurately determine the time range when the package to be detected enters the detection area, so as to obtain the X-ray images collected by the security inspection equipment to be detected during this period, and then The determination of the target X-ray image including the package to be detected effectively improves the accuracy of the acquired target X-ray image including the package to be detected.

In another optional embodiment, when obtaining the image collected by the security inspection equipment and including the target X-ray image of the package to be detected, in addition to using steps S1031-step S1033 shown in Figure 5 above, the image analysis module can monitor in real time The distance between the above-mentioned package to be detected and the detection area, and when the distance is 0, trigger the above-mentioned X-ray detection module to collect the X-ray image of the package to be detected, that is, the above-mentioned target X-ray image including the package to be detected, so as to obtain the same Target X-ray image associated with the package to be detected.

For the above step S104, the corresponding relationship among the package to be detected, the target person and the X-ray image of the target is established.

In this step, since the target person of the package to be detected as determined in the above step S102 is unique, after the target X-ray image including the package to be detected is determined through the above step S103, the information association module can directly use the The target person to whom the package belongs, and the target X-ray image including the package to be detected, establishes a correspondence between the package to be detected, the target person, and the target X-ray image. The corresponding relationship includes at least the package image of the package to be detected, the person image of the target person, and the target X-ray image.

In an optional embodiment, according to the method shown in FIG. 5 , this embodiment of the present application also provides an information association method. As shown in FIG. 7 , FIG. 7 is a schematic flowchart of a third information association method provided by the embodiment of the present application. The method includes the following steps.

Step S701, acquiring the monitoring image at the entrance of the security inspection equipment.

Step S702, determining the package to be detected in the surveillance image and the target person to whom the package to be detected belongs.

Step S703, according to the placement position of the package to be detected on the conveyor belt of the security inspection device and the location of the detection area of the security inspection device, determine the distance between the package to be detected and the detection area.

Step S704, based on the determined distance and the conveying speed of the conveyor belt, determine the time range for the package to be inspected to be conveyed to the inspection area.

The above step S701 - step S704 are the same as the above step S101 - step S1032 shown in FIG. 5 .

Step S705, acquiring the first package feature of the package to be detected in the surveillance image.

In this step, the above-mentioned information association module can obtain the package features of the package to be detected in the above-mentioned monitoring image, and obtain the first package feature of the package to be detected.

In an optional embodiment, the above-mentioned first package feature may be a multi-dimensional feature. For example, the aforementioned first package feature may include package weight, package size, package color, package shape, etc. of the package to be detected. Here, the above-mentioned first package feature is not specifically limited.

In an optional embodiment, when the above-mentioned first package feature includes the above-mentioned package weight, the above-mentioned pressure analysis module can determine the package weight of the package to be detected according to the pressure signal that the package to be detected is placed on the conveyor belt, so as to determine the The weight of the package is sent to the above information association module.

For ease of understanding, the above determination of the package weight will be described with reference to FIG. 8 . FIG. 8 is a schematic diagram of a pressure analysis module provided in an embodiment of the present application. The pressure analysis module shown in FIG. 8 may include a pressure signal input unit 801 , a signal filter unit 802 , a signal conversion unit 803 and a trigger unit 804 .

Specifically, when the target person places the package to be detected on the conveyor belt, the pressure sensor on the conveyor belt will generate a corresponding electrical signal, which is recorded as a pressure signal. At this time, the pressure signal input unit 801 will acquire the pressure signal.

In the actual process, after the target personnel put the package to be detected on the conveyor belt, the package to be detected will be sent to the detection area for detection. During this period, the package to be detected has not left the conveyor belt, that is, the pressure sensor in the placement area corresponding to the package to be detected is always in a triggered state. Therefore, the above-mentioned electrical signal is a tracking electrical signal obtained by the triggered pressure sensor.

The pressure signal input unit 801 may transmit the pressure signal to the signal filtering unit 802 after obtaining the pressure signal. The signal filtering unit 802 performs filtering processing on the received pressure signal to obtain a filtered pressure signal, and transmits the filtered pressure signal to the signal converting unit 803 . Wherein, the filtering methods adopted by the signal filtering unit 802 include but not limited to mean filtering and median filtering.

In the embodiment of the present application, by filtering the above pressure signal, the noise signal in the pressure signal can be effectively suppressed, thereby improving the accuracy of the package weight determined based on the filtered pressure signal.

After receiving the filtered pressure signal sent by the signal filtering unit 802, the signal-to-pressure unit 803 can determine the pressure value corresponding to the pressure signal generated by the pressure sensor according to the preset correspondence between the pressure signal and the pressure value as the pressure value to be detected. The package weight of the package.

In the embodiment of the present application, the package weight of the above-mentioned package to be detected is expressed as the value of pressure generated by the package to be detected on the conveyor belt. In addition, the package weight of the package to be detected can also be expressed as the real weight of the package to be detected. For example, the above-mentioned pressure analysis module can obtain the real weight through mathematical calculation according to the pressure value generated by the package to be detected on the conveyor belt and the contact area between the package to be detected and the conveyor belt. Here, the calculation process of the real weight of the package to be detected will not be specifically described.

The above trigger unit 804 is also included in the pressure analysis module shown in FIG. 8 . The triggering unit 804 may be connected with the aforementioned face capture module and image analysis module. When the above-mentioned pressure signal input unit 801 obtains the above-mentioned pressure signal, the trigger unit 804 can send a trigger signal to the face capture module and the image analysis module, thereby triggering the face capture module to take a face capture, and triggering the information association module to analyze the package To associate with a person, that is, to determine the target person to whom the person to be detected belongs in the above step S102.

In an optional embodiment, in order to improve the accuracy of the correlation between the package to be detected and the X-ray image, after the pressure signal input unit 801 acquires the pressure signal, the trigger unit 804 can send the above-mentioned The face capture module and the image analysis module send trigger signals.

For ease of understanding, the above-mentioned preset trigger logic rule may be the duration of the above-mentioned pressure signal. Specifically, after the personnel put the package on the conveyor belt, the user may immediately pick up the package. At this time, the above-mentioned pressure signal will be interrupted. If the trigger unit 804 sends the above-mentioned trigger signal, it will affect the subsequent detection of the package and X-ray detection. Accuracy of image association. In addition, in the embodiment of the present application, no specific limitation is made on the above-mentioned preset trigger logic rule.

In an optional embodiment, the pressure signal input unit 801 may determine the placement position of the package to be detected on the conveyor belt according to the position of the triggered pressure sensor while acquiring the pressure signal.

In an optional embodiment, when the above-mentioned first package feature wraps the above-mentioned package color, the above-mentioned image analysis module can acquire the package color of the package to be detected after determining the package to be detected in the above-mentioned monitoring image, so that the obtained The color of the package is sent to the above information association module. For example, the above-mentioned image analysis module can use a deep learning algorithm to classify the above-mentioned monitoring image to obtain the package color of the package to be detected. Here, the acquisition process of the package color is not specifically limited.

In an optional embodiment, when the above-mentioned first package feature includes the above-mentioned package size, after the above-mentioned image analysis module determines the package to be detected in the above-mentioned monitoring image, it can also determine the package size of the package to be detected, so that the determined The size of the package is sent to the above information association module.

When the image analysis module determines the package size of the package to be detected, it can use the deep learning algorithm to perform image segmentation on the above monitoring image, and perform pixel-level classification on the package to be detected and the conveyor belt in the monitoring image, that is, to classify the package in the monitoring image. The pixel points of the area where the package to be detected is located are distinguished from the pixel points of the area where the conveyor belt is located, so as to determine the pixel area or pixel volume corresponding to the package to be detected.

In the embodiment of the present application, when the package size of the package to be detected is expressed as the pixel volume of the package to be detected, the package size may be expressed as the product of the pixel area and the package height. Among them, the package height can be calculated by the following formula.

ΔD=D ₁ -D ₂

Among them, ΔD is the package height, D ₁ is the distance from the upper surface of the package to be detected to the binocular camera, D ₂ is the distance from the lower surface of the package to be detected to the binocular camera, that is, the contact surface of the package to be detected and the conveyor belt to The distance between the binocular cameras.

_The above D1 and D2 can be calculated according to the lens focal length of the binocular camera, the baseline distance between the _two cameras of the binocular camera and the parallax of the binocular camera, which can be specifically expressed as:

D=f*B/XL

Among them, D is the target distance, that is, the above D ₁ or D ₂ , f is the lens focal length of the binocular camera, B is the baseline distance between the two cameras of the binocular camera, and XL is the parallax of the binocular camera.

In an optional embodiment, affected by the shooting angle of the binocular camera when collecting the package to be detected, there may be a certain difference in the pixel area of the package to be detected in the monitoring images collected by the two cameras in the binocular camera. In order to improve the accuracy of the above-mentioned pixel area of the package to be detected, the image analysis module can determine the mean value of the pixel area of the package to be detected in the monitoring images collected by the two cameras in the binocular camera as the pixel area of the package to be detected .

In an optional embodiment, when the above-mentioned first package feature includes the above-mentioned package shape, after the above-mentioned image analysis module determines the package to be detected in the above-mentioned monitoring image, it can also determine the package shape of the package to be detected, so that the determined The shape of the package is sent to the above-mentioned information association module.

When the above image analysis module determines the package shape of the package to be detected, it can use a deep learning algorithm to extract the contour information of the package to be detected in the above monitoring image to obtain the package shape of the package to be detected. Among them, the deep learning algorithm includes but not limited to Holistically-nested Edge Detection (HED) algorithm and Semantic Edge Detection (Deep Category-Aware Semantic Edge Detection, CASENet) algorithm.

The above step S705 may be executed simultaneously with the above step S703.

Step S706, acquiring the characteristics of the second package in each X-ray image collected by the security inspection equipment within the time range.

In this step, the association information module acquires all the X-ray images collected by the above-mentioned X-ray detection module within the time range determined in step S704, and for each X-ray image, obtains the package characteristics of the package in the X-ray image , as the second package feature of the X-ray image.

In an optional embodiment, the above-mentioned second package feature may be a multi-dimensional feature. For example, the above-mentioned second package feature may include package weight, package size, package shape, etc. of the package to be detected. Here, the above-mentioned second package feature is not specifically limited.

The acquisition method of the second package feature of the package to be detected can refer to the method of acquisition of the first package feature of the package to be detected, and the method of obtaining the second package feature of the package to be detected will not be specifically described here.

In the embodiment of the present application, the above-mentioned first package feature and the second package feature may include one or more of the above-mentioned package weight, package size and package shape.

Step S707, matching the feature of the first package with the feature of the second package in each X-ray image to obtain the target X-ray image collected by the security inspection equipment including the package to be detected.

In an optional embodiment, when the first package feature and the second package feature are multi-dimensional features, the above step S707 may specifically include the following steps:

Step 1, for each X-ray image, calculate the weighted sum of the matching confidence degrees corresponding to each feature dimension in the second package feature of the X-ray image and the first package feature, and obtain the package and the package in the X-ray image. Check the matching confidence between packages.

In an optional embodiment, for each X-ray image, the following formula can be used to calculate the matching confidence Conf between the package in the X-ray image and the package to be detected:

Among them, n is the number of feature dimensions of the package features included in the first package feature or the second package feature, i is the i-th feature dimension, a _i is the weight of the i-th feature dimension, A _i is the first package feature at The matching confidence between the wrapping feature on the i feature dimension and the wrapping feature on the i feature dimension of the second wrapping feature.

For ease of understanding, the matching confidence of the first package feature and the second package feature in the area dimension is taken as an example for illustration. Now assume that the package feature at the position of the area area in the first package feature is the area A corresponding to the contact surface between the package and the conveyor belt, and the package feature at the area area position in the second package feature is the area B corresponding to the contact surface between the package and the conveyor belt. The union of area A and area B is area C. At this time, the matching confidence of the first package feature and the second package feature in the area dimension can be expressed as: the ratio of area A to area C, or the ratio of area B to area C. When the ratio is larger, that is, when the matching confidence of the first package feature and the second package feature in the area dimension is larger, the possibility that area A and area B are the same area is greater, and the X-ray image The greater the matching confidence between the package and the package to be detected is.

In the embodiment of the present application, the matching confidence between the package in the X-ray image and the package to be detected may represent the degree of correlation between the package in the X-ray image and the package to be detected. Specifically, the greater the matching confidence between the package in the X-ray image and the package to be detected is, the higher the degree of correlation between the package in the X-ray image and the package to be detected is. The smaller the matching confidence between the package in the X-ray image and the package to be detected is, the lower the degree of correlation between the package in the X-ray image and the package to be detected is.

Step 2: Based on the matching confidence between the package in each X-ray image and the package to be detected, and the target person to whom the package to be detected belongs, make a logical judgment on each X-ray image, and determine that the security inspection equipment includes the package to be detected Target X-ray image of the package.

In this step, the information association module can use the preset association logic to judge the The rule is to make a logical judgment on each acquired X-ray image, and determine the target X-ray image collected by the security inspection equipment including the package to be inspected.

For ease of understanding, the above-mentioned preset association logic judgment rules are a preset matching confidence threshold and a preset pixel threshold as an example for illustration. For each acquired X-ray image, when the information association module makes a logical judgment on the X-ray image, if the matching confidence between the package in the X-ray image and the package to be detected is less than the preset matching confidence threshold , then the information association module can discard the X-ray image. Alternatively, for each acquired X-ray image, the above-mentioned information association module can calculate the difference between the pixel wrapped in the X-ray image and the pixel to be detected in the monitoring image, if the difference is greater than the preset pixel threshold , then the information association module can discard the X-ray image.

In the embodiment of the present application, in addition to the above-mentioned preset matching confidence threshold and preset pixel threshold, the above-mentioned preset association logic judgment rules may also include other logic judgment rules. The above-mentioned information correlation module performs logical judgment on the above-mentioned obtained X-ray images through a plurality of logical judgment rules, so as to obtain the target X-ray image including the package to be detected. Here, the aforementioned preset association logic judgment rules are not specifically limited.

Through the above logical judgment process for each acquired X-ray image, the accuracy of the association between the package to be detected and the X-ray image can be effectively improved, thereby improving the accuracy of the detection association of the package to be detected, the target person, and the target X-ray image .

The above step S706-step S707 is refinement of the above step S1033 shown in FIG. 5 .

Step S708, establishing the corresponding relationship between the package to be detected, the target person and the X-ray image of the target.

The above step S708 is the same as the above step S104.

In an optional embodiment, the above step S104, that is, establishing the corresponding relationship between the package to be detected, the target person and the target X-ray image, can be subdivided into the following steps, namely step 1 - step 3.

Step 1, acquire the package image before the package to be detected enters the detection area.

In this step, the information association module can intercept the image of the area where the package to be detected is located in the above monitoring image, and obtain the image of the package before the package to be detected enters the detection area.

Regarding the acquisition of the above-mentioned package image, since the above-mentioned binocular camera has been collecting images at the entrance of the security inspection equipment, the information association module acquires the binocular camera when the package to be detected is placed on the conveyor belt until the package to be detected enters the above-mentioned detection area The monitoring video in this period of time, so as to obtain the package image of the package to be detected from any monitoring image in the monitoring video.

Step 2, obtain the person image of the target person.

The above-mentioned person image may be a human body image of the target person, or may be a captured image of the face of the target person.

In an optional embodiment, when the above-mentioned person image is a target person's body image, the information association module may acquire the target person's body image from the above-mentioned surveillance image. For details, reference may be made to the method for obtaining the package image above, which will not be described in detail here.

In another optional embodiment, when the above-mentioned person image is a face capture image of the target person. When the pressure sensor detects that the package to be detected is placed on the conveyor belt, it can trigger the face capture module to capture the face of the person at the entrance of the security inspection device to obtain a face capture image of the person. The above-mentioned information association module can obtain the face capture image of the target person from the face capture image obtained by the face capture module according to the target person to whom the package to be detected belongs.

In the embodiment of the present application, after the above-mentioned face capture module captures the face capture image, it can also perform operations such as face positioning, face key point detection, and face modeling based on the face capture image, so as to obtain Get the personnel information of the personnel. The personal information includes but not limited to face attributes and face feature vectors. Among them, face attributes include but are not limited to age or whether to wear a mask. Here, no specific description will be given on the process of obtaining the person information in the face capture image.

In an optional embodiment, in order to improve the accuracy of the above-mentioned personnel information, when the above-mentioned face capture module obtains the personnel information of the above-mentioned personnel, it can obtain multiple face capture images within a certain period of time for each person, so that For each face capture image, the face score is performed on the face capture image, and the above-mentioned personnel information is obtained from the face capture image with the highest face score of each person.

Step 3, based on the package image, the person image and the target X-ray image, the corresponding relationship between the package to be detected, the target person and the target X-ray image is established.

In the embodiment of the present application, the correspondence between the package to be detected, the target person and the target X-ray image established by the above-mentioned information association module includes at least the above-mentioned package image, person image and target X-ray image. As shown in Table 1, Table 1 is an information association manner provided by the embodiment of the present application.

Table 1

包裹pack	人员personnel	X光图像X-ray image
包裹图像1Package image 1	人员图像1Person image 1	X光图像1X-ray image 1
包裹图像2Package Image 2	人员图像2Person Image 2	X光图像2X-ray image 2

In the embodiment of the present application, the corresponding relationship between the package to be detected, the target person and the target X-ray image may also include other information besides the above package image, person image and target X-ray image, for example, the person of the target person information, and the detection results of the target X-ray image, as shown in Table 2.

Table 2

The detection results in the above Table 2 are obtained by the above-mentioned X-ray detection module detecting the abnormality of the package in the X-ray image. Wherein, a normal detection result indicates that no abnormal items are contained in the package. An abnormal test result indicates that the package contains abnormal items, such as the above-mentioned prohibited items.

Based on the same inventive concept, and according to the information association method provided by the above-mentioned embodiment of the present application, the embodiment of the present application further provides an abnormality warning method. As shown in FIG. 9 , FIG. 9 is a schematic flowchart of an abnormality warning method provided by an embodiment of the present application. Obtain the surveillance image at the entrance of the security inspection equipment.

Step S901, acquiring the monitoring image at the entrance of the security inspection equipment.

Step S902, determining the package to be detected in the surveillance image and the target person to whom the package to be detected belongs.

Step S903, according to the placement position of the package to be detected on the conveyor belt of the security inspection device and the location of the detection area of the security inspection device, determine the target X-ray image collected by the security inspection device including the package to be detected.

Step S904, establishing a corresponding relationship between the package to be detected, the target person and the X-ray image of the target.

The above step S901-step S904 are the same as the above-mentioned step S101-step S104.

Step S905, when it is detected that the package in the X-ray image of the target contains abnormal items, an alarm message is output based on the correspondence between the package to be detected, the target person, and the X-ray image of the target.

In this step, the above-mentioned X-ray detection module can perform abnormality detection on the items contained in the package in the X-ray image for each collected X-ray image. When the X-ray detection module detects that the package in the above-mentioned target X-ray image contains abnormal items, such as the above-mentioned prohibited items, the X-ray detection module can establish the relationship between the package to be detected, the target person, and the target X-ray image according to the information association module. The corresponding relationship, the output shows the alarm message.

The above-mentioned warning message is generated according to the corresponding relationship among the above-mentioned package to be detected, the target person and the target X-ray image. For example, the above-mentioned warning message can wrap the above-mentioned X-ray image of the target, the personnel information of the target person, the package image of the package to be detected, etc., which will help the detection personnel to accurately determine the location of the abnormal item when seeing the warning message. The package, and the owner of the package, that is, the above-mentioned target person, thereby improving the safety of aircraft, trains, etc., and protecting the lives of other passengers.

For ease of understanding, the above information association process will be described below with reference to FIG. 10 and FIG. 11 . FIG. 10 is a signaling diagram of the information association process provided by the embodiment of the present application. FIG. 11 is a schematic structural diagram of an information association module provided by an embodiment of the present application.

Step S1001, the binocular camera collects the monitoring image at the entrance of the security inspection equipment.

Step S1002, when the pressure sensor detects that the target person puts the package to be detected on the conveyor belt, a pressure signal is generated, and the pressure signal is sent to the pressure analysis module.

In step S1003, the pressure analysis module sends trigger signals to the image analysis module and the face capture module respectively according to the received pressure signal.

In this step, the pressure analysis module can also determine the position of the package to be detected on the conveyor belt according to the position of the pressure signal detected by the pressure sensor, and determine the position of the package to be detected according to the corresponding relationship between the preset pressure signal and the pressure value. The package weight of the package, and the determined location and package weight of the package to be detected are sent to the information association module through the image analysis module.

Step S1004, after the image analysis module receives the trigger signal, it acquires the monitoring image collected by the binocular camera when the pressure sensor detects the package.

Step S1005, the image analysis module determines the package to be detected in the acquired monitoring image, and obtains the package characteristics of the package to be detected in the monitoring image.

The aforementioned package characteristics may include package size, package color, package shape, and the like.

Step S1006, the image analysis module sends the acquired monitoring image and the package features of the package to be detected to the information association module.

Step S1007, after receiving the trigger signal, the face capture module captures the face of the person at the entrance of the security inspection device to obtain a face capture image.

The above-mentioned face capture module can also analyze the face capture image of each person to obtain the personnel information of each person.

Step S1008, the information association module determines the person closest to the package to be detected in the monitoring image as the target person to whom the package to be detected belongs, and establishes a correspondence between the package to be detected and the target person according to the target person to whom the package to be detected belongs.

In this step, after receiving the monitoring image sent by the image analysis module, the information association module can use the personnel and package association unit shown in Figure 11 to determine the person closest to the package to be detected in the monitoring image as the package to be detected belong to the target person.

The person and package association unit can obtain the face capture image and personnel information of the target person to whom the package to be detected belongs from the above-mentioned face capture module, and according to the face capture image and personnel information of the target person, the package image of the package to be detected, and The first package feature of the package to be detected establishes the correspondence between the package to be detected and the target person.

Step S1009, the X-ray detection module collects X-ray images in the detection area, and performs abnormality detection on the collected X-ray images.

In step S1010, the information association module determines the target X-ray image collected by the security inspection device including the package to be detected according to the location of the package to be detected on the conveyor belt of the security inspection device and the location of the detection area of the security inspection device.

In this step, the package and X-ray image associating unit shown in FIG. 11 can obtain the information including the package to be detected from the X-ray detection module according to the placement position of the package to be detected on the conveyor belt of the security inspection device and the position of the detection area of the security inspection device. Target X-ray image of the inspection package.

After determining that the package and X-ray image association unit includes the target X-ray image of the package to be detected, the package image of the package to be detected, the first package feature of the package to be detected, the target X-ray image, and the first package feature of the target X-ray image Two package features, establishing the corresponding relationship between the package to be detected and the target X-ray image.

In step S1011, the information association module establishes the corresponding relationship between the package to be detected, the target person and the X-ray image of the target.

In this step, the personnel, package and X-ray image association unit shown in Figure 11 can be based on the correspondence between the package to be detected and the target person established by the personnel and package association unit, and the to-be-detected relationship established by the package and X-ray association unit. The corresponding relationship between the package and the target X-ray image establishes the corresponding relationship among the package to be detected, the target person and the target X-ray image.

In step S1012, the information association module stores the corresponding relationship among the package to be detected, the target person and the target X-ray image in the information storage module.

Based on the same inventive concept, according to the information associating method provided by the above embodiment of the present application, the embodiment of the present application further provides an information associating device. As shown in FIG. 12 , FIG. 12 is a schematic structural diagram of an information associating device provided in an embodiment of the present application. The device is the above security inspection equipment. The security inspection equipment may include the following modules.

The first acquiring module 1201 is configured to acquire the monitoring image at the entrance of the security inspection equipment;

The first determining module 1202 is configured to determine the package to be detected in the surveillance image and the target person to whom the package to be detected belongs;

The second determining module 1203 is configured to determine the target X-ray image collected by the security inspection equipment including the package to be detected according to the placement position of the package to be detected on the conveyor belt of the security inspection device and the position of the detection area of the security inspection device;

The establishment module 1204 is used to establish the correspondence between the package to be detected, the target person and the X-ray image of the target.

Optionally, a pressure sensor may be included on the conveyor belt of the above-mentioned security inspection equipment;

The above-mentioned first acquisition module 1201 can be specifically configured to acquire the monitoring image at the entrance of the security inspection device collected at the current moment when the pressure sensor detects that there is a package placed on the conveyor belt.

Optionally, the above-mentioned first determining module 1202 may specifically be used to determine the package on the conveyor belt of the security inspection equipment in the monitoring image as the package to be detected;

The person closest to the package to be detected in the surveillance image is determined as the target person to whom the package to be detected belongs.

Optionally, the above-mentioned second determination module 1203 may include:

The first determination sub-module is used to determine the distance between the package to be detected and the detection area according to the placement position of the package to be detected on the conveyor belt of the security inspection device and the position of the detection area of the security inspection device;

The second determination submodule is used to determine the time range for the package to be detected to be transmitted to the detection area based on the determined distance and the transmission speed of the conveyor belt;

The third determination sub-module is used to determine the target X-ray image collected by the security inspection device including the package to be detected from the X-ray images collected by the security inspection device within the time range.

The above-mentioned information association device may also include:

The identification module is used to identify the monitoring image, and obtain the placement position of the package to be detected on the conveyor belt of the security inspection device; and/or

The third determination module is configured to determine the position of the pressure signal detected by the pressure sensor as the placement position of the package to be detected on the conveyor belt.

Optionally, the above-mentioned information association device may also include:

The second acquisition module is used to obtain the first package feature of the package to be detected in the monitoring image before determining the target X-ray image collected by the security inspection device including the package to be detected from the X-ray images collected by the security inspection device within the time range;

The third determination sub-module is specifically used to obtain the second package characteristics of the package in each X-ray image collected by the security inspection equipment within the time range;

The features of the first package are matched with the features of the second package in each X-ray image to obtain the target X-ray image collected by the security inspection equipment including the package to be detected.

Optionally, the above-mentioned package features may include multi-dimensional features;

The third determination sub-module is specifically used to calculate the weighted sum of the matching confidence degrees corresponding to each feature dimension in the second package feature of the X-ray image and the first package feature for each X-ray image, to obtain the X-ray The matching confidence between the package in the image and the package to be detected;

Based on the matching confidence between the package in each X-ray image and the package to be detected, and the target person to whom the package to be detected belongs, a logical judgment is made on each X-ray image to determine the target collected by the security inspection equipment, including the package to be detected X-ray image.

Optionally, the aforementioned package characteristics may include one or more of package weight, package size and package shape.

Optionally, the above-mentioned establishment module 1204 can be specifically used to obtain the package image and the person image of the target person before the package to be detected enters the detection area; based on the package image, person image and target X-ray image, establish The corresponding relationship with the target X-ray image.

The alarm module is configured to output an alarm message based on the correspondence between the package to be detected, the target person, and the target X-ray image when it is detected that the package in the target X-ray image contains abnormal items.

With the device provided in the embodiment of the present application, by acquiring the monitoring image at the entrance of the security inspection equipment, the target person to whom the package to be detected in the monitoring image belongs can be determined, that is, the target person associated with the package to be detected can be determined. In addition, according to the placement position of the package to be detected on the conveyor belt of the security inspection device and the position of the detection area of the security inspection device, the target X-ray image collected by the security inspection device including the package to be detected can be determined, that is, it can be determined that the package to be detected is associated with X-ray image of the target. This enables the security inspection equipment to establish the correspondence between the package to be detected, the target person, and the target X-ray image according to the target person to whom the package to be detected belongs, and the target X-ray image including the package to be detected, that is, the package, person and X-ray image The three are connected. Since the association process is directly carried out by the security inspection equipment based on the collected monitoring images and X-ray images, no manual participation is required, which saves the time required for manual association, and reduces manual association in scenes with a large number of people or complex environments Possibility of errors increases the accuracy of the efficiency of information association.

Based on the same inventive concept, and according to the information association method provided by the above embodiments of the present application, the embodiments of the present application also provide a security inspection device. As shown in FIG. 13 , FIG. 13 is a fourth structural schematic diagram of the security inspection device provided by the embodiment of the present application. The security inspection equipment may include a camera 1301, a conveyor belt 1302, an image analysis module 1303, an X-ray detection module 1304, and an information association module 1305; the conveyor belt 1302 may include a pressure sensor 1306:

The above-mentioned camera 1301 is used to obtain the monitoring image at the entrance of the security inspection equipment collected at the current moment when the pressure sensor 1306 detects that there is a package placed on the conveyor belt 1302;

The above-mentioned conveyor belt 1302 is used for conveying parcels;

The above-mentioned image analysis module 1303 is used to obtain the monitoring image; determine the package to be detected in the monitoring image;

The above-mentioned X-ray detection module 1304 is used to collect the X-ray image of the package when the package on the conveyor belt 1302 is transported to the detection area, and perform abnormal detection on the package based on the X-ray image;

The above-mentioned information association module 1305 is used to determine the target person to whom the package to be detected belongs in the monitoring image; according to the placement position of the package to be detected on the conveyor belt 1302 of the security inspection equipment, and the position of the detection area of the security inspection equipment, determine the information collected by the security inspection equipment. Target X-ray image of the package to be detected; establish correspondence between the package to be detected, the target person and the target X-ray image.

Optionally, the above-mentioned security inspection equipment may also include a pressure analysis module, a face capture module, an information storage module and an information display module;

The above-mentioned pressure analysis module is used to obtain the pressure signal generated by the pressure sensor 1306 when the package to be detected is placed on the conveyor belt 1302, and determine the pressure signal generated by the pressure sensor 1306 according to the preset corresponding relationship between the pressure signal and the pressure value The corresponding pressure value is used as the package weight of the package to be detected;

The above-mentioned face capture module is used to capture the face of the person at the entrance of the security inspection equipment when the pressure sensor 1306 detects that there is a package placed on the conveyor belt 1302;

The above-mentioned information storage module is used to store the corresponding relationship between the package to be detected, the target person and the target X-ray image;

The above information display module is configured to output an alarm message based on the correspondence between the package to be detected, the target person and the target X-ray image when it is detected that the package in the target X-ray image contains abnormal items.

With the security inspection device provided in the embodiment of the present application, by acquiring the monitoring image at the entrance of the security inspection device, the target person to whom the package to be detected in the monitoring image belongs can be determined, that is, the target person associated with the package to be detected can be determined. In addition, according to the placement position of the package to be detected on the conveyor belt of the security inspection device and the position of the detection area of the security inspection device, the target X-ray image collected by the security inspection device including the package to be detected can be determined, that is, it can be determined that the package to be detected is associated with X-ray image of the target. This enables the security inspection equipment to establish the correspondence between the package to be detected, the target person, and the target X-ray image according to the target person to whom the package to be detected belongs, and the target X-ray image including the package to be detected, that is, the package, person and X-ray image The three are connected. Since the association process is directly carried out by the security inspection equipment based on the collected monitoring images and X-ray images, no manual participation is required, which saves the time required for manual association, and reduces manual association in scenes with a large number of people or complex environments Possibility of errors increases the accuracy of the efficiency of information association.

Based on the same inventive concept, according to the information association method provided by the above embodiments of the present application, the embodiments of the present application also provide a computer-readable storage medium, in which a computer program is stored in the computer-readable storage medium, and the computer program is The processor implements the steps of any one of the above information association methods when executed.

Based on the same inventive concept, according to the information association method provided by the above-mentioned embodiments of the present application, the embodiments of the present application also provide a computer program product containing instructions, which, when run on a computer, cause the computer to execute any of the above-mentioned embodiments. An information association method.

In the above embodiments, all or part of them may be implemented by software, hardware, firmware or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the processes or functions according to the embodiments of the present application will be generated in whole or in part. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website, computer, server or data center Transmission to another website site, computer, server, or data center by wired (eg, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server or a data center integrated with one or more available media. The available medium may be a magnetic medium (for example, a floppy disk, a hard disk, or a magnetic tape), an optical medium (for example, DVD), or a semiconductor medium (for example, a Solid State Disk (SSD)).

It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is a relationship between these entities or operations. There is no such actual relationship or order between them. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or device. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

Each embodiment in this specification is described in a related manner, the same and similar parts of each embodiment can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for embodiments such as devices, security inspection equipment, computer-readable storage media, and computer program products, since they are basically similar to the method embodiments, the description is relatively simple. For relevant information, please refer to the part of the description of the method embodiments. .

The above descriptions are only preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application are included within the protection scope of this application.

The above descriptions are only preferred embodiments of the application, and are not intended to limit the application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the application shall be included in the protection of the application. within the range.

Claims

An information association method is characterized in that it is applied to security inspection equipment, and the method includes:

Acquiring the monitoring image at the entrance of the security inspection equipment;

determining the package to be detected in the surveillance image and the target person to whom the package to be detected belongs;

According to the placement position of the package to be detected on the conveyor belt of the security inspection device and the position of the detection area of the security inspection device, determine the target X-ray image collected by the security inspection device including the package to be detected;

A corresponding relationship between the package to be detected, the target person, and the target X-ray image is established.
The method according to claim 1, wherein the conveyor belt of the security inspection equipment includes a pressure sensor;

The step of acquiring the monitoring image at the entrance of the security inspection equipment includes:

When it is detected by the pressure sensor that there is a package placed on the conveyor belt, the monitoring image at the entrance of the security inspection device collected at the current moment is acquired.
The method according to claim 1, wherein the step of determining the package to be detected in the surveillance image and the target person to whom the package to be detected belongs comprises:

Determining the package on the conveyor belt of the security inspection device in the monitoring image as the package to be detected;

The person closest to the package to be detected in the monitoring image is determined as the target person to whom the package to be detected belongs.
The method according to claim 1, characterized in that, according to the placement position of the package to be detected on the conveyor belt of the security inspection equipment and the location of the detection area of the security inspection equipment, it is determined that the collection of the security inspection equipment The step of including the target X-ray image of the package to be detected includes:

determining the distance between the package to be detected and the detection area according to the placement position of the package to be detected on the conveyor belt of the security inspection device and the position of the detection area of the security inspection device;

Based on the determined distance and the transmission speed of the conveyor belt, determine the time range for the package to be detected to be transmitted to the detection area;

From the X-ray images collected by the security inspection equipment within the time range, determine a target X-ray image collected by the security inspection equipment that includes the package to be inspected.
The method according to claim 4, characterized in that, the conveyor belt of the security inspection equipment includes a pressure sensor;

The method also includes:

Recognizing the monitoring image to obtain the placement position of the package to be detected on the conveyor belt of the security inspection device; and/or

The position of the pressure signal detected by the pressure sensor is determined as the placement position of the package to be detected on the conveyor belt.
The method according to claim 4, characterized in that, among the X-ray images collected by the security inspection equipment within the time range, before determining the target X-ray image collected by the security inspection equipment including the package to be inspected ,Also includes:

Acquiring the first package feature of the package to be detected in the monitoring image;

The step of determining, from the X-ray images collected by the security inspection equipment within the time range, the target X-ray images collected by the security inspection equipment that include the package to be detected includes:

Obtain the second package feature of the package in each X-ray image collected by the security inspection equipment within the time range;

The first package feature is matched with the second package feature of the package in each X-ray image to obtain a target X-ray image collected by the security inspection device including the package to be detected.
The method according to claim 6, wherein the package feature comprises a multi-dimensional feature;

The step of matching the characteristics of the first package with the characteristics of the second package in each X-ray image to obtain the target X-ray image collected by the security inspection equipment and including the package to be detected includes:

For each X-ray image, calculate the weighted sum of the matching confidence degrees corresponding to the second package feature of the X-ray image and each feature dimension in the first package feature, and obtain the package in the X-ray image and the The matching confidence between the packages to be detected;

Based on the matching confidence between the package in each X-ray image and the package to be detected, and the target person to whom the package to be detected belongs, perform a logical judgment on each X-ray image to determine the security inspection equipment The collected target X-ray image includes the package to be detected.
The method according to claim 6, wherein the package characteristics include one or more of package weight, package size and package shape.
The method according to claim 1, wherein the step of establishing the corresponding relationship between the package to be detected, the target person and the target X-ray image comprises:

Acquiring the image of the package before the package to be detected enters the detection area and the person image of the target person;

Based on the package image, the person image, and the target X-ray image, a corresponding relationship between the package to be detected, the target person, and the target X-ray image is established.
The method according to claim 1, further comprising:

When it is detected that the package in the X-ray image of the target contains abnormal items, an alarm message is output based on the correspondence between the package to be detected, the person of the target, and the X-ray image of the target.
An information association device is characterized in that it is applied to security inspection equipment, and the device includes:

The first acquisition module is used to acquire the monitoring image at the entrance of the security inspection equipment;

A first determining module, configured to determine the package to be detected in the surveillance image and the target person to whom the package to be detected belongs;

The second determining module is configured to determine the information collected by the security inspection equipment including the package to be detected according to the placement position of the package to be detected on the conveyor belt of the security inspection device and the position of the detection area of the security inspection device. Target X-ray image;

An establishment module, configured to establish correspondence between the package to be detected, the target person, and the target X-ray image.
A security inspection device, characterized in that the security inspection device includes a camera, a conveyor belt, an image analysis module, an X-ray detection module, and an information association module; the conveyor belt includes a pressure sensor:

The camera is used to acquire the monitoring image at the entrance of the security inspection equipment collected at the current moment when the pressure sensor detects that there is a package placed on the conveyor belt;

The conveyor belt is used for conveying parcels;

The image analysis module is used to obtain the monitoring image; determine the package to be detected in the monitoring image;

The X-ray detection module is configured to collect an X-ray image of the package when the package on the conveyor belt is transported to the detection area, and perform abnormal detection on the package based on the X-ray image;

The information association module is configured to determine the target person to whom the package to be detected belongs in the monitoring image; according to the placement position of the package to be detected on the conveyor belt of the security inspection device and the detection area of the security inspection device Determine the location of the target X-ray image collected by the security inspection equipment including the package to be detected; establish a corresponding relationship between the package to be detected, the target person, and the target X-ray image.
The security inspection device according to claim 12, wherein the security inspection device further comprises a pressure analysis module, a face capture module, an information storage module and an information display module;

The pressure analysis module is configured to obtain the pressure signal generated by the pressure sensor when the package to be detected is placed on the conveyor belt, and determine the pressure sensor pressure signal according to the preset correspondence between the pressure signal and the pressure value. The pressure value corresponding to the generated pressure signal is used as the package weight of the package to be detected;

The face capture module is used to capture the face of the person at the entrance of the security inspection device when the pressure sensor detects that a package is placed on the conveyor belt;

The information storage module is used to store the corresponding relationship between the package to be detected, the target person and the target X-ray image;

The information display module is configured to output an alarm based on the correspondence between the package to be detected, the target person, and the target X-ray image when detecting that the package in the target X-ray image contains abnormal items information.
A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method steps described in any one of claims 1-10 are implemented.
A computer program product comprising instructions which, when run on a computer, cause the computer to perform the method of any one of claims 1-10.