CN115469375B - System for associating visible light image and X-ray image of person bag - Google Patents
System for associating visible light image and X-ray image of person bag Download PDFInfo
- Publication number
- CN115469375B CN115469375B CN202211417597.0A CN202211417597A CN115469375B CN 115469375 B CN115469375 B CN 115469375B CN 202211417597 A CN202211417597 A CN 202211417597A CN 115469375 B CN115469375 B CN 115469375B
- Authority
- CN
- China
- Prior art keywords
- visible light
- ray
- package
- image
- person
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01V—GEOPHYSICS; GRAVITATIONAL MEASUREMENTS; DETECTING MASSES OR OBJECTS; TAGS
- G01V11/00—Prospecting or detecting by methods combining techniques covered by two or more of main groups G01V1/00 - G01V9/00
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B65—CONVEYING; PACKING; STORING; HANDLING THIN OR FILAMENTARY MATERIAL
- B65G—TRANSPORT OR STORAGE DEVICES, e.g. CONVEYORS FOR LOADING OR TIPPING, SHOP CONVEYOR SYSTEMS OR PNEUMATIC TUBE CONVEYORS
- B65G43/00—Control devices, e.g. for safety, warning or fault-correcting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10116—X-ray image
Landscapes
- Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Theoretical Computer Science (AREA)
- Geophysics (AREA)
- General Life Sciences & Earth Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Psychiatry (AREA)
- Social Psychology (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Analysing Materials By The Use Of Radiation (AREA)
Abstract
The invention discloses a system for associating a human package visible light image with an X-ray image, which captures the action of delivering articles to a conveyor belt by people by using delivery action capture, wherein the action image comprises the people and the belonged packages, belongs to a human-package visible light association image, and is used as an image to be associated. The invention relates to a pre-processing idea, which is characterized in that when packages fall onto a conveying belt and a person is triggered to capture the packages in time when the person makes a package delivery action, an image containing both the person and the packages is obtained, the situation that the person leaves quickly or the person cannot capture the packages due to errors in system responsiveness or pressure sensor responsiveness is avoided, the association between a visible light package picture and a person head portrait is realized without calculating the nearest person according to the packages, the picture containing both the person and the packages is directly used as an object to be associated of an X-ray package picture, and the high-accuracy association system is provided.
Description
Technical Field
The invention relates to the technical field of security inspection, in particular to a system for associating a human bag visible light image with an X-ray image.
Background
In a subway security inspection system, an X-ray machine is generally used for performing X-ray transmission scanning on a package carried by a person to obtain a package X-ray image, then the package X-ray image is pushed to a contraband identification system for identification, if contraband information is identified in the package X-ray image, an alarm is executed, and the package is subjected to escorting inspection.
Since after contraband is identified, people who are related to the contraband need to be searched for carrying out a mortgage examination treatment, no mature people bag related technology is available on the market at present to support the requirement. Therefore, a technique for correlating the person image, the package visible light image, and the package X-ray image is required.
The related art, which is more related at present, is a pressure sensing correlation method, for example, patent application 202110644127.7. The design idea of the technology is as follows: the method comprises the steps that a pressure sensor is arranged on a transmission belt, when the pressure sensor senses pressure, a monitoring image of an entrance of the security inspection equipment is obtained through a visible light camera, a package in the monitoring image is driven to be a package to be detected, and a person which is in the monitoring image and is closest to the package to be detected is driven to be a target person to which the package to be detected belongs.
We have found that the above method is triggered by pressure sensing of the package after it is placed on the conveyor belt, and when the system sensitivity is poor, the target person to which the package belongs is far away, so that the image obtained does not contain the person closest to the package to be detected. Meanwhile, the pressure sensor is installed on the conveying belt, and whether the pressure sensor can influence the X-ray shooting and whether the pressure sensor can cause performance influence after being irradiated by the X-ray are all to be confirmed when the pressure sensor enters the X-ray shooting range.
Therefore, there is a need for developing an association system that can improve the accuracy of association, and an association system that is not affected by the X-ray irradiation apparatus.
Disclosure of Invention
The invention aims to provide a system for associating a visible light image and an X-ray image of a person bag.
In order to improve the association accuracy, the invention provides a system which takes a picture captured by the action of delivering articles to a conveyor belt as a picture to be associated. The method comprises the following specific steps:
on the one hand, induction system shoots, including the X-ray machine, the X-ray machine includes the separation storehouse, run through the conveyer belt subassembly in separation storehouse, the conveyer belt subassembly includes the transmission roller, the transmission band of suit on the transmission roller, a serial communication port, still include vertical to delivering position detection device, vertical to delivering position detection device includes the sensitization array, the sensitization array includes a plurality of alignment, equidistant delivery position detects uses the sensitization head, detect and arrange in at least one side that is located the transmission direction of the outside transmission band in separation storehouse with the sensitization head, it is parallel with the transmission direction of transmission band with the range line of sensitization head to detect, detect and to use the photoinduction face of sensitization head directly over, it is equipped with the luminous surface and directly under to detect with the range line of sensitization head and delivers position detection and use normally open light source.
In order to realize the action of capturing the delivered articles of people on the conveying belt, the invention constructs a motion capture device on the basis of the existing X-ray machine, the motion capture device comprises a photosensitive array consisting of a plurality of photosensitive heads for detecting delivery positions which are arranged in a straight line and are at equal intervals, and a normally open light source for detecting the delivery positions which is positioned right above the photosensitive array, wherein the arrangement line of the photosensitive array is parallel to the transmission direction of the conveying belt and is arranged on at least one side of the transmission direction of the conveying belt positioned outside the isolation bin, so that a vertically open light curtain consisting of the photosensitive array and the light source for detecting the delivery positions is constructed in the vertical direction of the side surface of the conveying belt. Personnel can destroy the light curtain earlier when delivering article to the conveyer belt, then article can fall into on the conveyer belt, and wherein, it has a plurality of to deliver the position and detect with photosensitive head, delivers the position and detects the length that has covered whole conveyer belt with laying of photosensitive head, and the user is when delivering the parcel, and the parcel all can be delivered from the side optional position point of conveyer belt. Generally, when packages are delivered, only the correlation between the adjacent photosensitive heads for detecting the delivery positions and the normally-open light source for detecting the delivery positions is destroyed. When the correlation is damaged, the visible light camera is triggered to obtain the picture captured by the action of the person delivering the article to the conveying belt. The correlation is broken, which is hereinafter referred to as a change state of the switching value in the present invention.
The difference from the moment of breaking the light curtain to the moment of the contact of the article with the conveyor belt is very short and therefore negligible. Therefore, in the invention, the delivery personnel can be captured certainly because the delivery action is captured. Thus, a picture is obtained that contains the person and the package touched by the person. The time of the picture and the time of the package contacting the conveying belt are very short, so that the picture can be ignored and can be directly used as the picture to be associated. The invention takes the picture as the picture to be associated, and can greatly ensure that the associated picture contains the parcel and the personnel, thereby improving the accuracy of association.
The delivery position detection photosensitive head and the delivery position detection normally-open light source can be understood as a common correlation type photoelectric switch, the delivery position detection photosensitive head can be understood as a photosensitive end of the correlation type photoelectric switch, and the delivery position detection normally-open light source can be understood as a light emitting end of the correlation type photoelectric switch. In the invention, the delivery position detection photosensitive head and the delivery position detection normally-open light source are fixed devices and do not move along with the conveying belt.
When installing transmission extension front end device additional, the delivery position detects and can extend to install transmission extension front end device additional on with the sensitization head, covers the total length of installing transmission extension front end device additional.
Further, the following installation methods of the delivery position detecting photosensitive head can be adopted, but any light curtain structure which is constructed to have the lateral and vertical property of the conveyor belt without departing from the design concept of the invention is within the protection scope of the invention.
The first installation mode: the conveying belt component further comprises conveying roller side plates positioned at two ends of the conveying roller, and the delivery position detection photosensitive heads are sequentially arranged on the upper surfaces, facing the right upper side, of the conveying roller side plates along the length direction of the conveying roller side plates positioned outside the isolation bin.
The second installation mode: the conveying belt assembly further comprises conveying roller side plates positioned at two ends of the conveying roller, the photosensitive arrays are assembled on a substrate, the substrate is mounted on the upper surface of the conveying roller side plate facing to the right upper side in a stacked mode or mounted on the outer side surface of the conveying roller side plate in parallel, and the photosensitive heads for delivery position detection are sequentially arranged along the length direction of the conveying roller side plate positioned outside the isolation bin.
Preferably, the normally open light source for detecting the delivery position is a line laser light source or a plurality of light source emitters,
when the normally open light source for detecting the delivery position is a line laser light source, the projection line of the line laser light source covers the arrangement line of the photosensitive heads for detection.
Preferably, when the normally-open light source for delivery position detection is a plurality of light source emitters corresponding to the light sensing heads one to one, the light rays of each light source emitter vertically project to the light sensing heads for detection right below one to one.
The line laser light source is a light source capable of emitting a line as a projection surface. In the invention, because the delivery positions need to be calibrated, a plurality of photosensitive heads are used for detection, and the length of the conveying belt extending out of the isolation bin is covered. The light source may be a normally open type, and may be the 1 line laser light source covering all the detection photosensitive heads, or may be a one-to-one corresponding light source. The invention takes the light signal received by the light sensing head for detection as the on-state switching value, and takes the personnel delivery action to destroy the light curtain, the light sensing head for detection is shielded, and the light signal is not received as the off-state, when the on-state is changed into the off-state, the delivery action is taken to occur, thereby triggering the visible light shooting, and carrying out the parcel and personnel capture. Meanwhile, the video can be shot by visible light all the time, and the video frame is searched by using the moment when the delivery action is triggered, so that a picture containing the packages and the personnel is obtained. The method comprises the following specific steps:
a capture trigger method:
the device also comprises a visible light camera, a visual angle area of the visible light camera covers a standing area and a conveying belt when the personnel carry out parcel delivery, the visible light camera is in a closed state,
also comprises a photographing controller which receives the switching value of the photosensitive head for detecting the delivery position and takes photosThe photographing controller sends a photographing instruction to the visible light camera after detecting that the switching value of the photosensitive head for detecting any delivery position is changed from on to off or from off to on, the visible light camera performs photographing for a viewing angle area once to generate a photographed image, the photographing controller obtains the photographed image from the visible light camera, and the photographed image is recorded as: human package visible light picture P1, storage human package visible light picture P1 and imaging time t thereof k 。
The picture searching method comprises the following steps:
the device also comprises a visible light camera, a visual angle area of the visible light camera covers a standing area and a conveying belt when people carry out parcel delivery, the visible light camera is in a continuous shooting state,
the shooting controller is used for storing a shooting video of the visible light camera, receiving the switching value of the light sensing head for detecting the delivery position, searching a frame image at the moment in the shooting video according to the changing state moment of the switching value after the shooting controller detects that the switching value of any light sensing head for detecting the delivery position is changed from on to off or from off to on, and recording the frame image as: human package visible light picture P1, storage human package visible light picture P1 and imaging time t thereof k 。
On the other hand, on the basis of the above, the invention constructs a photo sensing system, by which the picture of the person at the time of delivering the parcel can be output, and the picture is taken as the object to be associated, so that the person package visible light graph P1 and the imaging time t thereof need to be based on k And the delivery location point of the person at the time the package is delivered, as well as other data needed for association. Based on the photographing induction system, the invention constructs a system for associating the visible light image and the X-ray image of the person bag, and the system for associating the visible light image and the X-ray image of the person bag calculates the time of the parcel after being scanned by the X-ray according to the delivery position point, thereby performing association based on close time, which is specifically as follows:
the system for associating the visible light image and the X-ray image of the person bag comprises an association computer, a package size detection system, a package X-ray photographing system and a photographing induction system;
a wrap size detection system for detecting the occupied length L2 of the wrap on the conveyor belt,
a photographing induction system for calculating the delivery position point according to the position point of the photosensitive head for detecting the change state of the triggered switching value, detecting the visible light pattern P1 of the person bag when the person carries out the package delivery action at the delivery position and the imaging time t thereof k The visible light graph P1 of the bag contains both the characters and the packages;
the parcel X-ray photographing system is used for carrying out X-ray scanning on parcels, splicing all frame scanning images corresponding to one parcel into a complete parcel X-ray image P2 after the last frame scanning image of one parcel appears, and taking the imaging time of the last frame scanning image as the imaging time t of the parcel X-ray image P2 x Store parcel X-ray map P2 and its imaging time t x ；
The correlation computer is used for calculating a total travel L of the package complete scanning according to a formula 2, wherein the formula 2 is that L = L2+ L1, and L1 is a distance from a delivery position point to an X-ray scanning position point of the package X-ray photographing system;
the correlation computer is used for calculating the estimated imaging time t of the scanning image of the last frame after the package completely passes through the X-ray scanning position point according to the formula 3 y Equation 3 y =t k + L/V, V is the moving speed of the transmission belt;
a correlation computer for estimating the imaging time t y Closest imaging time t x The corresponding parcel X-ray map P2 is marked as the imaging instant t k And obtaining the association pair of the visible light image P1 of the person bag and the parcel X-ray image P2 by the corresponding association image of the visible light image P1 of the person bag.
It should be noted that: in the present system, the position of each inspection photosensitive head is designed in advance, that is, the distance from the position point of each inspection photosensitive head to the X-ray scanning position point of the envelope radiography system is known. If a certain photosensitive head for detection is triggered, the distance between the photosensitive head for detection and an X-ray scanning position point of a parcel X-ray photographing system can be directly obtained, and the position point of the photosensitive head for detection of the change state of a switching value can be triggered to calculate a delivery position point, and the delivery position and a parcel fall are calculatedThere is little difference in position on the conveyor belt and therefore the delivery location point can be directly considered as the initial location of the package. Through the movement speed and the movement distance, the estimated imaging time t of the scanning image of the last frame after the package completely passes through the X-ray scanning position point can be calculated y . Thus, the imaging time t can be estimated y The visible light image P1 of the person package and the X-ray image P2 of the package are associated.
Preferably, the package radiography system includes: the X-ray machine and the X-ray machine comprise an isolation bin and an X-ray scanning device which is arranged in the isolation bin, and the X-ray scanning position point is the position point of the X-ray scanning device.
Preferably, when there are a plurality of detection heads that trigger the change state of the switching value, the delivery position point is a midpoint of the position points of the plurality of detection heads that trigger the change state of the switching value or a position point of the detection head farthest from the isolation chamber.
Preferably, the package size detection system comprises: the X-ray machine comprises an isolation bin and a conveyor belt assembly penetrating through the isolation bin, wherein the conveyor belt assembly comprises a conveying roller and a conveyor belt sleeved on the conveying roller, the isolation bin is provided with an inlet, a transverse size detection device is arranged at the inlet, the transverse size detection device is a photoelectric switch for package size detection, the photoelectric switch for package size detection is a correlation type photoelectric switch or a reflection type photoelectric switch, and the normal line of the light emitting surface of the correlation type photoelectric switch and the reflection type photoelectric switch is parallel to the horizontal plane and is perpendicular to the conveying direction of the conveyor belt;
still include size record controller, size record controller record parcel size detects with photoelectric switch's switching value, and the controller of shooing detects that parcel size detects with photoelectric switch's switching value by record time T1 when switching to the change state of switch, record time T2 when the change state of switching to the switch, calculates the length that occupies of parcel on the transmission band according to formula 1, formula 1: l2= (| time T1-time T2 |) V, L2 is the occupied length of the package on the conveyor belt, and V is the moving speed of the conveyor belt.
The beneficial effects of the invention are as follows: different from a pressure sensor of a comparison file triggering and capturing person, the invention is a pre-processing idea, namely, when the parcel falls on a conveying belt, the capturing person is triggered immediately when the person makes the parcel delivery action, so that an image containing both the person and the parcel is obtained, the situation that the person can not be captured due to quick departure of the person or system responsiveness or pressure sensor responsiveness errors is avoided, the association between a visible light parcel picture and a person head portrait is realized without calculating the nearest person according to the parcel, the picture containing both the person and the parcel is directly used as an object to be associated of an X-ray parcel picture, and a high-accuracy association system is provided. In addition, the electronic element of the invention can not collide with the package, has high stability, can not be easily damaged generally, and can not cause any obstructive influence on personnel.
Drawings
FIG. 1 is a schematic view of the transmission direction of an X-ray machine according to the present invention.
FIG. 2 is a schematic view of the X-ray machine of the present invention from a side view.
FIG. 3 is a diagram illustrating data interaction according to the present invention.
1. X-ray machine, 2, visible light camera, 3, delivery position detects with normally open light source, 4, delivery position detects with sensitization head, 5, the controller of shooing, 6, associated computer, 7, parcel size detects with photoelectric switch (correlation formula or reflective), 8, transmission extension front end device, 10A, conveyer belt subassembly, 10B, separation storehouse, 11, transmission roller curb plate, 12, the transmission band, 13, X-ray scanning device, 14, import.
Detailed Description
The present invention will be described in further detail with reference to examples and drawings, but the present invention is not limited to these examples.
Example 1
Referring to the accompanying drawings in which figures 1-3 are shown,
as shown in fig. 1 and fig. 2, an X-ray machine 1 according to the present invention is an X-ray machine 1 for package security inspection, the X-ray machine 1 is a prior art, the X-ray machine 1 includes an isolation bin 10B, a conveyor belt assembly 10A penetrating the isolation bin 10B, the conveyor belt assembly 10A includes a conveying roller, and a conveyor belt sleeved on the conveying roller.
The photographing induction system comprises an X-ray machine 1, wherein the X-ray machine 1 comprises an isolation bin 10B, a conveyor belt assembly 10A penetrating through the isolation bin 10B, the conveyor belt assembly 10A comprises a transmission roller and a conveyor belt sleeved on the transmission roller, and the photographing induction system is characterized by further comprising a vertical delivery position detection device, the vertical delivery position detection device comprises a photosensitive array, the photosensitive array comprises a plurality of linear arrays, photosensitive heads 4 for delivery position detection at equal intervals, the photosensitive heads 4 for detection are arranged on at least one side of the transmission direction of the conveyor belt positioned outside the isolation bin, the arrangement line of the photosensitive heads 4 for detection is parallel to the transmission direction of the conveyor belt, the photosensitive surface of the photosensitive heads 4 for detection faces to the upper side, and a normally open light source 3 for delivery position detection with a luminous surface facing to the lower side is arranged right above the arrangement line of the photosensitive heads 4 for detection. The general delivery position detection is installed from the ceiling of the site by using a normally-open light source 3.
In order to realize the action of capturing the delivered articles of people on the conveyor belt, the invention constructs a action capturing device on the basis of the prior X-ray machine 1, wherein the action capturing device comprises a photosensitive array consisting of a plurality of photosensitive heads 4 for detecting the delivery positions and arranged in a straight line at equal intervals and a normally open light source 3 for detecting the delivery positions, which is positioned right above the photosensitive array, wherein the arrangement line of the photosensitive array is parallel to the transmission direction of the conveyor belt and arranged on at least one side of the transmission direction of the conveyor belt outside the isolation bin, which is equivalent to that a light curtain in the vertical direction consisting of the photosensitive array and the normally open light source 3 for detecting the delivery positions is constructed in the vertical direction of the side surface of the conveyor belt. Personnel can destroy the light curtain earlier when delivering article to the conveyer belt, then article can fall into on the conveyer belt, and wherein, it has a plurality of to deliver position detection with photosensitive head 4, delivers position detection with photosensitive head 4 lay the length that the region has covered whole conveyer belt, the user when delivering the parcel, delivers the parcel all can from the side optional position point of conveyer belt. Generally, when delivering the parcel, only the correlation between the adjacent delivery position detection photosensitive heads 4 and the normally open light source 3 for the delivery position detection is destroyed. When the correlation is damaged, the visible light camera is triggered to obtain pictures captured by the action of delivering articles to the conveying belt. The correlation is broken, which is hereinafter referred to as a change state of the switching value in the present invention.
The difference from the moment of breaking the light curtain to the moment of the contact of the article with the conveyor belt is very short and therefore negligible. Therefore, in the invention, the delivery personnel can be captured certainly because the delivery action is captured. Thus, a picture is obtained that contains the person and the package touched by the person. The time of the picture and the time of the package contacting the conveying belt are very short, so that the picture can be ignored and can be directly used as the picture to be associated. The invention takes the picture as the picture to be associated, can greatly ensure that the associated picture contains the package part and the personnel, thereby improving the accuracy of association.
The delivery position detection photosensitive head 4 and the delivery position detection normally-open light source 3 can be understood as common correlation type photoelectric switches, the delivery position detection photosensitive head 4 can be understood as a photosensitive end of the correlation type photoelectric switches, and the delivery position detection normally-open light source 3 can be understood as a luminous end of the correlation type photoelectric switches. In the invention, the delivery position detection photosensitive head 4 and the delivery position detection normally-open light source 3 are fixed devices and do not move along with a conveying belt.
As shown in fig. 2, when the transporting extension front-end unit 8 is attached, the delivery position detecting photosensitive head 4 may be arranged to extend to the attaching transporting extension front-end unit so as to cover the total length of the attaching transporting extension front-end unit.
Further, the following installation methods of the delivery position detecting photosensitive head can be adopted, but any light curtain structure which is constructed to have the lateral and vertical property of the conveyor belt without departing from the design concept of the invention is within the protection scope of the invention.
The first installation mode: as shown in fig. 1 and 2, the conveyor belt assembly 10A further includes conveying roller side plates 11 at both ends of the conveying roller, and the photosensitive heads for detecting delivery positions are sequentially arranged on the upper surfaces of the conveying roller side plates 11 facing directly upward along the length direction of the conveying roller side plates 11 located outside the isolation bin. As shown in the figure, the delivery position detection photosensitive head is arranged on the upper surface of a conveying roller side plate 11 at one side of a conveying belt; if the X-ray machine is an X-ray machine used in the middle layout, that is, both sides of the transmission belt are pedestrian paths, it is necessary to install the photosensitive heads for detecting the delivery positions on the side plates of the transmission rollers on both sides of the transmission belt. As shown in the figure, assuming that the length of the conveying roller side plate 11 located outside the isolation chamber is 1 meter, 1 delivery position detecting photosensitive heads can be provided every 5cm for a total of 20.
The second installation mode: as not shown, the conveyor belt assembly 10A further includes conveying roller side plates 11 located at both ends of the conveying roller, the photosensitive arrays are assembled on a substrate which is mounted on the upper surface of the conveying roller side plate 11 facing directly above in a stacked manner or on the outer side surface of the conveying roller side plate 11 in parallel, and the photosensitive heads for delivery position detection are sequentially arranged along the length direction of the conveying roller side plate 11 located outside the isolation bin. It is understood that the delivery position detecting photosensitive head is mounted on a substrate, the delivery position detecting photosensitive head faces upward in its entirety, the side surface of the substrate is attached to the outer side surface of the transfer roller side plate 11, or the bottom surface of the substrate is attached to the upper surface of the transfer roller side plate 11.
Preferably, the normally-open light source 3 for detecting the delivery position is a line laser light source or a plurality of light source emitters,
when the normally open light source 3 for detecting the delivery position is a line laser light source, the projection line of the line laser light source covers the arrangement line of the photosensitive heads 4 for detection.
Preferably, when the normally open light source 3 for delivery position detection is a plurality of light source emitters corresponding to the light sensing heads 4 for detection one to one, the light of each light source emitter vertically projects to the light sensing head for detection right below in one to one correspondence.
The line laser light source is a light source capable of emitting a line as a projection surface. In the invention, because the delivery positions need to be calibrated, a plurality of photosensitive heads 4 for detection cover the length of the conveying belt extending out of the isolation bin. The light source may be a normally open type, and may be the 1 line laser light source that covers all the detection photosensitive heads 4, or may be a one-to-one light source. The invention takes the light signal received by the light sensing head 4 for detection as the on-state switching value, and takes the light curtain damaged by the delivery action of the personnel, the light sensing head 4 for detection is shielded, and the light signal is not received as the off-state, when the on-state is changed into the off-state, the delivery action is taken, thereby triggering the visible light shooting, and capturing the parcel and the personnel. Meanwhile, the system can also be shot by visible light all the time, and the video frame is searched by using the moment when the delivery action is triggered, so that a picture containing the parcels and the personnel is obtained. The method comprises the following specific steps:
a capture trigger method:
the device also comprises a visible light camera 2, a visual angle area of the visible light camera 2 covers a standing area and a conveying belt when people carry out parcel delivery, the visible light camera 2 is in a closed state,
still including taking a picture controller 5, the controller 5 of taking a picture receives the switching value of delivering position detection with photographic head 4, the controller 5 of taking a picture detects arbitrary delivery position detection with the switching value of photographic head 4 by opening to the switching or by closing to the on change state after, send the shooting instruction for visible light camera 2, visible light camera 2 carries out once to the regional picture generation of taking a picture of viewing angle, the controller 5 of taking a picture obtains this picture of taking a picture from visible light camera 2, will take a picture and mark as: the human bag visible light image P1 stores the human bag visible light image P1 and the imaging time t thereof k 。
The picture searching method comprises the following steps:
the system also comprises a visible light camera 2, a standing area and a conveying belt when the visible light camera 2 covers the parcel delivery of the personnel, the visible light camera 2 is in a continuous shooting state,
the device also comprises a photographing controller 5, the photographing controller 5 stores a photographing video of the visible light camera 2, the photographing controller 5 receives the switching value of the light sensing head 4 for detecting the delivery position, and after the photographing controller 5 detects the switching value of any light sensing head 4 for detecting the delivery position from on to off or from off to on, a frame image at the moment in the photographing video is searched according to the moment of the changing state of the switching value, and the frame image is recorded as: human package visible light picture P1, storage human package visible light picture P1 and imaging time t thereof k 。
Example 2, as shown in the figure1. Fig. 2 and fig. 3, on the basis of the above embodiment 1, a photo sensing system is constructed, by which a picture of a person at the time of delivering a parcel can be output, the picture is taken as an object to be associated, and therefore, it is necessary to use the above visible light map P1 of the parcel and the imaging time t thereof k And the delivery location point of the person at the time the package is delivered, as well as other data needed for association. Based on the photographing sensing system, the invention constructs a system for associating the human bag visible light image with the X-ray image, and the system for associating the human bag visible light image with the X-ray image calculates the time of the parcel after being scanned by the X-ray according to the delivery position point, thereby performing association based on the time proximity, which is specifically as follows:
the system for associating the visible light image and the X-ray image of the person bag comprises an association computer 6, a package size detection system, a package X-ray photographing system and a photographing induction system;
a wrap size detection system for detecting the occupied length L2 of the wrap on the conveyor belt,
a photographing induction system for calculating a delivery position point according to the position point of the detection photosensitive head 4 triggering the change state of the switching value, detecting a visible light figure P1 of a person package when the person carries out package delivery action at the delivery position and an imaging time t thereof k The visible light picture P1 of the person bag contains both the person and the package;
the parcel X-ray photographing system is used for carrying out X-ray scanning on parcels, splicing all frame scanning images corresponding to one parcel into a complete parcel X-ray image P2 after the last frame scanning image of one parcel appears, and taking the imaging time of the last frame scanning image as the imaging time t of the parcel X-ray image P2 x Storing the package X-ray map P2 and the imaging time t thereof x ；
The correlation computer 6 is configured to calculate a total travel L of the package complete scanning according to formula 2, where L = L2+ L1, and L1 is a distance from the delivery position point to an X-ray scanning position point of the package X-ray photographing system;
a correlation computer 6 for calculating the estimated imaging time t of the last frame of scanning image after the package completely passes through the X-ray scanning position point according to the formula 3 y Equation 3 y =t k + L/V, V is the moving speed of the transmission belt;
a correlation computer 6 for estimating the imaging time t y Closest imaging time t x The corresponding parcel X-ray map P2 is marked as the imaging instant t k And obtaining the association pair of the visible light image P1 of the person bag and the parcel X-ray image P2 by the corresponding association image of the visible light image P1 of the person bag.
It should be noted that: in the present system, the position of each of the detecting photosensitive heads 4 is designed in advance, that is, the distance from the position point of each of the detecting photosensitive heads 4 to the X-ray scanning position point of the envelope radiography system is known. If a certain detecting photosensitive head 4 is triggered, the distance between the detecting photosensitive head 4 and an X-ray scanning position point of the package X-ray photographing system can be directly obtained, the position point of the detecting photosensitive head 4 in the change state of the switching value can be triggered to calculate a delivery position point, and the delivery position point can be directly regarded as the initial position of the package because the delivery position is almost not different from the position of the package on a conveying belt. Through the movement speed and the movement distance, the estimated imaging time t of the scanning image of the last frame after the package completely passes through the X-ray scanning position point can be calculated y . Thus, the imaging time t can be estimated y The visible light image P1 of the person package and the X-ray image P2 of the package are associated.
Preferably, the package radiography system includes: the X-ray machine 1 and the X-ray machine include an isolation chamber 10B, and an X-ray scanning device 13 built in the isolation chamber 10B, and the X-ray scanning position point is a position point where the X-ray scanning device 13 is located.
Preferably, when there are a plurality of detection heads 4 for which the switching amount change state is triggered, the delivery position point is a midpoint of the position points of the plurality of detection heads 4 for which the switching amount change state is triggered or a position point of the detection head 4 farthest from the isolation chamber 10B.
Preferably, the package size detection system comprises: the X-ray machine 1 comprises an isolation bin 10B and a conveyor belt component 10A penetrating through the isolation bin 10B, wherein the conveyor belt component 10A comprises a conveying roller and a conveyor belt sleeved on the conveying roller, the isolation bin 10B is provided with an inlet 14, a transverse size detection device is arranged at the inlet, the transverse size detection device is a wrapping size detection photoelectric switch 7, the wrapping size detection photoelectric switch is a correlation type photoelectric switch or a reflection type photoelectric switch, and the normal line of the light emitting surface of the correlation type photoelectric switch and the reflection type photoelectric switch is parallel to the horizontal plane and is perpendicular to the conveying direction of the conveyor belt;
still include size record controller, size record controller record parcel size detects with photoelectric switch 7's switching value, and controller 5 of shooing detects that parcel size detects with photoelectric switch 7's switching value by record time T1 once when the change state of switching on to switching off, record time T2 once when the change state of switching off to switching on, calculate the length that occupies of parcel on the transmission band according to formula 1, formula 1: l2= (| time T1-time T2 |) V, L2 is the occupied length of the package on the conveyor belt, and V is the moving speed of the conveyor belt.
As shown in fig. 3, the size recording controller and the photographing controller are the same device.
The system comprises hardware and software, wherein the hardware part comprises an X-ray machine, a visible light camera, a photographing controller and a correlation computer; the X-ray machine comprises an isolation bin, a transmission part penetrating through the isolation bin, and an optical signal sensing head with a linear array arranged on a side plate exposed out of the transmission part, wherein an optical signal emitter is arranged right above the side plate, a sensing surface vertical to a transmission surface is formed between the optical signal sensing head and the optical signal emitter, and an X-ray shooting device is arranged inside the isolation bin; the photographing view field of the visible light camera can cover a standing area when a person places luggage on a conveyor belt, the photographing controller receives a sensing signal of the optical signal sensing head and then controls the visible light camera to photograph and capture a picture of the action of the person placing the luggage, the visible light camera photographs and then uploads the picture of the action of the person placing the luggage to the associated computer, the photographing controller receives point location information of the optical signal sensing head and uploads the point location information to the associated computer, the X-ray photographing device photographs an X-ray image of the luggage, then the X-ray image and corresponding imaging time of the X-ray image are sent to the associated computer, the associated computer calculates estimated imaging time of the X-ray image of the luggage according to length information, speed information and point location information of the conveyor belt, pairing is carried out according to the estimated imaging time and the latest principle of the imaging time of the X-ray image, and the picture of the action of the person placing the luggage and the X-ray image are associated according to pairing.
It will be understood that the above embodiments are merely exemplary embodiments taken to illustrate the principles of the invention/inventions, which are not limited thereto. It will be apparent to those skilled in the art that various modifications and improvements can be made without departing from the spirit and substance of the invention, and these modifications and improvements are also considered to be within the scope of the invention.
Claims (9)
1. The system for associating the human bag visible light image with the X-ray image is characterized by comprising an association computer (6), a package size detection system, a package X-ray photographing system and a photographing induction system;
the photographing induction system comprises an X-ray machine (1), wherein the X-ray machine (1) comprises an isolation bin (10B) and a conveyor belt assembly (10A) penetrating through the isolation bin (10B), the conveyor belt assembly (10A) comprises a conveying roller and a conveyor belt sleeved on the conveying roller, and the photographing induction system further comprises a vertical delivery position detection device, the vertical delivery position detection device comprises a photosensitive array, the photosensitive array comprises a plurality of delivery position detection photosensitive heads (4) which are arranged in a straight line and at equal intervals, the detection photosensitive heads (4) are arranged on at least one side of the conveying direction of the conveyor belt outside the isolation bin, the arrangement line of the detection photosensitive heads (4) is parallel to the conveying direction of the conveyor belt, the photosensitive surface of the detection photosensitive heads (4) faces to the upper side, and a normally open light source (3) for delivery position detection with a light-emitting surface facing to the lower side is arranged right above the arrangement line of the detection photosensitive heads (4);
a package size detection system for detecting the occupied length L2 of the packages on the conveyor belt,
a photographing induction system used for calculating a delivery position point according to the position point of the light sensing head (4) for detecting the change state of the triggered switching value, detecting a visible light picture P1 of a person bag when the person carries out the package delivery action at the delivery position and the imaging time t thereof k The visible light picture P1 of the person bag contains both the person and the package;
the parcel X-ray photographing system is used for carrying out X-ray scanning on parcels, splicing all frame scanning images corresponding to one parcel into a complete parcel X-ray image P2 after the last frame scanning image of one parcel appears, and taking the imaging time of the last frame scanning image as the imaging time t of the parcel X-ray image P2 x Storing the package X-ray map P2 and the imaging time t thereof x ；
The correlation computer (6) is used for calculating the total travel L of the package complete scanning according to the formula 2, wherein the formula 2 is that L = L2+ L1, and L1 is the distance from the delivery position point to the X-ray scanning position point of the package X-ray photographing system;
a correlation computer (6) for calculating the estimated imaging time t of the last frame of scanning image after the package completely passes through the X-ray scanning position point according to the formula 3 y Equation 3 y =t k + L/V, V is the moving speed of the conveying belt;
a correlation computer (6) for correlating the estimated imaging time t y Closest imaging time t x The corresponding parcel X-ray map P2 is marked as the imaging instant t k And obtaining the association pair of the visible light image P1 of the person bag and the parcel X-ray image P2 by the corresponding association image of the visible light image P1 of the person bag.
2. The system of claim 1, wherein the human bag visible light map and X-ray map are associated,
the conveying belt assembly (10A) further comprises conveying roller side plates (11) positioned at two ends of the conveying roller, and the delivery position detection photosensitive heads are sequentially arranged on the upper surface, facing the right upper side, of the conveying roller side plates (11) along the length direction of the conveying roller side plates (11) positioned outside the separation bin.
3. The system according to claim 1, wherein the human bag visible light map and X-ray map are associated with each other,
the conveyor belt assembly (10A) further comprises transmission roller side plates (11) located at two ends of the transmission roller, the photosensitive arrays are assembled on one substrate, the substrate is mounted on the upper surface of the transmission roller side plate (11) facing to the right upper side in a stacked mode or mounted on the outer side surface of the transmission roller side plate (11) in parallel, and the photosensitive heads for delivery position detection are sequentially arranged along the length direction of the transmission roller side plate (11) located outside the isolation bin.
4. The system of claim 1, wherein the human bag visible light map and X-ray map are associated,
the normally open light source (3) for detecting the delivery position is a linear laser light source or a plurality of light source emitters,
when the normally open light source (3) for detecting the delivery position is a line laser light source, the projection line of the line laser light source covers the arrangement line of the photosensitive head (4) for detecting,
when the normally-open light source (3) for delivery position detection is a plurality of light source emitters corresponding to the light sensing heads (4) one to one, the light rays of each light source emitter vertically project to the light sensing heads (4) for detection right below one to one.
5. The system of claim 1, wherein the human bag visible light map and X-ray map are associated,
the photographing induction system also comprises a visible light camera (2), a standing area and a conveying belt when the visible light camera (2) covers the personnel to carry out parcel delivery, the visible light camera (2) is in a closed state,
induction system shoots, still including controller (5) of shooing, controller (5) of shooing receives the switching value of delivering position detection with photographic head (4), controller (5) of shooing detect arbitrary delivery position detection with photographic head (4) the switching value by switch to close or by closing to the change state of opening after, send out and shoot the instruction and give visible light camera (2), visible light camera (2) carry out once to the regional picture of shooing of visual angle and generate a picture of shooing, controller (5) of shooing obtain this picture of shooing from visible light camera (2), write the picture of shooing into: human package visible light picture P1, storage human package visible light picture P1 and imaging time t thereof k 。
6. The system of claim 1, wherein the human bag visible light map and X-ray map are associated,
the photographing induction system also comprises a visible light camera (2), a standing area and a conveying belt when the visible light camera (2) covers the personnel to carry out parcel delivery, the visible light camera (2) is in a continuous photographing state,
the photographing induction system further comprises a photographing controller (5), the photographing controller (5) stores a photographing video of the visible light camera (2), the photographing controller (5) receives the switching value of the delivery position detection photosensitive head (4), after the photographing controller (5) detects that the switching value of any delivery position detection photosensitive head (4) is switched from on to off or from off to on, a frame image of the moment in the photographing video is searched according to the moment of the switching value change state, and the frame image is recorded as: human package visible light picture P1, storage human package visible light picture P1 and imaging time t thereof k 。
7. The system for correlating a visible light map and an X-ray map of a person's bag according to any one of claims 1 to 6,
parcel radiography system includes: and an X-ray scanning device (13) built in the isolation chamber (10B), wherein the X-ray scanning position point is the position point of the X-ray scanning device (13).
8. The system for correlating a visible light map and an X-ray map of a person's bag according to any one of claims 1 to 6,
when there are a plurality of detection photosensitive heads (4) that trigger the changing state of the switching value, the delivery position point is the midpoint of the position points of the plurality of detection photosensitive heads (4) that trigger the changing state of the switching value or the position point of the detection photosensitive head (4) that is farthest from the isolation chamber (10B).
9. The system for correlating a visible light map and an X-ray map of a person's bag according to any one of claims 1-6, wherein the package size detection system comprises: the X-ray machine comprises an X-ray machine (1), wherein the X-ray machine (1) comprises an isolation bin (10B) and a conveyor belt assembly (10A) penetrating through the isolation bin (10B), the conveyor belt assembly (10A) comprises a conveying roller and a conveyor belt sleeved on the conveying roller, an inlet (14) is formed in the isolation bin (10B), a transverse size detection device is arranged at the inlet, the transverse size detection device is a photoelectric switch (7) for package size detection, the photoelectric switch for package size detection is a correlation type photoelectric switch or a reflection type photoelectric switch, and the normal line of the light emitting surface of the correlation type photoelectric switch and the reflection type photoelectric switch is parallel to the horizontal plane and is perpendicular to the conveying direction of the conveyor belt;
still include size record controller, size record controller record parcel size detects the switching value with photoelectric switch (7), controller of shooing (5) detect parcel size detect with photoelectric switch (7) the switching value by record time T1 when switching to the change state of switch, record time T2 when switching to the change state of switch, calculate the length that the parcel occupies on the transmission band according to formula 1, formula 1: l2= (| time T1-time T2 |) V, L2 is the occupied length of the package on the conveyor belt, and V is the moving speed of the conveyor belt.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211417597.0A CN115469375B (en) | 2022-11-14 | 2022-11-14 | System for associating visible light image and X-ray image of person bag |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211417597.0A CN115469375B (en) | 2022-11-14 | 2022-11-14 | System for associating visible light image and X-ray image of person bag |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115469375A CN115469375A (en) | 2022-12-13 |
| CN115469375B true CN115469375B (en) | 2023-02-14 |
Family
ID=84338157
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211417597.0A Active CN115469375B (en) | 2022-11-14 | 2022-11-14 | System for associating visible light image and X-ray image of person bag |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115469375B (en) |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN205139394U (en) * | 2015-10-27 | 2016-04-06 | 东莞市矩正电子科技有限公司 | Take protection device's security check machine |
| WO2016097724A1 (en) * | 2014-12-18 | 2016-06-23 | Metrasens Limited | Security system and method of detecting contraband items |
| CN206930785U (en) * | 2016-11-28 | 2018-01-26 | 胡斌 | A kind of people wraps corresponding rays safety detection apparatus and safety check transportation system |
| CN109490976A (en) * | 2018-12-28 | 2019-03-19 | 同方威视技术股份有限公司 | Safety inspection control system and method |
| CN110929066A (en) * | 2019-11-18 | 2020-03-27 | 浙江大华技术股份有限公司 | Method and device for displaying personnel and package information in association manner and security check machine |
| CN111323835A (en) * | 2020-03-20 | 2020-06-23 | 安徽启新明智科技有限公司 | Security inspection method and device based on personal bag association |
| CN112798537A (en) * | 2019-11-12 | 2021-05-14 | 通用电气公司 | Method and apparatus for detecting fuel, lube oil degradation and FOB in oil by optical and color characterization |
| CN113267828A (en) * | 2021-06-09 | 2021-08-17 | 杭州海康威视数字技术股份有限公司 | Information association method and device, security check equipment and storage medium |
| CN113759431A (en) * | 2021-06-09 | 2021-12-07 | 同方威视技术股份有限公司 | Security check data association method and device and X-ray security check system |
| US11208270B1 (en) * | 2020-09-16 | 2021-12-28 | Amazon Technologies, Inc. | Systems and methods for scanning and dimensioning items in motion |
| CN114019572A (en) * | 2021-10-11 | 2022-02-08 | 安徽太测临峰光电科技股份有限公司 | X-ray security inspection method and security inspection device based on multi-camera fusion |
| CN216526342U (en) * | 2021-06-07 | 2022-05-13 | 盛视科技股份有限公司 | Article tracking device and system |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6944322B2 (en) * | 2001-03-28 | 2005-09-13 | Visiongate, Inc. | Optical tomography of small objects using parallel ray illumination and post-specimen optical magnification |
| JP2004120009A (en) * | 2002-09-20 | 2004-04-15 | Sumitomo Osaka Cement Co Ltd | Monitoring system and monitoring method |
| CN106548263A (en) * | 2015-09-21 | 2017-03-29 | 同方威视技术股份有限公司 | A kind of intelligent and safe inspection system and method |
| CN114677740A (en) * | 2022-04-01 | 2022-06-28 | 广州广电运通智能科技有限公司 | Person-bag association rapid person searching method, system, storage medium and terminal |
-
2022
- 2022-11-14 CN CN202211417597.0A patent/CN115469375B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016097724A1 (en) * | 2014-12-18 | 2016-06-23 | Metrasens Limited | Security system and method of detecting contraband items |
| CN205139394U (en) * | 2015-10-27 | 2016-04-06 | 东莞市矩正电子科技有限公司 | Take protection device's security check machine |
| CN206930785U (en) * | 2016-11-28 | 2018-01-26 | 胡斌 | A kind of people wraps corresponding rays safety detection apparatus and safety check transportation system |
| CN109490976A (en) * | 2018-12-28 | 2019-03-19 | 同方威视技术股份有限公司 | Safety inspection control system and method |
| CN112798537A (en) * | 2019-11-12 | 2021-05-14 | 通用电气公司 | Method and apparatus for detecting fuel, lube oil degradation and FOB in oil by optical and color characterization |
| CN110929066A (en) * | 2019-11-18 | 2020-03-27 | 浙江大华技术股份有限公司 | Method and device for displaying personnel and package information in association manner and security check machine |
| CN111323835A (en) * | 2020-03-20 | 2020-06-23 | 安徽启新明智科技有限公司 | Security inspection method and device based on personal bag association |
| US11208270B1 (en) * | 2020-09-16 | 2021-12-28 | Amazon Technologies, Inc. | Systems and methods for scanning and dimensioning items in motion |
| CN216526342U (en) * | 2021-06-07 | 2022-05-13 | 盛视科技股份有限公司 | Article tracking device and system |
| CN113267828A (en) * | 2021-06-09 | 2021-08-17 | 杭州海康威视数字技术股份有限公司 | Information association method and device, security check equipment and storage medium |
| CN113759431A (en) * | 2021-06-09 | 2021-12-07 | 同方威视技术股份有限公司 | Security check data association method and device and X-ray security check system |
| CN114019572A (en) * | 2021-10-11 | 2022-02-08 | 安徽太测临峰光电科技股份有限公司 | X-ray security inspection method and security inspection device based on multi-camera fusion |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115469375A (en) | 2022-12-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103502121B (en) | A kind of method for image acquisition | |
| CN102232222B (en) | For receiving the system and method for shipped package | |
| CN104755874B (en) | Motion sensor device with multiple light sources | |
| ES2701024T3 (en) | System and method for product identification | |
| CN112949577B (en) | Information association method, device, server and storage medium | |
| US6201579B1 (en) | Virtual studio position sensing system | |
| KR20070098990A (en) | Image recording device and image recording method | |
| TW201211860A (en) | Optical touch control apparatus and touch sensing method thereof | |
| US6348962B1 (en) | Device for image-shooting both sides of documents | |
| CN115469375B (en) | System for associating visible light image and X-ray image of person bag | |
| CN106056774A (en) | Identification card receiving self-service device based on process protection, and identification card receiving method | |
| KR101613120B1 (en) | Apparatus for reading a bar code | |
| US7122800B2 (en) | Optical density sensor | |
| CN111902843A (en) | Medicine determination device and boundary recess detection method | |
| EP2402914A1 (en) | Detector for the inspection of people | |
| CN106548214A (en) | International flight suspect baggage Third-party Logistics | |
| KR100968205B1 (en) | Apparatus and Method for Space Touch Sensing and Screen Apparatus sensing Infrared Camera | |
| CN102878959B (en) | Method and apparatus for the quality inspection of the leaflet of face shape | |
| JP3312564B2 (en) | Measuring device for long objects | |
| CN107527308A (en) | International flight suspect baggage Third-party Logistics | |
| JP6934915B2 (en) | Image reader | |
| JPH1188870A (en) | Monitoring system | |
| CN110035235A (en) | Monitoring modular, monitoring modular device, monitoring modular device and method | |
| EP2402916B1 (en) | A method for inspecting a person | |
| WO2021101159A1 (en) | Automatic payment apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |