CN113963178A - Method, device, equipment and medium for detecting infrared dim and small target under ground-air background - Google Patents
Method, device, equipment and medium for detecting infrared dim and small target under ground-air background Download PDFInfo
- Publication number
- CN113963178A CN113963178A CN202111345039.3A CN202111345039A CN113963178A CN 113963178 A CN113963178 A CN 113963178A CN 202111345039 A CN202111345039 A CN 202111345039A CN 113963178 A CN113963178 A CN 113963178A
- Authority
- CN
- China
- Prior art keywords
- image
- infrared
- ground
- edge
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 238000001514 detection method Methods 0.000 claims abstract description 162
- 238000004364 calculation method Methods 0.000 claims abstract description 42
- 238000000605 extraction Methods 0.000 claims abstract description 33
- 238000001914 filtration Methods 0.000 claims description 19
- 238000012545 processing Methods 0.000 claims description 19
- 238000004590 computer program Methods 0.000 claims description 13
- 238000012216 screening Methods 0.000 claims description 12
- 238000005192 partition Methods 0.000 claims description 11
- 230000001629 suppression Effects 0.000 claims description 5
- 230000000877 morphologic effect Effects 0.000 claims description 4
- 230000009466 transformation Effects 0.000 claims description 4
- 238000000638 solvent extraction Methods 0.000 claims description 2
- 238000012549 training Methods 0.000 description 23
- 238000013145 classification model Methods 0.000 description 21
- 238000004088 simulation Methods 0.000 description 12
- 238000013527 convolutional neural network Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 8
- 230000008569 process Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 238000012360 testing method Methods 0.000 description 5
- 230000003044 adaptive effect Effects 0.000 description 4
- 238000004422 calculation algorithm Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000011218 segmentation Effects 0.000 description 3
- 238000013461 design Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000007123 defense Effects 0.000 description 1
- 239000006185 dispersion Substances 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/20—Image enhancement or restoration by the use of local operators
-
- G06T5/70—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/136—Segmentation; Edge detection involving thresholding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10048—Infrared image
Abstract
The invention provides a method, a device, equipment and a medium for detecting infrared dim targets under a ground-air background, wherein the method comprises the following steps: acquiring an infrared detection image to be processed; performing target extraction on the infrared detection image to obtain an infrared target image; performing ground-air background estimation on the infrared detection image to obtain a ground-air background image; extracting the edge of the ground-air background image to obtain a ground-air background edge image; performing first difference calculation on the infrared target image and the ground-to-air background edge image to obtain an infrared target image with ground-to-air background edges removed; and carrying out target detection according to the infrared target image with the ground-to-air background edge removed. According to the scheme, the infrared target image with the ground-air background edge removed can be obtained by extracting the ground-air background edge and then performing differential calculation, so that the influence of edge noise on the detection result is eliminated, and the detection probability of the infrared weak and small target detection under the ground-air background is improved.
Description
Technical Field
The embodiment of the invention relates to the technical field of image processing, in particular to a method, a device, equipment and a medium for detecting infrared small and weak targets under the ground-air background.
Background
Currently, anti-unmanned aerial vehicle detection for the ground-air area is an important subject in the fields of military air defense and civil security. The infrared search system has become an indispensable target detection means for security monitoring in key areas due to the characteristics of high-resolution imaging and 360-degree all-directional detection. The ground-air background is the most common background in the anti-unmanned aerial vehicle detection application scene, so the research on the infrared small and weak target detection algorithm under the ground-air background is very necessary.
The traditional infrared weak and small target detection method is a pre-tracking detection algorithm for processing a single frame image, and can be applied to an embedded platform needing real-time response. However, the conventional infrared weak and small target detection method has a high detection probability when the background is single, but has a low detection probability under the ground-air background with complex edge noise.
Disclosure of Invention
Based on the problem that the traditional infrared dim target detection method has low detection probability in the ground-air background with complex edge noise, the embodiment of the invention provides a method, a device, equipment and a medium for detecting infrared dim targets in the ground-air background, which can improve the detection probability of infrared dim target detection in the ground-air background.
In a first aspect, an embodiment of the present invention provides a method for detecting an infrared small and weak target in a ground-air background, including:
acquiring an infrared detection image to be processed;
performing target extraction on the infrared detection image to obtain an infrared target image;
performing ground-air background estimation on the infrared detection image to obtain a ground-air background image;
extracting the edge of the ground-air background image to obtain a ground-air background edge image;
performing first difference calculation on the infrared target image and the ground-to-air background edge image to obtain an infrared target image with ground-to-air background edges removed;
and carrying out target detection according to the infrared target image with the ground-to-air background edge removed.
Preferably, after the performing the first difference calculation on the infrared target image and the ground-to-air background edge image, before obtaining the infrared target image with the ground-to-air background edge removed, the method further includes:
performing edge extraction on the infrared detection image to obtain an infrared edge image; the edge precision in the infrared edge image is greater than that in the ground-to-air background edge image;
and performing second difference calculation on the infrared edge image and the infrared target image subjected to the first difference calculation to obtain the infrared target image with the ground-to-air background edge removed.
Preferably, the edge extraction of the infrared detection image includes:
carrying out Gaussian filtering processing on the infrared detection image;
calculating the gradient amplitude and the gradient direction of each pixel point in the infrared detection image after Gaussian filtering processing;
carrying out non-maximum suppression on the infrared detection image after Gaussian filtering processing according to the gradient amplitude and the gradient direction of each pixel point so as to filter out non-edge pixel points;
and determining an edge line from the infrared detection image with the non-edge pixel points filtered out according to two preset pixel thresholds to obtain the infrared edge image.
Preferably, the determining an edge line from the infrared detection image with non-edge pixel points filtered out according to two preset pixel thresholds includes:
partitioning the pixel points in the infrared detection image with the non-edge pixel points filtered;
for each partition, performing: connecting the pixel points of which the gradient amplitudes are greater than a first pixel threshold value in the partition, determining whether an edge line formed after connection is closed, if not, determining target pixel points of which the gradient amplitudes are greater than a second pixel threshold value in adjacent pixel points of the end points aiming at the end points of the edge line which is not closed, and connecting the target pixel points with the end points until the formed edge line is closed;
the first pixel threshold is greater than the second pixel threshold.
Preferably, after the obtaining the infrared target image with the edge of the ground-to-air background removed, the method further includes:
calculating the pixel value of each pixel point in the infrared target image with the ground-to-air background edge removed;
and screening out pixel points with pixel values meeting preset conditions, and executing target detection by using the infrared target image with the ground-to-air background edge removed after the pixel points are screened out.
Preferably, the screening out the pixel points whose pixel values meet the preset condition includes:
determining the product of the maximum pixel value and a preset proportion as a comparison threshold;
and screening out the pixel points with the pixel values smaller than the comparison threshold value.
Preferably, the first and second liquid crystal materials are,
performing the target extraction of the infrared detection image by using morphological top hat transformation;
and/or the presence of a gas in the gas,
performing ground-air background estimation on the infrared detection image by using a median filter operator with a set scale; the set dimension is the minimum dimension covering the target dimension;
and/or the presence of a gas in the gas,
performing the extracting the edge of the ground-to-air background image using laplacian filtering.
In a second aspect, an embodiment of the present invention further provides a device for detecting an infrared small and weak target in a ground-air background, including:
the image acquisition unit is used for acquiring an infrared detection image to be processed;
the target extraction unit is used for carrying out target extraction on the infrared detection image to obtain an infrared target image;
the background estimation unit is used for carrying out ground-to-air background estimation on the infrared detection image to obtain a ground-to-air background image;
the background edge extraction unit is used for extracting the edge of the ground-air background image to obtain a ground-air background edge image;
the first difference calculation unit is used for performing first difference calculation on the infrared target image and the ground-air background edge image to obtain an infrared target image with ground-air background edges removed;
and the target detection unit is used for carrying out target detection according to the infrared target image with the ground-to-air background edge removed.
In a third aspect, an embodiment of the present invention further provides a computing device, including a memory and a processor, where the memory stores a computer program, and the processor, when executing the computer program, implements the method described in any embodiment of this specification.
In a fourth aspect, the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed in a computer, the computer program causes the computer to execute the method described in any embodiment of the present specification.
The embodiment of the invention provides a method, a device, equipment and a medium for detecting infrared weak and small targets under a ground-air background. Therefore, according to the scheme, the infrared target image with the ground-air background edge removed is obtained by extracting the ground-air background edge and performing differential calculation, so that the influence of edge noise on the detection result is eliminated, and the detection probability of the infrared weak and small target detection under the ground-air background is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
Fig. 1 is a flowchart of a method for detecting infrared small and weak targets in a ground-air background according to an embodiment of the present invention;
fig. 2 is a flowchart of another method for detecting infrared weak and small targets in a ground-air background according to an embodiment of the present invention;
FIG. 3 is a diagram of a hardware architecture of a computing device according to an embodiment of the present invention;
fig. 4 is a structural diagram of an infrared weak and small target detection device under a ground-air background according to an embodiment of the present invention;
FIG. 5 is a block diagram of an infrared weak small target detection apparatus under an earth-air background according to another embodiment of the present invention;
fig. 6 is a structural diagram of another infrared weak and small object detection device in an earth-air background according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer and more complete, the technical solutions in the embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention, and based on the embodiments of the present invention, all other embodiments obtained by a person of ordinary skill in the art without creative efforts belong to the scope of the present invention.
As described above, the conventional infrared weak and small target detection method has a high detection probability when the background is single, but in the ground-air background with complex edge noise, the edge noise greatly affects the detection of the weak and small target, resulting in a low detection probability. Therefore, if the detection of the infrared weak and small target needs to be performed in a complex background, it is first considered to remove the edge noise. The infrared target and the ground-air background edge can be respectively extracted from the infrared detection image, and then the difference calculation is carried out on the extracted infrared target image and the ground-air background edge image, so that the ground-air background edge existing in the infrared target image can be removed, the infrared target image with the ground-air background edge removed is obtained, the purpose of removing the ground-air background edge noise is achieved, and the detection probability of the infrared weak and small target detection under the ground-air background is improved.
Specific implementations of the above concepts are described below.
Referring to fig. 1, an embodiment of the present invention provides a method for detecting a small infrared target in a ground-air background, where the method includes:
and step 100, acquiring an infrared detection image to be processed.
And 102, extracting the target of the infrared detection image to obtain an infrared target image.
And 104, performing ground-to-air background estimation on the infrared detection image to obtain a ground-to-air background image.
And 106, extracting the edge of the ground-to-air background image to obtain a ground-to-air background edge image.
And 108, performing first difference calculation on the infrared target image and the ground-to-air background edge image to obtain an infrared target image with the ground-to-air background edge removed.
And step 110, performing target detection according to the infrared target image with the ground-to-air background edge removed.
In the embodiment of the invention, the infrared target image is obtained by extracting the target of the infrared detection image, the ground-air background estimation is carried out on the infrared detection image, the edge of the ground-air background is extracted, and the ground-air background edge image with the target removed is obtained, so that the first difference calculation can be carried out on the obtained infrared target image and the ground-air background edge image, the infrared target image with the ground-air background edge removed is obtained, and finally, the target detection is carried out according to the infrared target image with the ground-air background edge removed. Therefore, according to the scheme, the infrared target image with the ground-air background edge removed is obtained by extracting the ground-air background edge and performing differential calculation, so that the influence of edge noise on the detection result is eliminated, and the detection probability of the infrared weak and small target detection under the ground-air background is improved.
The manner in which the various steps shown in fig. 1 are performed is described below.
First, in step 100, an infrared detection image to be processed is acquired.
Because all objects can radiate infrared rays in the nature, compared with visible light images, infrared detection images obtained by an infrared detection instrument can be more suitable for the conditions of insufficient light intensity and poor contrast at night, and therefore infrared detection is an indispensable target detection means for security monitoring of key areas.
In the embodiment of the invention, the ground-air background not only has obvious edge noises such as the horizon and a boundary between a building and the sky, but also has tiny edge noises caused by factors such as cloud layer drift, tree shaking, background noise clutter and the like, and in order to realize the detection of the infrared weak and small target under the ground-air background by eliminating the edge noises, a single-frame infrared detection image to be processed needs to be acquired at first.
Then, aiming at step 102, target extraction is performed on the infrared detection image to obtain an infrared target image.
In the embodiment of the invention, the infrared detection image is subjected to target extraction by using morphological top hat transformation, and because the size of an infrared weak and small target in the infrared detection image is generally 2 × 2 pixels, 2 × 2 structural elements are selected to process the infrared detection image to obtain an infrared target image.
Next, in step 104, performing ground-to-air background estimation on the infrared detection image to obtain a ground-to-air background image.
In the embodiment of the invention, ground-to-air background estimation is carried out on the infrared detection image by using a median filter operator with a set scale, wherein the set scale is the minimum scale covering a target scale. The size of the infrared small and weak target is 4 pixels by 4 at most and cannot exceed 5 pixels by 5 at most, so that the ground-air background estimation is carried out on the infrared detection image by selecting the 5 pixels by 5 median filtering, and the ground-air background image with the infrared target filtered is obtained.
Next, in step 106, the edge of the ground-to-air background image is extracted, so as to obtain a ground-to-air background edge image.
In the embodiment of the present invention, since the ground-to-air background image obtained in step 104 has the infrared target filtered out, the edge of the ground-to-air background image obtained in step 104 is directly extracted by using laplacian transform, so that the ground-to-air background edge image can be directly obtained.
Then, for step 108, performing a first difference calculation on the infrared target image and the ground-to-air background edge image to obtain an infrared target image with the ground-to-air background edge removed.
In the embodiment of the present invention, the infrared target image obtained in step 102 and the ground-air background edge image obtained in step 106 are subjected to the first difference calculation, and since the accuracy of the ground-air background edge extracted by the laplacian transform is low, that is, the edge line type is thick, the infrared target image obtained after the first difference calculation still has a fine ground-air background edge, and if the target detection is directly performed, the false alarm rate is easily high, and the detection probability is low.
In an embodiment of the present invention, after the first difference calculation is performed on the infrared target image and the ground-to-air background edge image, before the infrared target image with the ground-to-air background edge removed is obtained, a second difference calculation may be further performed through the following steps S1 to S2:
s1, performing edge extraction on the infrared detection image to obtain an infrared edge image; the edge precision in the infrared edge image is greater than that in the ground-to-air background edge image.
Since the infrared target image obtained after the first difference calculation still has a fine ground-to-air background edge, the remaining ground-to-air background edge needs to be secondarily suppressed. Then, in order to achieve this purpose, it is first necessary to perform edge extraction with higher precision on the infrared detection image, and perform second difference calculation on the infrared target image obtained after the first difference calculation and the edge extraction image with higher precision to remove the fine ground-to-air background edge.
It should be noted that, in the embodiment of the present invention, the infrared edge image is obtained by performing edge extraction on the infrared detection image, and is not obtained by performing edge extraction on the infrared target image obtained after the first difference calculation, because the infrared edge image is obtained by removing the ground-to-air background thin edge in the infrared target image obtained after the first difference calculation through the second difference calculation, the edge extraction is directly performed on the infrared detection image, the obtained ground-to-air background edge information is more complete, and then the ground-to-air background thin edge in the infrared target image can be removed to a greater extent.
In addition, because the edge accuracy in the ground-to-air background edge image obtained by using laplacian transform is low, the extracted ground-to-air background edge is thick in line type, and the infrared target image after the first difference calculation still needs to remove fine edges, the edge extraction method with higher accuracy for the infrared detection image in this step can refine the ground-to-air background edge, that is, the extracted edge accuracy is higher, so that the edge accuracy in the obtained infrared edge image is higher than that in the ground-to-air background edge image.
In the embodiment of the present invention, referring to fig. 2, the edge extraction with higher precision is performed on the infrared detection image as shown in step 200 and step 204:
and 200, performing Gaussian filtering processing on the infrared detection image.
And 204, performing non-maximum suppression on the infrared detection image after Gaussian filtering processing according to the gradient amplitude and the gradient direction of each pixel point to filter out non-edge pixel points.
In the embodiment of the invention, whether each pixel point in the infrared detection image after Gaussian filtering processing is an eight-value neighborhood maximum pixel point in an eight-pixel neighborhood taking the pixel point as a center is judged according to the gradient amplitude of each pixel point, if so, the eight-value neighborhood maximum pixel point is reserved, and if not, the eight-value neighborhood maximum pixel point is determined to be a non-edge pixel point.
Executing the following steps aiming at each eight-value neighborhood maximum pixel: determining intersection points with the eight neighborhood pixel points in the gradient direction according to the gradient direction of the eight-value neighborhood maximum pixel point, and then performing interpolation according to the gradient amplitudes of two pixel points which are closest to each intersection point in the eight neighborhood pixel points to obtain the gradient amplitude of each intersection point; and judging whether the gradient amplitude of the eight-value neighborhood maximum pixel point is larger than that of each intersection point, if so, retaining the eight-value neighborhood maximum pixel point, and if not, determining the eight-value neighborhood maximum pixel point as a non-edge pixel point.
After judging whether all the eight-value neighborhood maximum pixel points are non-edge pixel points, setting the gradient amplitudes of all the non-edge pixel points to be 0, and achieving the purpose of filtering the non-edge pixel points.
And step 206, determining an edge line from the infrared detection image with the non-edge pixel points filtered out according to two preset pixel thresholds, and obtaining the infrared edge image.
In the step, firstly, the pixel points in the infrared detection image with the non-edge pixel points filtered are partitioned; for each partition, performing: connecting the pixel points of which the gradient amplitudes are greater than a first pixel threshold value in the partition, determining whether an edge line formed after connection is closed, if not, determining target pixel points of which the gradient amplitudes are greater than a second pixel threshold value in adjacent pixel points of the end points aiming at the end points of the edge line which is not closed, and connecting the target pixel points with the end points until the formed edge line is closed; the first pixel threshold is greater than the second pixel threshold.
In the embodiment of the invention, the position information of all eight-value neighborhood maximum pixel points reserved in the infrared detection image for filtering the non-edge pixel points is determined, and then all eight-value neighborhood maximum pixel points are partitioned according to the position information of the target and the ground-to-air background in the original infrared detection image and experience.
For each partition, performing: connecting all eight-value neighborhood maximum pixel points with gradient amplitudes larger than a first pixel threshold value in the partition, determining whether edge lines formed after connection are closed, if not, judging whether target pixel points with gradient amplitudes larger than a second pixel threshold value and smaller than the first pixel threshold value exist in eight neighborhood pixel points aiming at the end points of the edge lines which are not closed, if so, connecting the target pixel points with the end points until the formed edge lines are closed, and obtaining the infrared edge image.
And S2, performing second difference calculation on the infrared edge image and the infrared target image after the first difference calculation to obtain the infrared target image with the ground-to-air background edge removed.
In step S2, a second difference calculation is performed on the infrared target image after the first difference calculation and the infrared edge image obtained by performing the edge extraction with higher precision in step S1, so as to obtain an infrared target image with the ground-air background edge removed, thereby achieving the purpose of performing secondary suppression on the remaining ground-air background thin edge.
In addition, after the infrared target image with the ground-to-air background edge removed is obtained, adaptive threshold segmentation needs to be performed on the infrared target image with the ground-to-air background edge removed, and the specific operation includes the following steps:
h1, calculating the pixel value of each pixel point in the infrared target image with the ground-to-air background edge removed.
H2, screening out pixel points with pixel values meeting preset conditions, and executing target detection by using the infrared target image with the ground-to-air background edge removed after the pixel points are screened out.
In step H2, the screening out pixels whose pixel values satisfy the preset condition includes: determining the product of the maximum pixel value and a preset proportion as a comparison threshold; and screening out the pixel points with the pixel values smaller than the comparison threshold value.
For example, if it is necessary to retain pixel points whose pixel values are 40% of the first pixel values, the preset proportion is 60%, if the maximum pixel value of each pixel point of the infrared target image without the edge of the ground-to-air background is 100, the product of the maximum pixel value and the preset proportion is determined as a comparison threshold, then the comparison threshold is calculated to be 60, the pixel point whose pixel value is less than 60 is set to be 0, and the pixel point whose pixel value is 60-100 is retained. And (4) carrying out the following target detection on the infrared target image with the reserved pixel value at 60-100 pixel points and without the ground-to-air background edge.
Finally, in step 110, target detection is performed according to the infrared target image with the edge of the ground-to-air background removed.
In an embodiment of the present invention, in order to perform target detection on an infrared target image with a ground-to-air background edge removed, at least two ways are adopted:
the method comprises the steps of judging a final detection target according to the motion characteristics of each target in the infrared target image with the edge of the ground-air background removed in the time series infrared detection image.
And secondly, classifying each target contained in the infrared target image without the ground-air background edge by using a pre-trained classification model to obtain a final detection target.
The following describes the above two modes, respectively.
First, the first embodiment will be described.
In this manner one, the embodiment of the present invention may specifically include: marking target positions corresponding to all targets in the infrared detection image and the infrared target image without the ground-to-air background edge, and determining all targets in the infrared detection image; and judging which target is the final detection target according to the motion characteristics of the targets in the time series infrared detection images.
For example, if the target is in a stationary state in the time-series infrared detection images, the target is determined as a ground-air background, and if the target is in a relatively gentle position movement in the time-series infrared detection images, which conforms to the motion characteristics of a real target, the target may be determined as a final detection target.
In the second mode, the embodiment of the present invention may specifically include the following steps N1-N3:
n1, training a classification model.
In an embodiment of the present invention, the training method of the classification model may specifically include the following steps M1-M2:
m1, acquiring a plurality of positive sample images and a plurality of negative sample images; the positive sample image is an image containing an infrared target; the negative sample image is an image containing no infrared target; the size of each sample image is the same.
In the embodiment of the invention, the classification model mainly has the function of judging each target and judging whether the target is an infrared target or a background edge, so that the classification model is essentially a two-classification network, and an input image is classified into two categories of the infrared target and the background edge when the classification model is trained in advance. And acquiring a plurality of images containing the infrared target as positive sample images, and acquiring a plurality of images not containing the infrared target as negative sample images.
In addition, because the infrared dim targets have no obvious texture features and contour features, and the number of infrared target pixels is small compared with the number of infrared target images with the ground-to-air background edges removed, when the classification model is used for classification, only the region containing each target needs to be locally judged. In order to increase the detection speed and accuracy and require a size larger than the target, in the embodiment of the present invention, the positive sample image and the negative sample image are set to be 13 × 13 pixels in size.
In the embodiment of the invention, because the diversity of the positive sample images and the negative sample images influences the classification performance of the classification network when the classification network is trained, besides the real target images in the actual infrared images, the images with a plurality of infrared targets as simulation targets are added in the positive sample images, so that the diversity of the training images is increased. The simulation target image in the positive sample image is constructed according to the following formula:
where α is the maximum gray value in the positive sample image, (x)0,y0) Is the position coordinate of the simulation target center, and I (x, y) is the gray value of the pixel at the (x, y) position in the positive sample image, σxAnd σyParameter, σ, for controlling the extent of dispersion of the simulation target in the transverse and longitudinal directionsxAnd σyThe value of (2) is within a set value range and is used for controlling the size of the simulation target in the constructed positive sample image not to be larger than a set size.
For example, when σxAnd σyIf the value of (a) is too small, the value of a single pixel is too high, and the size of the target is generally not more than 5 x 5 due to weak infrared rays, when the value of the (a) is too smallxAnd σyWhen the value of (a) is too large, the size of the simulation target exceeds the reality, the simulation target loses the authenticity, and the two conditions influence the detection result, so that the sigma is used for detecting the simulation targetxAnd σyIs controlled in the value range of [0.5,2 ]]。
In the embodiment of the invention, the training data combining the simulation target image and the real target image is beneficial to the classification network to improve the generalization capability of the classification network, so 3500 simulation target images are constructed according to the above mode, the real target image in 3500 actual infrared images is intercepted, and 7000 images are taken as the positive sample image of the classification network. The negative sample image is a non-target infrared background image randomly intercepted from the actual infrared image, and 7000 negative sample images are also intercepted because the proportion of the positive sample image and the negative sample image is too large, which easily causes the deviation of the classification result of the trained classification model.
In one embodiment of the invention, the centers of the simulation target and the real target in the positive sample image are both located at the center position corresponding to the positive sample image, so that the design of a regression network for the target position can be omitted, the complexity and the calculation cost of a network structure can be greatly reduced, and the detection speed is improved. In addition, the center of the target is located at the center of the positive sample image, so that the classification model can pay more attention to the characteristics of the pixel points in the center area of the sample image in the training process, and the classification model can also classify the image to be classified by utilizing the characteristics of the pixel points in the center area of the image to be classified.
M2, training a convolutional neural network by using the positive sample images and the negative sample images to obtain the classification model; the last convolutional layer of the convolutional neural network is a 1 × 1 convolutional kernel.
In the embodiment of the present invention, a training set and a test set are determined from the positive sample image and the negative sample image determined in step H1, and 10% of images are randomly selected from the positive sample image as the test set, where the test set includes both the real target image and the simulation target image. 1400 positive sample images are extracted as a test set, and the remaining 12600 positive sample images and negative sample images are training sets.
In the embodiment of the invention, compared with the traditional method that the characteristics need to be artificially designed, the characteristics are extracted based on a single-frame image, and finally the target category is judged by artificially adjusting the parameters, the local receptive field and weight sharing mechanism of the convolutional neural network has extremely high achievement in the field of image processing. The convolutional neural network has strong feature extraction capability, and can automatically extract the features of the target image and then classify the target image.
However, since the size of the infrared target is too small, only the area including each target needs to be locally determined, and therefore, the sizes of the positive sample image and the negative sample image are both set to be 13 × 13 pixels, it is not applicable to use a common convolutional neural network structure, the number of network layers in the common structure is large, and the input images are all whole images, which causes a lot of waste of computing resources and time. Therefore, the lightweight convolutional neural network architecture can be redesigned to carry out classification detection on infrared weak and small targets.
In the design of network structure, it can try network with various structures, and train by using the same method, and detect probability P on the test set by losing function convergence condition on the training setdAnd false alarm probability FaThe classification effect of the networks on the infrared weak and small targets is evaluated.
In the embodiment of the invention, the selected convolutional neural network comprises three convolutional layers and two fully-connected layers, the convolution kernels of the former two convolutional layers are 3 x 3, and the convolution kernel of the last convolutional layer is 1 x 1.
Training the constructed convolutional neural network by using the determined training set, processing the positive sample images and the negative sample images in the training set by using normalization processing during training, setting the size of a training batch (batch size) to be 72, disordering the sequence of the positive sample images and the negative sample images during training, and using cross entropy errors as a loss function during training, wherein a calculation formula of the cross entropy errors is as follows:
wherein, ykRepresenting the output of a convolutional neural network, tkRepresenting correct de-tagging. The optimizer selects an Adam algorithm, and by using the optimization algorithm, training can be performed more quickly and effectively, the convergence speed of the network is accelerated, and the training time is shortened. The learning rate is set to be 0.00003, the training is carried out by dividing the learning rate into 7 epochs, the evaluation is carried out once every 50 steps, finally, the loss function is minimum, and the detection probability is highest, and the training is used as a final classification model.
And N2, respectively cutting out local images of the infrared target images with the ground-to-air background edges removed so that each target is respectively positioned in the corresponding local image, aiming at each target contained in the infrared target images with the ground-to-air background edges removed.
In the embodiment of the invention, if a 256 × 256-pixel infrared target image without the edge of the ground-to-air background is directly divided into a plurality of local images for classification and judgment by taking each pixel point as the center, a great deal of computing resources and time are wasted. In addition, since the classification model uses the positive sample image and the negative sample image of the same size in the training process, in order to reduce the influence of images of different sizes on the classification result, when the local image is obtained by segmentation, the size of the segmented local image is the same as the size of the sample image used in training the classification model.
Further, as can be seen from step N1, since the center position of the target is located at the center position of the positive sample image used in the training process of the classification model, it can be understood that the classification model focuses more on the features of the pixel points in the center region of the input image, and therefore, in order to improve the accuracy of the classification result, when the corresponding local image is segmented for each target, the center position of the target is located at the center position of the corresponding local image. In addition, in the infrared target image without the edge of the ground-to-air background, there may be a case where a plurality of targets are close in distance, and therefore, when a corresponding local image is divided for each target, there may be other targets in the edge region of the local image, and by locating the center position of the target at the center position of the corresponding local image, the classification model can be made to pay more attention to the features of the pixel points in the center region of the local image, so that the influence of the target in the edge region on the classification result can be reduced.
When the partial image is segmented, the partial image of the complete target with 13 × 13 pixels can be intercepted with each target as the central region by means of the position information of each target in the infrared target image obtained in the previous step 108, wherein the edges of the ground-to-air background are removed.
And N3, inputting each local image into the classification model respectively, and determining a final detection target according to the output of the classification model.
And respectively inputting each intercepted local image into a trained classification model, and obtaining a final detection target according to an output result of the classification model.
And finally, marking the target position corresponding to the final detection target in the infrared detection image and the infrared target image without the edge of the ground-air background so as to realize the detection of the infrared dim target.
Therefore, the target detection can be completed according to the infrared target image with the ground-to-air background edge removed in the first mode or the second mode, and the detection of the infrared weak and small target is finally realized.
As shown in fig. 3 and 4, an embodiment of the present invention provides an infrared weak and small target detection apparatus in a ground-air background. The device embodiments may be implemented by software, or by hardware, or by a combination of hardware and software. From a hardware aspect, as shown in fig. 3, for a hardware architecture diagram of a computing device in which an infrared weak and small target detection apparatus is located in a ground-to-air background provided in an embodiment of the present invention, in addition to the processor, the memory, the network interface, and the nonvolatile memory shown in fig. 3, the computing device in which the apparatus is located in the embodiment may also generally include other hardware, such as a forwarding chip responsible for processing a packet, and the like. Taking a software implementation as an example, as shown in fig. 4, as a logical apparatus, a CPU of a computing device in which the apparatus is located reads a corresponding computer program in a non-volatile memory into a memory to run. The device for detecting the infrared weak and small target under the ground-air background provided by the embodiment comprises:
an image acquisition unit 401, configured to acquire an infrared detection image to be processed;
a target extraction unit 402, configured to perform target extraction on the infrared detection image to obtain an infrared target image;
a background estimation unit 403, configured to perform ground-to-air background estimation on the infrared detection image to obtain a ground-to-air background image;
a background edge extracting unit 404, configured to extract an edge of the ground-to-air background image to obtain a ground-to-air background edge image;
a first difference calculation unit 405, configured to perform a first difference calculation on the infrared target image and the ground-air background edge image to obtain an infrared target image with a ground-air background edge removed;
and the target detection unit 406 is configured to perform target detection according to the infrared target image with the edge of the ground-to-air background removed.
In an embodiment of the present invention, the first difference calculating unit 405, after performing the first difference calculation on the infrared target image and the ground-air background edge image, further includes a second difference calculating unit 407, as shown in fig. 5, before obtaining the infrared target image without the ground-air background edge, as another structure diagram of the device for detecting weak and small infrared targets in ground-air background according to an embodiment of the present invention.
In an embodiment of the present invention, the second difference calculating unit 407 is specifically configured to perform edge extraction on the infrared detection image to obtain an infrared edge image; the edge precision in the infrared edge image is greater than that in the ground-to-air background edge image; the second difference calculating unit 407 is specifically configured to perform second difference calculation on the infrared edge image and the infrared target image after the first difference calculation to obtain the infrared target image without the ground-to-air background edge.
In an embodiment of the present invention, the second difference calculating unit 407 is specifically configured to perform gaussian filtering processing on the infrared detection image when performing edge extraction on the infrared detection image; calculating the gradient amplitude and the gradient direction of each pixel point in the infrared detection image after Gaussian filtering processing; carrying out non-maximum suppression on the infrared detection image after Gaussian filtering processing according to the gradient amplitude and the gradient direction of each pixel point so as to filter out non-edge pixel points; and determining an edge line from the infrared detection image with the non-edge pixel points filtered out according to two preset pixel thresholds to obtain the infrared edge image.
In an embodiment of the present invention, the second difference calculating unit 407 is further configured to partition the pixel points in the infrared detection image with non-edge pixel points filtered out when determining the edge line from the infrared detection image with non-edge pixel points filtered out according to the two preset pixel thresholds; for each partition, performing: connecting the pixel points of which the gradient amplitudes are greater than a first pixel threshold value in the partition, determining whether an edge line formed after connection is closed, if not, determining target pixel points of which the gradient amplitudes are greater than a second pixel threshold value in adjacent pixel points of the end points aiming at the end points of the edge line which is not closed, and connecting the target pixel points with the end points until the formed edge line is closed; the first pixel threshold is greater than the second pixel threshold.
In an embodiment of the present invention, after the first difference calculating unit 405 performs the operation of obtaining the infrared target image with the edge of the ground-to-air background removed, as shown in fig. 6, which is a structural diagram of another infrared weak target detecting apparatus under the ground-to-air background according to an embodiment of the present invention, the first difference calculating unit further includes an adaptive threshold dividing unit 408.
In an embodiment of the present invention, the adaptive threshold segmentation unit 408 is specifically configured to calculate a pixel value of each pixel point in the infrared target image without the edge of the ground-to-air background; and screening out pixel points with pixel values meeting preset conditions, and executing target detection by using the infrared target image with the ground-to-air background edge removed after the pixel points are screened out.
In an embodiment of the present invention, the adaptive threshold dividing unit 408 is specifically configured to determine a product of a maximum pixel value and a preset ratio as a comparison threshold when performing screening out pixels whose pixel values satisfy a preset condition; and screening out the pixel points with the pixel values smaller than the comparison threshold value.
In an embodiment of the present invention, the target extracting unit 402 is specifically configured to perform the target extraction on the infrared detection image by using a morphological top hat transformation; and/or the background estimation unit 403 is specifically configured to perform ground-to-air background estimation on the infrared detection image by using a median filter operator with a set scale; the set dimension is the minimum dimension covering the target dimension; and/or, the background edge extraction unit 404 is specifically configured to perform the extracting the edge of the ground-space background image by using laplacian filtering.
It is understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation to an infrared weak and small target detection device in a ground-air background. In other embodiments of the present invention, an infrared weak target detection apparatus in a ground-to-air background may include more or fewer components than those shown, or some components may be combined, some components may be separated, or a different arrangement of components may be used. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
Because the content of information interaction, execution process, and the like among the modules in the device is based on the same concept as the method embodiment of the present invention, specific content can be referred to the description in the method embodiment of the present invention, and is not described herein again.
The embodiment of the invention also provides computing equipment which comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method for detecting the infrared dim target in the ground-air background is realized.
The embodiment of the invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the processor is caused to execute a method for detecting an infrared weak and small target in an earth-space background in any embodiment of the invention.
Specifically, a system or an apparatus equipped with a storage medium on which software program codes that realize the functions of any of the above-described embodiments are stored may be provided, and a computer (or a CPU or MPU) of the system or the apparatus is caused to read out and execute the program codes stored in the storage medium.
In this case, the program code itself read from the storage medium can realize the functions of any of the above-described embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.
Examples of the storage medium for supplying the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (e.g., CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD + RW), a magnetic tape, a nonvolatile memory card, and a ROM. Alternatively, the program code may be downloaded from a server computer via a communications network.
Further, it should be clear that the functions of any one of the above-described embodiments may be implemented not only by executing the program code read out by the computer, but also by causing an operating system or the like operating on the computer to perform a part or all of the actual operations based on instructions of the program code.
Further, it is to be understood that the program code read out from the storage medium is written to a memory provided in an expansion board inserted into the computer or to a memory provided in an expansion module connected to the computer, and then causes a CPU or the like mounted on the expansion board or the expansion module to perform part or all of the actual operations based on instructions of the program code, thereby realizing the functions of any of the above-described embodiments.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an …" does not exclude the presence of other similar elements in a process, method, article, or apparatus that comprises the element.
Those of ordinary skill in the art will understand that: all or part of the steps for realizing the method embodiments can be completed by hardware related to program instructions, the program can be stored in a computer readable storage medium, and the program executes the steps comprising the method embodiments when executed; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A method for detecting infrared small and weak targets under the ground-air background is characterized by comprising the following steps:
acquiring an infrared detection image to be processed;
performing target extraction on the infrared detection image to obtain an infrared target image;
performing ground-air background estimation on the infrared detection image to obtain a ground-air background image;
extracting the edge of the ground-air background image to obtain a ground-air background edge image;
performing first difference calculation on the infrared target image and the ground-to-air background edge image to obtain an infrared target image with ground-to-air background edges removed;
and carrying out target detection according to the infrared target image with the ground-to-air background edge removed.
2. The method according to claim 1, before obtaining the infrared target image with the ground-to-air background edge removed after the first difference calculation of the infrared target image and the ground-to-air background edge image, further comprising:
performing edge extraction on the infrared detection image to obtain an infrared edge image; the edge precision in the infrared edge image is greater than that in the ground-to-air background edge image;
and performing second difference calculation on the infrared edge image and the infrared target image subjected to the first difference calculation to obtain the infrared target image with the ground-to-air background edge removed.
3. The method of claim 2, wherein performing edge extraction on the infrared detection image comprises:
carrying out Gaussian filtering processing on the infrared detection image;
calculating the gradient amplitude and the gradient direction of each pixel point in the infrared detection image after Gaussian filtering processing;
carrying out non-maximum suppression on the infrared detection image after Gaussian filtering processing according to the gradient amplitude and the gradient direction of each pixel point so as to filter out non-edge pixel points;
and determining an edge line from the infrared detection image with the non-edge pixel points filtered out according to two preset pixel thresholds to obtain the infrared edge image.
4. The method according to claim 3, wherein the determining an edge line from the infrared detection image with non-edge pixel points filtered out according to two preset pixel thresholds comprises:
partitioning the pixel points in the infrared detection image with the non-edge pixel points filtered;
for each partition, performing: connecting the pixel points of which the gradient amplitudes are greater than a first pixel threshold value in the partition, determining whether an edge line formed after connection is closed, if not, determining target pixel points of which the gradient amplitudes are greater than a second pixel threshold value in adjacent pixel points of the end points aiming at the end points of the edge line which is not closed, and connecting the target pixel points with the end points until the formed edge line is closed;
the first pixel threshold is greater than the second pixel threshold.
5. The method according to claim 1, further comprising, after the obtaining the infrared target image with the ground-to-air background edge removed, the steps of:
calculating the pixel value of each pixel point in the infrared target image with the ground-to-air background edge removed;
and screening out pixel points with pixel values meeting preset conditions, and executing target detection by using the infrared target image with the ground-to-air background edge removed after the pixel points are screened out.
6. The method according to claim 5, wherein the screening out the pixels whose pixel values satisfy the predetermined condition comprises:
determining the product of the maximum pixel value and a preset proportion as a comparison threshold;
and screening out the pixel points with the pixel values smaller than the comparison threshold value.
7. The method according to any one of claims 1 to 6,
performing the target extraction of the infrared detection image by using morphological top hat transformation;
and/or the presence of a gas in the gas,
performing ground-air background estimation on the infrared detection image by using a median filter operator with a set scale; the set dimension is the minimum dimension covering the target dimension;
and/or the presence of a gas in the gas,
performing the extracting the edge of the ground-to-air background image using laplacian filtering.
8. The utility model provides a little target detection device of infrared under ground empty background which characterized in that includes:
the image acquisition unit is used for acquiring an infrared detection image to be processed;
the target extraction unit is used for carrying out target extraction on the infrared detection image to obtain an infrared target image;
the background estimation unit is used for carrying out ground-to-air background estimation on the infrared detection image to obtain a ground-to-air background image;
the background edge extraction unit is used for extracting the edge of the ground-air background image to obtain a ground-air background edge image;
the first difference calculation unit is used for performing first difference calculation on the infrared target image and the ground-air background edge image to obtain an infrared target image with ground-air background edges removed;
and the target detection unit is used for carrying out target detection according to the infrared target image with the ground-to-air background edge removed.
9. A computing device comprising a memory having stored therein a computer program and a processor that, when executing the computer program, implements the method of any of claims 1-7.
10. A computer-readable storage medium, on which a computer program is stored which, when executed in a computer, causes the computer to carry out the method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111345039.3A CN113963178A (en) | 2021-11-15 | 2021-11-15 | Method, device, equipment and medium for detecting infrared dim and small target under ground-air background |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111345039.3A CN113963178A (en) | 2021-11-15 | 2021-11-15 | Method, device, equipment and medium for detecting infrared dim and small target under ground-air background |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113963178A true CN113963178A (en) | 2022-01-21 |
Family
ID=79470398
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111345039.3A Pending CN113963178A (en) | 2021-11-15 | 2021-11-15 | Method, device, equipment and medium for detecting infrared dim and small target under ground-air background |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113963178A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114648547A (en) * | 2022-03-09 | 2022-06-21 | 中国空气动力研究与发展中心计算空气动力研究所 | Weak and small target detection method and device for anti-unmanned aerial vehicle infrared detection system |
-
2021
- 2021-11-15 CN CN202111345039.3A patent/CN113963178A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114648547A (en) * | 2022-03-09 | 2022-06-21 | 中国空气动力研究与发展中心计算空气动力研究所 | Weak and small target detection method and device for anti-unmanned aerial vehicle infrared detection system |
| CN114648547B (en) * | 2022-03-09 | 2023-06-27 | 中国空气动力研究与发展中心计算空气动力研究所 | Weak and small target detection method and device for anti-unmanned aerial vehicle infrared detection system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110472627B (en) | End-to-end SAR image recognition method, device and storage medium | |
| CN109614985B (en) | Target detection method based on densely connected feature pyramid network | |
| Xie et al. | Multilevel cloud detection in remote sensing images based on deep learning | |
| Hu et al. | Fast forest fire smoke detection using MVMNet | |
| Bhanu | Automatic target recognition: State of the art survey | |
| CN110084234B (en) | Sonar image target identification method based on example segmentation | |
| US9189867B2 (en) | Adaptive image processing apparatus and method based in image pyramid | |
| CN113567984A (en) | Method and system for detecting artificial small target in SAR image | |
| CN109902715B (en) | Infrared dim target detection method based on context aggregation network | |
| CN110309781B (en) | House damage remote sensing identification method based on multi-scale spectrum texture self-adaptive fusion | |
| AU2021208647A1 (en) | Systems for multiclass object detection and alerting and methods therefor | |
| CN111046880A (en) | Infrared target image segmentation method and system, electronic device and storage medium | |
| CN107273832B (en) | License plate recognition method and system based on integral channel characteristics and convolutional neural network | |
| CN109934216B (en) | Image processing method, device and computer readable storage medium | |
| CN112734775A (en) | Image annotation, image semantic segmentation and model training method and device | |
| CN110008900B (en) | Method for extracting candidate target from visible light remote sensing image from region to target | |
| Doi et al. | The effect of focal loss in semantic segmentation of high resolution aerial image | |
| CN116469020A (en) | Unmanned aerial vehicle image target detection method based on multiscale and Gaussian Wasserstein distance | |
| CN115115601A (en) | Remote sensing ship target detection method based on deformation attention pyramid | |
| CN113963178A (en) | Method, device, equipment and medium for detecting infrared dim and small target under ground-air background | |
| CN111368865A (en) | Method and device for detecting remote sensing image oil storage tank, readable storage medium and equipment | |
| Kusetogullari et al. | Unsupervised change detection in landsat images with atmospheric artifacts: a fuzzy multiobjective approach | |
| CN110334703B (en) | Ship detection and identification method in day and night image | |
| CN117218545A (en) | LBP feature and improved Yolov 5-based radar image detection method | |
| CN112132207A (en) | Target detection neural network construction method based on multi-branch feature mapping |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |