JP6113406B2 - Infrared target detection device - Google Patents

Description

  The present invention relates to an infrared target detection apparatus that is mounted on a ship or an aircraft and detects a small target such as an aircraft or a flying object based on an infrared image captured in all directions of 360 degrees.

  Conventionally, as an infrared target detection device that captures an infrared image in all directions of 360 degrees and detects a target based on the captured infrared image, for example, an infrared imaging device mainly composed of a line sensor, An apparatus that optically scans and images omnidirectional images by cooperation with a rotation prism and a scanner is known (see, for example, Patent Document 1 (FIG. 1)). In addition, an infrared imaging device composed of an area sensor is pivotally supported by a gimbal mechanism and mechanically scanned to capture an omnidirectional image (for example, Patent Document 2 (FIG. 1). )reference).

  The infrared target detection apparatus described in Patent Document 1 includes an incident optical unit capable of rotating 360 degrees, a derotation prism that corrects image rotation, and infrared rays that are incident through the incident optical unit and the derotation prism. A scanner for scanning, and an infrared imaging device for detecting infrared rays incident through the scanner. The infrared target detection device rotates the incident optical unit within a range of 360 degrees, scans the infrared image with the scanner while correcting the rotation corresponding to the rotation angle with the derotation prism, and detects the infrared rays condensed on the lens. Detect with an infrared imager. The infrared target detection apparatus repeats these series of operations to capture an omnidirectional infrared image, and detects a target from the infrared image.

  The infrared target detection apparatus described in Patent Document 2 supports an infrared imager in advance on a gimbal mechanism that can turn in the azimuth and elevation directions, and scans the gimbal mechanism in a range of 0 to 360 degrees in the azimuth direction. Infrared light collected on the lens is detected by an infrared imager. The infrared target detection apparatus repeats these series of operations to capture an omnidirectional infrared image, and detects a target from the infrared image.

JP 2000-346923 A JP-A-9-27954

  According to the prior art, the infrared target detection device is equipped with an optical mechanism including a derotation prism and a scanner, or a mechanical mechanism such as a gimbal mechanism in order to capture an omnidirectional infrared image. The mechanical parts constituting such an optical mechanism or mechanical mechanism are often large and expensive. For this reason, according to the prior art, there is a problem that the infrared target detection device is increased in size and cost.

  Since the infrared target detection device captures an omnidirectional infrared image by scanning using an optical mechanism or a mechanical mechanism, the frame update time of the omnidirectional infrared image depends on the scanning time of these mechanisms. Become. Usually, the scanning time in all directions by an optical mechanism or a mechanical mechanism takes about several seconds. The infrared target detection device waits for the scanning time to elapse to perform target detection, so that it is difficult to update target detection in real time.

  The present invention has been made in view of the above, and an object of the present invention is to obtain an infrared target detection apparatus that enables target detection in real time, and that can achieve downsizing and cost reduction.

  In order to solve the above-described problems and achieve the object, the present invention is arranged to be able to image all directions by combining the fields of view of each other, and detects a collected infrared ray to pick up an infrared image. An image pickup device, a space stabilization processing unit for performing a space stabilization process for correcting movement of the infrared image for each frame due to displacement of the infrared image pickup device, and the space stabilization in the space stabilization processing unit A binarization process of a luminance value for each pixel is performed on the infrared image that has undergone the binarization process, and a target detection processing unit that detects a target from the infrared image, and the binarization process in the target detection processing unit And a feature amount calculation unit that calculates the target feature amount based on the binary image acquired by the above.

  According to the present invention, the infrared target detection device employs a plurality of infrared imagers arranged so as to be able to image in all directions in place of optical mechanisms and mechanical mechanisms that lead to an increase in size and cost of the device. To do. The infrared imager can output an infrared image at high speed by using an image transmission standard such as GigE-vision that enables data transfer at a maximum of 1 Gbps, for example. For example, the infrared imager outputs an infrared image at the same cycle as the frame rate. Since the infrared target detection device can pick up an omnidirectional infrared image regardless of scanning by an optical mechanism or a mechanical mechanism, the target detection result can be updated for each frame rate regardless of the scanning time. . Thereby, the infrared target detection apparatus can perform target detection in real time, and can realize downsizing and cost reduction.

FIG. 1 is a block diagram showing a schematic configuration of the infrared target detection apparatus according to the first embodiment of the present invention. FIG. 2 is a diagram illustrating an installation example of a plurality of infrared imaging devices in the horizontal direction. FIG. 3 is a diagram for explaining the principle of the space stabilization processing by the space stabilization processing unit. FIG. 4 is a diagram for explaining processing timings of the infrared imaging device, the space stabilization processing unit, and the target detection processing unit under the control of the infrared imaging device control unit. FIG. 5 is a block diagram illustrating a configuration of the space stabilization processing unit. FIG. 6 is a conceptual diagram illustrating bilinear interpolation. FIG. 7 is a conceptual diagram illustrating an alignment process using an image pyramid. FIG. 8 is a block diagram showing a schematic configuration of the infrared target detection apparatus according to the second embodiment of the present invention. FIG. 9 is a diagram for explaining generation of a panoramic image in the panoramic image synthesis unit. FIG. 10 is a block diagram showing a schematic configuration of the infrared target detection apparatus according to the third embodiment of the present invention. FIG. 11 is a diagram illustrating an example of a panoramic image on which symbols are superimposed. FIG. 12 is a diagram for explaining the coordinate system of the panoramic image and the coordinate system of the target center-of-gravity position output from the feature amount calculation circuit.

  Embodiments of an infrared target detection apparatus according to the present invention will be described below in detail with reference to the drawings. Note that the present invention is not limited to the embodiments.

Embodiment 1 FIG.
FIG. 1 is a block diagram showing a schematic configuration of the infrared target detection apparatus according to the first embodiment of the present invention. The infrared target detection apparatus 1 is mounted on a ship or an aircraft and detects small targets such as aircraft and flying objects based on infrared images captured in all directions of 360 degrees.

  The infrared target detection apparatus 1 includes four infrared imaging devices 2-1, 2-2, 2-3, and 2-4, a space stabilization processing unit 3, a target detection processing unit 4, a feature amount calculation circuit (a feature amount calculation unit). ) 5 and an infrared imager controller 6.

  The infrared imaging devices 2-1 to 2-4 collect the infrared rays 10 from the target and detect the collected infrared rays 10. The infrared imaging devices 2-1 to 2-4 convert the infrared rays 10 into signal charges and take infrared images. The infrared imaging devices 2-1, 2-2, 2-3, and 2-4 output infrared images 11-1, 11-2, 11-3, and 11-4, respectively.

  The infrared imaging device control unit 6 outputs imaging commands 14-1, 14-2, 14-3, and 14-4 to the infrared imaging devices 2-1, 2-2, 2-3, and 2-4, respectively. . The infrared imaging device control unit 6 controls the infrared imaging devices 2-1 to 2-4 according to imaging commands 14-1 to 14-4. The infrared imaging devices 2-1 to 2-4 capture the infrared images 11-1 to 11-4 according to the imaging commands 14-1 to 14-4. The infrared imager control unit 6 outputs a frame timing signal 15 to the space stabilization processing unit 3, the target detection processing unit 4, and the feature amount calculation circuit 5.

  The positions of the infrared imaging devices 2-1 to 2-4 are changed by the shaking of a ship or an aircraft on which the infrared target detection device 1 is mounted. The space stabilization processing unit 3 controls the infrared images 11-1 to 11-11 from the infrared imaging devices 2-1 to 2-4 in order to suppress blurring of the infrared images 11-1 to 11-4 due to shaking of a ship or an aircraft. -4 is subjected to a space stabilization process.

  The infrared image pickup devices 2-1 to 2-4 are continuously displaced in response to vibrations or swings of a ship or an aircraft on which the infrared target detection device 1 is mounted. The space stabilization process is a process for correcting the movement of the infrared images 11-1 to 11-4 for each frame due to the displacement of the infrared imaging devices 2-1 to 2-4. The space stabilization processing unit 3 outputs infrared images 12-1, 12-2, 12-3, and 12-4 that have undergone space stabilization processing.

  The target detection processing unit 4 detects a target from the infrared images 12-1 to 12-4. The target detection processing unit 4 measures the luminance value of the pixel of interest and the luminance distribution around the pixel of interest for the infrared images 12-1 to 12-4. The target detection processing unit 4 obtains a binarization threshold value for each target pixel using the measurement result of the luminance distribution. The target detection processing unit 4 performs a binarization process of the luminance value for each pixel and detects a target from the infrared images 12-1 to 12-4. The target detection processing unit 4 outputs binary images 13-1, 13-2, 13-3, and 13-4 that have undergone binarization processing.

  The feature amount calculation circuit 5 calculates a target feature amount based on the binary images 13-1 to 13-4 from the target detection processing unit 4. The feature amount calculation circuit 5 labels the targets in the binary images 13-1 to 13-4, and calculates, for example, feature amounts such as a center of gravity position and an area for the targets. The feature amount calculation circuit 5 outputs the calculation result as the target feature amount 16. The infrared target detection apparatus 1 outputs a target feature value 16.

  The space stabilization processing unit 3, the target detection processing unit 4, the feature amount calculation circuit 5, and the infrared imager control unit 6 that constitute the infrared target detection device 1 correspond to real-time calculation processing. A circuit element such as an FPGA (Field Programmable Gate Array) capable of synchronous operation at an operating frequency is provided.

  FIG. 2 is a diagram illustrating an installation example of a plurality of infrared imaging devices in the horizontal direction. The infrared imaging devices 2-1, 2-2, 2-3, and 2-4 have an imaging visual field range 17-1, 17-2, 17-3 of 90 degrees or more in the azimuth direction (horizontal direction), for example, about 120 degrees. And 17-4 can be secured. The infrared imaging devices 2-1 to 2-4 are arranged with the directions of the optical axes AX different from each other by 90 degrees. Thereby, the infrared imaging devices 2-1 to 2-4 are arranged so as to be able to capture images in all azimuth directions (0 degrees to 360 degrees) by combining the imaging field-of-view ranges 17-1 to 17-4. The infrared target detection apparatus 1 suppresses leakage of target detection by overlapping the ends of adjacent imaging visual field ranges 17-1 to 17-4.

  The infrared target detection apparatus 1 applies the four infrared imaging devices 2-1 to 2-4 that enable imaging in the imaging visual field range 17-1 to 17-4 of 90 degrees or more, thereby changing the omnidirectional direction. Take an image. Note that the number of infrared imagers is not limited to four, and can be changed as appropriate as long as the omnidirectional directions can be imaged by combining the fields of view. Moreover, the angle of the imaging visual field range of each infrared image sensor is only required to be able to image the omnidirectional direction by combining the visual field of each infrared image sensor, and can be appropriately determined according to the number of infrared image sensors and the like. To do.

  Next, the operation of the infrared target detection apparatus 1 will be described. The infrared imaging device controller 6 generates a synchronization signal synchronized with the frame rate of the infrared images 11-1 to 11-4 output from the infrared imaging devices 2-1 to 2-4. The infrared imaging device controller 6 gives imaging commands 14-1 to 14-4 synchronized with the frame rate to the infrared imaging devices 2-1 to 2-4. Infrared imaging devices 2-1 to 2-4 capture infrared images 11-1 to 11-4 in synchronization with the frame rate in response to imaging commands 14-1 to 14-4.

  The infrared imager control unit 6 supplies a frame timing signal 15 synchronized with the frame rate to the space stabilization processing unit 3, the target detection processing unit 4, and the feature amount calculation circuit 5. The space stabilization processing unit 3, the target detection processing unit 4, and the feature amount calculation circuit 5 operate in synchronization with the frame rate.

  The infrared image pickup devices 2-1 to 2-4 receive the infrared images 11-1 to 11-4 picked up in response to the image pickup commands 14-1 to 14-1 from the infrared image pickup device control unit 6 within the frame period. As described above, the data is transferred to the space stabilization processing unit 3.

For example, the frame rate of the infrared images 11-1 to 11-4 is 30 Hz (≈33.3 ms), the image size of the infrared images 11-1 to 11-4 is VGA (640 × 480 pixels), and the image data length is 16 bits. Suppose that the infrared data 11-1 to 11-4 is serially transferred from the infrared imaging devices 2-1 to 2-4 to the space stabilization processing unit 3 at a data transfer rate of 1 Gbps. In this case, the data transfer time per frame is 640 × 480 (pixels) × 16 (bits) / 10 ⁹ (bps) ≈5 ms. The infrared target detection apparatus 1 can transfer the infrared images 11-1 to 11-4 so as to be sufficiently within the frame period even in consideration of the overhead in data transfer.

  The space stabilization processing unit 3 performs a space stabilization process for suppressing blurring of the infrared images 11-1 to 11-4 due to vibrations and swings of ships and aircraft in which the infrared target detection device 1 is mounted. The space stabilization processing unit 3 obtains the amount of motion of the infrared images 11-1 to 11-4 for each frame in synchronization with the frame rate in accordance with the frame timing signal 15 from the infrared imager control unit 6.

  The space stabilization processing unit 3 extracts the amount of motion of the infrared images 11-1 to 11-4 using, for example, an iterative gradient method. The space stabilization processing unit 3 calculates a motion correction amount according to the extracted motion amount. The space stabilization processing unit 3 outputs infrared images 12-1 to 12-4 that have been subjected to motion correction based on the calculated motion correction amount.

  FIG. 3 is a diagram for explaining the principle of the space stabilization processing by the space stabilization processing unit. For example, the infrared imaging device 2-1 converts the frames I (t), I (t + 1), and I (t + 2) as the infrared image 11-1 at times t, t + 1, t + 2,. .., I (t + n) is captured. Each of the frames I (t) to I (t + n) has a change in position and inclination in the three-dimensional direction according to the shaking of the ship or aircraft on which the infrared target detection device 1 is mounted.

  The space stabilization processing unit 3 uses a predetermined frame, for example, a frame I (t) at time t as a reference, for each frame I (t + 1) to (t + n) from time (t + 1) to (t + n) in the three-dimensional direction. The amount of change in position and inclination is extracted as the amount of movement. The space stabilization processing unit 3 calculates a motion correction amount according to the extracted motion amount. The space stabilization processing unit 3 performs motion correction based on the motion correction amount for each frame I (t + 1) to (t + n) from the time (t + 1) to (t + n), and the infrared image 12− after the space stabilization processing is performed. Set to 1.

  The space stabilization processing unit 3 similarly performs motion correction on the infrared images 11-2, 11-3, and 11-4 captured by the other infrared imagers 2-2, 2-3, and 2-4. The infrared images 12-2, 12-3, and 12-4 after the space stabilization process are used. The detailed configuration of the space stabilization processing unit 3 will be described later.

The target detection processing unit 4 uses the following expression (1) for the infrared images 12-1 to 12-4 from the space stabilization processing unit 3, for example, for the luminance value for each pixel. Ask for. In calculating the binarization threshold for the target pixel, the target detection processing unit 4 refers to the luminance values of the peripheral pixels located around the target pixel.
TH (i, j) = MEAN (i, j) + K * VAR (I, j) (1)

  Here, (i, j) is the coordinate of the pixel of interest, TH (i, j) is the binarization threshold for the pixel of interest, MEAN (i, j) is the average value of the luminance values of the surrounding pixels, and VAR (i, j j) is a standard deviation value of luminance values of peripheral pixels, and K is a predetermined constant.

  The target detection processing unit 4 obtains binary image data using the following equation (2), for example.

  Here, (i, j) is the coordinate of the pixel of interest, TH (i, j) is the binarization threshold for the pixel of interest, D (i, j) is the luminance value of the pixel of interest, and B (i, j) is Binary image data at the target pixel.

  The target detection processing unit 4 obtains binary image data in which “1” is set for pixels whose luminance value is equal to or greater than the binarization threshold and “0” is set for pixels whose luminance value is less than the binarization threshold. , And output as binary images 13-1 to 13-4. The target detection processing unit 4 performs binarization processing in synchronization with the frame rate in accordance with the frame timing signal 15 from the infrared imaging device control unit 6. The target detection processing unit 4 outputs binary images 13-1 to 13-4 for each frame.

  The feature amount calculation circuit 5 performs a target labeling process by a known method based on the binary images 13-1 to 13-4 from the target detection processing unit 4. The feature quantity computing circuit 5 computes feature quantities such as the coordinates of the center of gravity of the target, the coordinates of the area occupied by the target, and the area. The feature amount calculation circuit 5 obtains a feature amount in synchronization with the frame rate in accordance with the frame timing signal 15 from the infrared imager control unit 6 and outputs it as the target feature amount 16.

  FIG. 4 is a diagram for explaining processing timings of the infrared imaging device, the space stabilization processing unit, and the target detection processing unit under the control of the infrared imaging device control unit. The infrared imaging devices 2-1 to 2-4 capture the infrared images 11-1 to 11-4 in synchronization with the frame rate in response to the imaging commands 14-1 to 14-1. The infrared image pickup devices 2-1 to 2-4 receive frames I (t), I (t + 1), and I (t + 2) as infrared images 11-1 to 11-4 at times t, t + 1, t + 2, t + 3. ), I (t + 3)... The infrared image pickup devices 2-1 to 2-4 perform the respective frames I (t), I (t + 1), I (t + 2), I (t + 3)... In a period 5 ms shorter than the frame period 33.3 ms, for example. Output.

  The space stabilization processing unit 3 performs space stabilization processing of the infrared images 11-1 to 11-4 in synchronization with the frame rate in accordance with the frame timing signal 15. For example, the space stabilization processing unit 3 performs the space stabilization processing for the frame I (t) in accordance with the frame timing signal 15 at time t + 1. The space stabilization processing unit 3 performs space stabilization processing for the frames I (t), I (t + 1), I (t + 2)... At times t + 1, t + 2, t + 3. The space stabilization processing unit 3 displays each frame I (t), I (t + 1), I (t + 2), I (t + 3)... As the infrared images 12-1 to 12-4 in a period shorter than the frame period. Output.

  The target detection processing unit 4 performs binarization processing of the infrared images 12-1 to 12-4 in synchronization with the frame rate in accordance with the frame timing signal 15. For example, the target detection processing unit 4 performs binarization processing for the frame I (t) according to the frame timing signal 15 at time t + 2. The target detection processing unit 4 performs binarization processing for the frames I (t), I (t + 1)... At times t + 2, t + 3. The target detection processing unit 4 displays the binary images 13-1 to 13-4 of each frame I (t), I (t + 1), I (t + 2), I (t + 3)... In a period shorter than the frame period. Output.

  The infrared target detection apparatus 1 performs the processing in each unit in synchronization with the frame rate, and completes each processing within the frame period. For this reason, the infrared target detection apparatus 1 can update the target detection result in synchronization with the frame rate. With the above operation, the infrared target detection apparatus 1 can perform target detection without performing scanning by an optical mechanism or a mechanical mechanism, and can perform target detection in real time.

  The infrared target detection device 1 is a plurality of infrared imaging devices 2-1 to 2-4 arranged so as to be capable of imaging in all directions, instead of an optical mechanism and a mechanical mechanism that lead to an increase in size and cost of the device. Apply. For this reason, the infrared target detection apparatus 1 can realize size reduction and cost reduction. The infrared target detection apparatus 1 can improve the efficiency of the production process by downsizing. The infrared target detection apparatus 1 eliminates the need for operations such as high-speed rotation and movement for scanning, thereby improving durability and reducing energy consumption compared to the case where an optical mechanism or a mechanical mechanism is used. Can be planned.

  Next, details of the configuration of the space stabilization processing unit 3 will be described. FIG. 5 is a block diagram illustrating a configuration of the space stabilization processing unit. The space stabilization processing unit 3 includes image storage memories 21-1, 21-2, 21-3, and 21-4, coarse alignment processing units 22-1, 22-2, 22-3, and 22-4, Interpolation processing units 23-1, 23-2, 23-3, and 23-4 are included.

  The image storage memories 21-1, 21-2, 21-3, and 21-4 are infrared images 11-1, 11 that are input from the infrared imaging devices 2-1, 2-2, 2-3, and 2-4. -2, 11-3 and 11-4 are respectively stored. The image storage memories 21-1 to 21-4 are the frames corresponding to the latest frame timing signal 15 from the infrared imaging device controller 6 and the previous frame timing signal 15 for the infrared images 11-1 to 11-4. A double buffer system is adopted that is capable of storing two frames: a frame corresponding to the frame number. The reason why two frames can be stored will be described later.

  The coarse alignment processing units 22-1, 22-2, 22-3 and 22-4 are based on the images read from the image storage memories 21-1, 21-2, 21-3 and 21-4. Alignment processing for space stabilization that suppresses image blur between frames is performed. The coarse alignment processing units 22-1 to 22-4 extract image pyramids from the image storage memories 21-1 to 21-4 each time the frame timing signal 15 is input from the infrared imaging device control unit 6. To do. In the known technology, the image pyramid has a hierarchical structure including a plurality of images (referred to as pyramid images as appropriate) having different resolutions from high resolution to low resolution.

  The coarse registration processing units 22-1 to 22-4 estimate correction coefficients for stabilizing the space of the image for each frame. The coarse registration processing units 22-1 to 22-4 perform geometric image correction using the estimated correction coefficient.

  The interpolation processing units 23-1 to 23-4 read out the images that have undergone the alignment processing in the coarse registration processing units 22-1 to 22-4 and the image storage memories 21-1 to 21-4. An image is input. Interpolation processing units 23-1, 23-2, 23-3, and 23-4 apply the images that have undergone the alignment processing in coarse alignment processing units 22-1, 22-2, 22-3, and 22-4. On the other hand, interpolation processing for improving image quality is performed.

  For example, the coarse registration processing units 22-1 to 22-4 perform geometric image correction using a known method called affine transformation. Affine transformation is one of the methods for geometrically moving and deforming an image by transforming the coordinates of each pixel in the image data. The coarse alignment processing units 22-1 to 22-4 perform affine transformation based on, for example, the following equation (3).

  Here, (x, y) are coordinates before affine transformation, (u, v) are coordinates after affine transformation, and a, b, c, d, e, and f are affine transformation coefficients.

  In the image data, when coordinates in units of pixels are defined, the coordinates are represented only by integers, whereas the coordinates after affine transformation are usually accompanied by a decimal point. Since a density value does not exist at a position designated by coordinates with a decimal point by affine transformation, resolution degradation before the conversion becomes a problem. Therefore, the interpolation processing units 23-1 to 23-4 calculate density values at coordinates after the affine transformation by interpolation processing.

  For example, the interpolation processing units 23-1 to 23-4 perform interpolation processing on the image data after affine transformation using a known method called bilinear interpolation. In bilinear interpolation, the density value at each position on the coordinate axis after the affine transformation is obtained by linear interpolation according to the distance from four pixels located around the corresponding position in the original image.

  FIG. 6 is a conceptual diagram illustrating bilinear interpolation. When the position of the original image corresponding to a certain coordinate after the coordinate conversion is the position P = (u, v) between the pixels as shown in FIG. 6, the luminance value at the position on the original image is not exist. For this reason, the interpolation processing units 23-1 to 23-4 have four pixels P1 = (i, j), P2 = (i, j + 1), P3 = (i + 1, j) and P4 = ( The luminance values of i + 1, j + 1) are added at a ratio corresponding to the distance from each pixel to the position P to obtain the luminance value at the position P. Note that i = [u], j = [v] ([] is a Gaussian symbol).

For example, the interpolation processing units 23-1 to 23-4 obtain the luminance value at the position P by bilinear interpolation using the following equation (4).
P = (1−Δi) (1−Δj) * P1 + (1−Δi) Δj * P2 + Δi (1−Δj) * P3 + ΔiΔj * P4 (4)

  In Expression (4), the terms “P”, “P1”, “P2”, “P3”, and “P4” are the luminance values at the position P and the pixels P1, P2, P3, and P4, respectively. And Further, Δi = u−i and Δj = v−j. By such interpolation processing in the interpolation processing units 23-1 to 23-4, the space stabilization processing unit 3 obtains infrared images 12-1 to 12-4 having the same resolution as that before the affine transformation.

  Next, the operation of the space stabilization processing unit 3 will be described. Infrared images 11-1 to 11-4 input from the infrared imaging devices 2-1 to 2-4 are temporarily stored in the image storage memories 21-1 to 21-4. The coarse registration processing units 22-1 to 22-4 calculate an affine transformation coefficient for spatial stabilization of each image, for example, using a known method called an iterative gradient method.

  According to the iterative gradient method, the coarse registration processing units 22-1 to 22-4 create and compare image pyramids for two consecutive frames. The coarse alignment processing units 22-1 to 22-4 obtain the affine transformation coefficients from the correlations between the low-resolution pyramid images, and correct the affine transformation coefficients by sequentially evaluating the correlations between the high-resolution pyramid images. To go.

  FIG. 7 is a conceptual diagram illustrating an alignment process using an image pyramid. For example, in the pyramid image, the image size (resolution) is gradually changed from level 0 to 4 (level 0: 640 × 480, level 1: 320 × 480, level 2: 160 × 240, level 3: 80 × 120, level 4). : 40 × 60). The coarse registration processing units 22-1 to 22-4 sequentially generate pyramid images at levels 2 to 4, for example, and perform motion detection between frames.

  The coarse alignment processing units 22-1 to 22-4 select pixels from the infrared images 11-1 to 11-4 stored in the image storage memories 21-1 to 21-4 according to the level of each resolution. The image data is read out after thinning out and an image pyramid is generated. As described above, the image storage memories 21-1 to 21-4 store two frames for the infrared images 11-1 to 11-4. The coarse registration processing units 22-1 to 22-4 generate image pyramids for both frames stored in the image storage memories 21-1 to 21-4. Here, as an example, image pyramids for frames I (t−1) and I (t) are generated and motion extraction is performed.

The coarse alignment processing units 22-1 to 22-4, for example, in a known method called optical flow, a correlation that minimizes the error Ec obtained from the following equation (5) that is a constraint equation. Based on the relationship, image motion between frames is estimated.
Ec = Σ (P _x I _x + P _y I _y + I _t ) ² (5)

P _x is the displacement of the image in the X direction (P _x = a + bx + cy), P _y is the displacement of the image in the Y direction (P _y = d + ex + fy), a, b, c, d, e, and f are affine transformations. Coefficient. I _x and I _y are spatial differentials, where I _x represents a difference in luminance value between adjacent pixels in the X direction of the image, and I _y represents a difference in luminance value between adjacent pixels in the Y direction of the image. . I _t is a time derivative, representing the difference in luminance value between image frames.

  The coarse alignment processing units 22-1 to 22-4 calculate an affine transformation coefficient by solving the following equation (6), for example, and perform affine transformation corresponding to the motion detection result. Thereby, the space stabilization process part 3 can suppress the blurring of the image between frames, and can aim at space stabilization.

  In the hierarchical structure of the image pyramid, the movement on the image decreases as the image size decreases. The feature points of a predetermined pyramid image (for example, level 3) having a low resolution are estimated by optical flow, and the obtained affine transformation coefficient is set as an initial value p0. In the next hierarchy (level 2), an affine transformation coefficient correction amount Δp1 is obtained through affine transformation and motion extraction with the affine transformation coefficient as an initial value p0.

  In the next hierarchy (level 1), an affine transformation coefficient correction amount Δp2 is obtained through affine transformation and motion extraction using a value p1 obtained by adding the correction amount Δp1 to the initial value p0. Finally, at level 0, an affine transformation coefficient p3 is obtained through affine transformation and motion extraction using a value p2 obtained by adding the correction amount Δp2 to the value p1.

  As described above, at the initial stage of calculating the affine transformation coefficients, the coarse registration processing units 22-1 to 22-4 obtain the initial value of the affine transformation coefficient by calculating between the images with low resolution, and then gradually increase the resolution. Modify the affine transformation coefficient while increasing. By repeating the processing up to level 0, which is the size of the original image, the reliability against a sudden change in the flow is improved. As a result, the coarse / fine alignment processing units 22-1 to 22-4 can realize high-accuracy and high-speed alignment processing. Note that the upper limit of the number of times the process is repeated for each level is, for example, four times in consideration of the behavior in the response until the correlation value converges.

  Next, the reason why the processing of the space stabilization processing unit 3 falls within the frame period (for example, 33.3 ms) will be described. In the above equation (6), the affine transformation coefficient is obtained by 18 kinds of product-sum operations in each element of the matrix. The product-sum operation circuit can be configured using a multiplier and an adder. For example, if the operation frequency of an FPGA (Field-Programmable Gate Array) constituting the space stabilization processing unit 3 is 100 MHz, the processing by the multiplier and the adder falls within the operation cycle.

  At this time, since the data of one pixel is processed in one clock cycle, the time required for estimating the affine transformation coefficient is calculated by (the pyramid image size of each level) × (number of repetitions) ÷ (operation frequency). Desired.

For example, for the above level 2 pyramid image,
160 × 240 (pixels) × 4 (times) ÷ (100 × 10 ⁻⁶ ) (s ⁻¹ ) ≈1.54 ms
For level 3 pyramid images,
80 × 120 (pixels) × 4 (times) ÷ (100 × 10 ⁻⁶ ) (s ⁻¹ ) ≈0.38 ms
For level 4 pyramid images,
40 × 60 (pixels) × 4 (times) ÷ (100 × 10 ⁻⁶ ) (s ⁻¹ ) ≈0.096 ms
It becomes. When these are added together, the time required for estimating the affine transformation coefficient is approximately 2.02 ms. Similarly, the time required for affine transformation is
640 × 480 (pixels) ÷ (100 × 10 ⁻⁶ ) (s ⁻¹ ) ≈3.07 ms
It becomes.

  The processing time of the space stabilization processing unit 3 is the sum of the time required for estimating the affine transformation coefficient and the time required for the affine transformation, and is approximately 5.1 ms in this example. As described above, the processing in the space stabilization processing unit 3 can be sufficiently accommodated in the frame period of 33.3 ms even when overhead is taken into consideration. Thereby, the space stabilization process part 3 can implement a space stabilization process in real time. The infrared target detection apparatus 1 can detect a target in real time.

Embodiment 2. FIG.
FIG. 8 is a block diagram showing a schematic configuration of the infrared target detection apparatus according to the second embodiment of the present invention. The same parts as those in the first embodiment are denoted by the same reference numerals, and repeated description will be omitted as appropriate. The infrared target detection device 30 according to the second embodiment corresponds to a configuration in which a panoramic image synthesis unit 31 is added to the configuration of the infrared target detection device 1 (see FIG. 1) according to the first embodiment.

  The panoramic image synthesis unit 31 combines a plurality of infrared images 12-1, 12-2, 12-3, and 12-4 to synthesize a panoramic image. The plurality of infrared images 12-1, 12-2, 12-3, and 12-4 are captured by the plurality of infrared imaging devices 2-1 to 2-4, and the stabilization process in the space stabilization processing unit 3 is performed. Then, it is input to the panoramic image composition unit 31. The infrared imager control unit 6 gives the frame timing signal 15 to the panoramic image synthesis unit 31. The panoramic image composition unit 31 operates in synchronization with the frame rate.

  The panoramic image composition unit 31 measures the correlation of the luminance of each infrared image 12-1 to 12-4, and pastes each infrared image 12-1 to 12-4 at a position with high correlation to generate a panoramic image. . The panorama image synthesis unit 31 outputs, for example, a panorama image 32 that has been regenerated for display on a liquid crystal monitor (not shown) that is a display device. The infrared target detection device 30 outputs the panorama image 32 from the panorama image synthesis unit 31.

  Next, the operation of the panoramic image composition unit 31 will be described. FIG. 9 is a diagram for explaining generation of a panoramic image in the panoramic image synthesis unit. Here, the image size of the infrared images 12-1 to 12-4 is VGA (640 × 480 pixels).

  For example, the infrared image 12-1 is used as a reference image, and the infrared image 12-2 is used as a combined image that is combined with the reference image. The panoramic image composition unit 31 designates a pasting instruction position for starting the pasting of the infrared image 12-2 with respect to the infrared image 12-1. The panoramic image composition unit 31 uses the pixels of the infrared image 12-1 in the overlapping region 33 as the base point in the overlapping region 33 of the infrared images 12-1 and 12-2 shown by hatching in the figure. A raster scan with the pixels of the infrared image 12-2 is performed. The panoramic image composition unit 31 measures the luminance correlation between the infrared images 12-1 and 12-2 in the overlapping region 33, for example, using the following equation (7).

Here, S is the evaluation value of the luminance correlation, (X1, Y1) is the coordinates of the bonding instruction position, (x, y) is the coordinates of the base point in the overlapping region 33, and _Rx is included in the overlapping region 33 in the X direction. The number of pixels, R _y is the number of pixels included in the overlapping region 33 in the Y direction, and I ₁ and I ₂ are the luminance values of the infrared images 12-1 and 12-2, respectively.

Panorama image combining unit 31, based instruction position (X1, Y1) bonding, in the X and Y directions, positive and negative of the movable range _E x infrared images 12-2, define the _{E y.} Panorama image combining unit 31, the range of _{X1-E x} of X1 + _{E x} in the X direction, in the range of Y1 + _{E y} from the Y-direction _{Y1-E y,} moving the infrared image 12-2.

  The panorama image composition unit 31 measures the luminance correlation each time the infrared image 12-2 is moved by the pixel unit. The panoramic image synthesis unit 31 performs the movement of the infrared image 12-2 and the measurement of the luminance correlation for all the pixel positions within the movable range. As described above, the panoramic image synthesis unit 31 measures the correlation between the luminances of the infrared images 12-1 and 12-2 in the overlapping region 33 when all the pixels in the movable range are set as the bonding positions.

  Expression (7) is an example of an expression for evaluating the luminance correlation, and uses a known technique called SSD (Sum of Squared Difference). The panoramic image synthesis unit 31 may use another known method for evaluating the luminance correlation, for example, SAD (Sum of Absolute Difference) or NCC (Normalized Cross Correlation).

  The panoramic image composition unit 31 assumes that the bonding position having the highest measurement value among the luminance correlation measurement results is correlated with the luminance of the infrared images 12-1 and 12-2 in the overlapping region 33, and the infrared image is detected at that position. The image 12-2 is pasted.

  When the infrared image 12-2 is pasted, the panoramic image composition unit 31 performs a process for providing brightness continuity at the boundary portion of the pasting. For example, the average of the luminance value of the reference pixel and the luminance value of the combined image is adopted as the luminance value of the panoramic image for the pixels at the same position in the overlapping region 33 in the reference image and the combined image. By this method, the boundary portion between the reference image and the combined image is smoothed.

  The panorama image combining unit 31 combines the infrared images 12-3 and 12-4 by the same procedure as that for combining the infrared image 12-2. When the infrared image 12-3 is used as a composite image, the infrared image 12-2 is used as a reference image. When the infrared image 12-4 is used as a composite image, the infrared image 12-3 is used as a reference image.

  As described above, the panoramic image synthesis unit 31 generates a panoramic image using the infrared images 12-1 to 12-4. The panoramic image combining unit 31 updates the panoramic image every frame period by synthesizing the panoramic image in synchronization with the frame timing signal 15.

Next, regeneration of a panoramic image for display on a liquid crystal monitor or the like will be described. If the image size of the infrared images 12-1 to 12-4 is VGA (640 × 480 pixels), the maximum number of pixels in the X direction of the panoramic image is 640 × 4- (3 × 2 × E _x ). When _Ex is 10 pixels, for example, the maximum number of pixels in the X direction of the panoramic image is 640 × 4− (3 × 2 × 10) = 2500. The maximum number of pixels in the Y direction of the panoramic image is 480+ (2 × E _y ). When the E _y for example, 10 pixels, the maximum number of pixels in the Y direction of the panoramic image becomes (480+ (2 × 10) = 500.

  For example, in a liquid crystal monitor of SXGA size (1280 × 1024 pixels), when a composite image of infrared images 12-1 and 12-2 and a composite image of infrared images 12-3 and 12-4 are arranged vertically Display using almost the entire screen is possible. The panoramic image composition unit 31 regenerates a panoramic image 32 in which the composite image of the infrared images 12-1 and 12-2 and the composite image of the infrared images 12-3 and 12-4 are arranged vertically.

  In other words, the combined image of the infrared images 12-1 and 12-2 arranged in the upper half of the panoramic image 32 corresponds to an azimuth image having an azimuth angle of 0 degrees to 180 degrees. In other words, the combined image of the infrared images 12-3 and 12-4 arranged in the lower half of the panoramic image 32 corresponds to an azimuth image having an azimuth angle of 180 degrees to 360 degrees.

  The panorama image synthesis unit 31 regenerates the panorama image 32 in which the two divided orientation images are arranged up and down, thereby reducing deterioration in pixel resolution of the displayed panorama image 32. It can be displayed effectively. The panorama image composition unit 31 updates the panorama image 32 every frame period by regenerating the panorama image 32 in synchronization with the frame timing signal 15. Thereby, the panoramic image synthesis unit 31 can synthesize the panoramic image 32 in real time.

  The infrared target detection device 30 correlates the infrared images 12-1 to 12-4 and generates and displays a panoramic image 32 in which the infrared images 12-1 to 12-4 are pasted together. Can visually check the target position in the omnidirectional image in real time.

Embodiment 3 FIG.
FIG. 10 is a block diagram showing a schematic configuration of the infrared target detection apparatus according to the third embodiment of the present invention. The same parts as those in Embodiments 1 and 2 are denoted by the same reference numerals, and repeated description is omitted as appropriate.

  The infrared target detection device 40 according to the third embodiment corresponds to a configuration in which a symbol superimposing unit 41 is added to the configuration of the infrared target detection device 30 (see FIG. 8) according to the second embodiment. In the present embodiment, the panorama image composition unit 31 adds the position information 42-1, 42-2 and 42- of the infrared images 12-2, 12-3 and 12-4 in addition to the panorama image 32. 3 is output.

  The symbol superimposing unit 41 superimposes a symbol on the panoramic image 32 according to the bonding position information 42-1 to 42-3 and the target feature amount 16 from the feature amount calculation circuit 5. The symbol is a graphic representing the detected target. The infrared imager control unit 6 gives the frame timing signal 15 to the symbol superimposing unit 41. The symbol superimposing unit 41 operates in synchronization with the frame rate. The symbol superimposing unit 41 outputs a panoramic image 32 on which symbols are superimposed. The infrared target detection device 40 outputs the panoramic image 32 from the symbol superimposing unit 41.

  Next, the operation of the symbol superimposing unit 41 will be described. FIG. 11 is a diagram illustrating an example of a panoramic image on which symbols are superimposed. The symbol superimposing unit 41 superimposes the symbol 43 and the scale pattern 44 on the panoramic image 32 synthesized from the infrared images 12-1 to 12-4.

  The scale pattern 44 is a scale that represents the azimuth and elevation angle of the panoramic image 32. The horizontal axis of the scale pattern 44 represents the azimuth angle, and the vertical axis represents the elevation angle. The scale pattern 44 is stored in, for example, an internal memory of the FPGA that constitutes the symbol superimposing unit 41. The symbol superimposing unit 41 reads the scale pattern 44 from the internal memory and superimposes it on a predetermined position of the panoramic image 32.

  The symbol 43 has, for example, a square outline. The center position of the square represents the target center of gravity position. The square area represents the target area. The symbol superimposing unit 41 grasps the target center-of-gravity position and area based on the target feature amount 16 from the feature amount calculation circuit 5. The symbol superimposing unit 41 generates a symbol 43 centered on the grasped center of gravity position and having a size adjusted according to the area.

  The symbol superimposing unit 41 obtains the position of the symbol 43 on the panoramic image 32 according to the pasting position information 42-1 to 42-3 from the panoramic image synthesizing unit 31. The definition of the coordinate system of the panoramic image 32 and the coordinate system of the target barycentric position output from the feature amount calculation circuit 5 are different. The symbol superimposing unit 41 converts the target center-of-gravity position so as to conform to the coordinate system of the panoramic image 32.

  FIG. 12 is a diagram for explaining the coordinate system of the panoramic image and the coordinate system of the target center-of-gravity position output from the feature amount calculation circuit. Here, a case where the coordinate system of the target center-of-gravity position is the coordinate system of the infrared image 12-2 is taken as an example. The coordinate system of the panoramic image 32 is defined with the origin O as a reference. The coordinate system of the target center-of-gravity position is defined with reference to the origin O ′. The symbol superimposing unit 41 converts the coordinates of the target barycentric position so as to eliminate such a difference in the definition of the coordinate system.

  The symbol superimposing unit 41 updates the symbol 43 every frame period in accordance with the target feature quantity 16 and the bonding position information 42-1 to 42-3 updated every frame period. Thereby, the symbol superimposing unit 41 can superimpose the symbol 43 representing the detected target in real time.

  By enabling the infrared target detection device 40 to display the symbol 43 superimposed on the panoramic image 32, the operator can easily visually check the target position and the feature amount in real time.

  The shape of the symbol 43 is not limited to the square shape described in the present embodiment, and can be changed as appropriate. The symbol 43 may display other feature amounts in addition to the center of gravity position and area for the target.

DESCRIPTION OF SYMBOLS 1, 30, 40 Infrared target detection apparatus 2-1, 2-2, 2-3, 2-4 infrared imaging device 3 Spatial stabilization processing part 4 Target detection processing part 5 Feature-value calculation circuit 6 Infrared imaging device control part 10 Infrared images 11-1, 11-2, 11-3, 11-4 Infrared images 12-1, 12-2, 12-3, 12-4 Infrared images 13-1, 13-2, 13-3, 13-4 Binary image 14-1, 14-2, 14-3, 14-4 Imaging command 15 Frame timing signal 16 Target feature 17-1, 17-2, 17-3, 17-4 Imaging field of view 21-1, 21-2, 21-3, 21-4 Image storage memory 22-1, 22-2, 22-3, 22-4 Coarse alignment processing unit 23-1, 23-2, 23-3, 23- 4 Interpolation Processing Unit 31 Panorama Image Composition Unit 32 Panorama Image 33 Overlap Area 41 symbol superimposing unit 42-1,42-2,42-3 bonding position information 43 symbol 44 scale pattern

Claims (3)

A plurality of infrared imagers that are arranged so as to be able to image all directions by combining each other's field of view, detect the collected infrared rays, and image infrared images;
A space stabilization processing unit that performs a space stabilization process for correcting the movement of the infrared image due to the displacement of the infrared imaging device;
A target detection processing unit that performs a binarization process of a luminance value for each pixel on the infrared image that has undergone the spatial stabilization process in the spatial stabilization processing unit, and detects a target from the infrared image;
A feature amount calculation unit that calculates the target feature amount based on the binary image acquired by the binarization processing in the target detection processing unit,
A frame timing signal synchronized with a frame rate of the infrared image is supplied to the space stabilization processing unit , the target detection processing unit, and the feature amount calculation unit ,
The space stabilization processing unit performs the space stabilization processing per frame within a cycle of the frame timing signal,
The target detection processing unit performs the binarization processing per frame within a cycle of the frame timing signal ,
The infrared target detection apparatus, wherein the feature amount calculation unit calculates the feature amount in synchronization with the frame rate .   A panoramic image synthesizing unit that synthesizes a panoramic image by combining the plurality of infrared images captured by the plurality of infrared imaging devices and subjected to the spatial stabilization processing in the spatial stabilization processing unit; The infrared target detection apparatus according to claim 1.   The infrared target detection apparatus according to claim 2, further comprising a symbol superimposing unit that superimposes a symbol that is a graphic representing the target on the panoramic image.

Images