JP6813046B2 - Image flow correction device, image flow correction method and program - Google Patents

Description

The present invention relates to an image flow correction device, an image flow correction method, and a program.

Patent Document 1 discloses a method of correcting an image flow that occurs in an image captured by an infrared imaging device. In Patent Document 1, the original brightness information is predicted from the difference brightness information between the correction target pixel and the reference pixel separated from the correction target pixel by the number of pixels corresponding to the rotation angular velocity in the rotation direction, and the image flow is corrected. ing. That is, in Patent Document 1, the luminance value of the signal deteriorated by the image flow is amplified by multiplying the luminance value by a coefficient corresponding to the rotational angular velocity.

Patent Document 2 discloses a method of correcting an image flow generated in an image captured by an imaging device based on an angular velocity detected by an angular velocity detector.

Patent Document 3 discloses that an angular velocity sensor and an acceleration sensor detect angular velocity and acceleration based on vibration generated in an imaging device such as a camera, and based on these, correct blurring that occurs in an image captured by the imaging device. doing.

Patent Document 4 detects vibration components and translational components based on vibration generated in an image pickup device such as a camera by an angular velocity sensor and an acceleration sensor, and corrects blurring that occurs in an image captured by the image pickup device based on them. Is disclosed.

Japanese Unexamined Patent Publication No. 2007-028349 Japanese Unexamined Patent Publication No. 2001-036794 Japanese Unexamined Patent Publication No. 2005-114845 Japanese Unexamined Patent Publication No. 2006-145769

In any of Patent Documents 1 to 4, the image flow generated in the image captured by the image pickup device is corrected based on the rotational motion of the image pickup device.
However, in Patent Documents 1 to 4, when an image pickup device is mounted on a moving body such as a vehicle or a drone to take an image, the image flow generated based on the rotational motion of the image pickup device can be corrected, but the image pickup device It was not possible to simultaneously correct the image flow generated based on the translational motion of.

Therefore, an object of the present invention is to provide an image flow correction device, an image flow correction method, and a program that solve the above-mentioned problems.

According to the first aspect of the present invention, the image flow correction device is
A conversion unit that converts the moving speed of the object in the captured image into the first rotational angular velocity,
The combined rotational angular velocity is calculated by synthesizing the second rotational angular velocity of the object in the captured image and the first rotational angular velocity converted by the conversion unit, and is based on the combined rotational angular velocity. , A correction unit that corrects the image flow of the object,
To be equipped.

According to the second aspect of the present invention, the image flow correction method is
The moving speed of the object in the captured image is converted into the first rotational angular velocity,
The combined rotational angular velocity is calculated by synthesizing the second rotational angular velocity of the object in the captured image and the converted first rotational angular velocity.
Image flow correction of the object is performed based on the combined rotational angular velocity.

According to the third aspect of the present invention, the computer of the image flow correction device is
A conversion means for converting the moving speed of an object in a captured image into a first rotation angle speed,
The combined rotation angle velocity is calculated by synthesizing the second rotation angle velocity of the object in the captured image and the first rotation angle velocity converted by the conversion means, and based on the combined rotation angle velocity. , A correction means for correcting the image flow of the object,
To function as.

According to the present invention, not only the image flow generated based on the rotational motion but also the image flow generated based on the translational motion can be corrected at the same time.

It is a block diagram which shows the structure of the image flow correction apparatus by 1st Embodiment. It is a figure which shows an example of the infrared image which the infrared detector by 1st Embodiment takes. It is a figure explaining the angular velocity based on the rotational motion by 1st Embodiment. It is a figure explaining the angular velocity based on the translational motion by 1st Embodiment. It is a figure explaining the synthetic angular velocity by 1st Embodiment. It is a figure explaining the angle of view per pixel by 1st Embodiment. It is a flowchart which shows the processing of the image flow correction apparatus by 1st Embodiment. It is a block diagram which shows the structure of the image flow correction apparatus by 2nd Embodiment. It is a flowchart which shows the processing of the image flow correction apparatus by 2nd Embodiment. It is a block diagram which shows the structure of the image flow correction apparatus which has the minimum structure. It is a flowchart which shows the process of the image flow correction apparatus which has the minimum configuration.

[First Embodiment]
First, the first embodiment of the present invention will be described.
In the first embodiment, a case where image flow due to rotational motion and translational motion occurs in two directions (x-axis direction and y-axis direction) will be described. For example, the image pickup device rotates so that the object, which is the subject, moves from the pixel position (x, y) on the screen in the direction of the pixel position (x + m, y + n), and the vehicle equipped with the image pickup device. However, a case of correcting the image flow generated during translational motion will be described.
Note that the image flow means that the image of the object in the captured image is blurred or blurred due to the imaging device taking an image of the object while performing a rotational motion or a translational motion.

FIG. 1 is a block diagram showing a configuration of an image flow correction device 100a according to the first embodiment.
The image flow correction device 100a includes an infrared image pickup device 10, a speed detector 20, a correction unit 30, and a conversion unit 37.
The infrared image pickup device 10 includes an infrared detector 11, a processing circuit 12, and an angular velocity detector 13.
The correction unit 30 includes an image memory 31, an image memory 32, an arithmetic unit 33, a subtractor 34, a divider 35, and an adder 36.

The infrared detector 11 is composed of an infrared camera or the like installed on the upper surface of a vehicle or the like. The infrared detector 11 takes an infrared image including an object as a subject by rotating around an axis perpendicular to the upper surface of the vehicle. The infrared detector 11 outputs an infrared image including the luminance information of each pixel to the processing circuit 12.
The processing circuit 12 converts the infrared image output from the infrared detector 11 into A / D (analog / digital), performs predetermined processing such as sensitivity correction, and then performs a predetermined process such as sensitivity correction, and then displays the image memory as two-dimensional digital image data. Output to 31 and the image memory 32.
The image memory 31 and the image memory 32 store the digital image data output from the processing circuit 12.

Further, the angular velocity detector 13 is configured by a gyroscope or the like, detects the rotational angular velocity based on the rotational motion of the infrared imaging device 10, and outputs the rotational angular velocity to the arithmetic unit 33.
The speed detector 20 detects the moving speed of a vehicle or the like equipped with the image flow correction device 100a provided with the speed detector 20 and outputs the speed to the conversion unit 37.
The angular velocity detector 13 and the velocity detector 20 are installed at positions where the rotational motion and translational motion equivalent to those of the infrared imaging device 10 are performed. However, when the vehicle and the infrared imaging device 10 are not moving relative to each other, the angular speed detector 13 or the speed detector 20 may be installed at an arbitrary position of the vehicle.
The conversion unit 37 converts the moving speed input from the speed detector 20 into a rotation angular velocity and outputs it to the arithmetic unit 33.

The image memory 31 sequentially outputs brightness data determined from the stored digital image data from the upper left pixel of the image. For example, the image memory 31 outputs the luminance data A (x, y) at the pixel position (x, y) as the luminance data of the pixel to be corrected at a predetermined time point.
The image memory 32 subtracts the brightness data A (x + m, y + n) of the pixel position (x + m, y + n) as the brightness data of the reference pixel based on the reference pixel position (x + m, y + n) output from the arithmetic unit 33. Output to the device 34 and the adder 36.

The subtractor 34 subtracts the luminance data output from the image memory 32 from the luminance data output from the image memory 31 and outputs it to the divider 35.
The divider 35 divides the data output from the subtractor 34 using the divisors α _X and α _Y output from the arithmetic unit 33, and outputs the data to the adder 36.
The adder 36 adds the data output from the divider 35 and the luminance data of the reference pixel output from the image memory 32.
The image flow correction device 100a outputs the data added by the adder 36 as the luminance data B (x, y) after the image flow correction.

The arithmetic unit 33 obtains a combined angular velocity ω obtained by combining the angular velocity ω _R output from the angular velocity detector 13 and the angular velocity ω _D output from the conversion unit 37.
For example, when the infrared imaging device 10 equipped with the angular velocity detector 13 is rotating, and the vehicle equipped with the image flow correction device 100a is moving in translation in a predetermined direction. , The infrared detector 11 captures an infrared image as shown in FIG.

The angular velocity vector based on the rotational motion is three-dimensional in the roll, pitch, and yaw directions, but is converted into two dimensions in the x-axis direction and the y-axis direction on the infrared image. The direction F2 of the image flow caused by the rotational motion is calculated from the two-dimensional vector by the arithmetic unit 33. In the present embodiment, the position of the reference pixel is obtained based on the calculated image flow direction and the magnitude of the rotation angular velocity, and the original luminance information is predicted from the difference luminance information between the correction target pixel and the reference pixel. And correct the image flow.
In the present embodiment, the moving speed based on the translational motion such as running is converted into the rotational angular velocity, which is combined with the rotational angular velocity vector based on the rotational motion, and the image flow correction process is performed. For example, when the infrared imaging device 10 is mounted on a vehicle, it is assumed that the vehicle to be mounted moves not only in a stationary state but also in a moving state. At this time, image flow occurs. At this time, if the traveling speed of the infrared imaging device 10 is known, the moving direction of the object on the infrared image is relative to the object (stationary state, constant velocity linear motion, etc.) whose motion state can be known. Can be calculated. If the distance between the infrared imaging device 10 and the object can be calculated, the magnitude of the amount of movement of the object on the infrared image (the number of moving pixels per unit time) can also be calculated. By converting the direction and amount of movement due to this translational motion into the direction and magnitude of the image flow due to the rotational angular velocity and adding them to the rotational angular velocity vector, in the present embodiment, not only the image flow based on the rotational angular velocity but also the image flow is obtained. It also corrects the image flow based on the translational motion.

The horizontal axis direction of FIG. 2 shows the pixel numbers of the pixels in the x-axis direction of the infrared image captured by the infrared detector 11. Further, the left side in the vertical axis direction of FIG. 2 shows the pixel numbers of the pixels in the y-axis direction of the infrared image captured by the infrared detector 11. Further, the right side in the vertical axis direction of FIG. 2 shows the brightness value of the infrared image captured by the infrared detector 11.
In FIG. 2, since the vehicle equipped with the image flow correction device 100a translates in a predetermined direction, an image flow based on the translation is generated in the F1 direction. Further, since the infrared imaging device 10 on which the angular velocity detector 13 is mounted rotates, an image flow based on the rotational movement occurs in the F2 direction.

Data of rotation angular velocity ω _R as shown in FIG. 3A is output from the angular velocity detector 13 to the arithmetic unit 33. Based on the rotational angular velocity ω _R and the vertical angle θ1 formed in FIG. 3A, the arithmetic unit 33 uses the following equations (1) and (2) to determine the angular velocity ω _{R_X} in the x-axis direction due to the rotational motion, and , _Obtain the angular velocity ω _{R_Y} in the y-axis direction due to the rotational motion.

ω _{R_X} = ω _R × sin (θ1) ・ ・ ・ (1)
ω _{R_X} = ω _R × cos (θ1) ・ ・ ・ (2)

Further, data of the rotational angular velocity ω _D as shown in FIG. 3B is input to the arithmetic unit 33 from the conversion unit 37. The calculator 33 uses the following equation (3) based on the rotation angle velocity ω _D and the angle θ2 formed by the vertical direction (optical axis direction of the infrared imaging device 10) in FIG. 3B, and uses the following equation (3) to move in the x-axis direction by translational motion. the angular velocity omega _{D_X,} and determines the y-axis direction of the angular velocity omega _{D_y} by translational movement.

ω _{D_X} = ω _D × sin (θ2) ・ ・ ・ (3)
ω _{D_Y} = ω _D × cos (θ2) ・ ・ ・ (4)

Calculator 33, the angular velocity omega _{r_x} the x-axis direction by the rotational movement, the angular velocity omega _{D_X} the x-axis direction by the translational movement, by adding, obtaining the synthetic angular velocity omega _X in the x-axis direction shown in FIG. 6C. The arithmetic unit 33 includes an angular velocity omega _{r_y} the y-axis direction by the rotational motion, the angular velocity omega _{D_y} the y-axis direction by the translational movement, by adding a synthetic angular velocity omega _Y in the y-axis direction shown in FIG. 6C Ask.

Here, the arithmetic unit 33 decomposes the rotation _angle velocity ω _R into ω _{R_X} and ω _{R_Y} , decomposes the rotation _angle velocity ω _D output from the conversion unit 37 into ω _{D_X} and ω _{D_Y,} and decomposes them into ω _{R_X.} and obtains the omega _X by combining omega _{D_X,} the description has been given of the case of obtaining the omega _Y by synthesizing omega _{r_y} and omega _{D_y,} but is not limited thereto. For example, the arithmetic unit 33 obtains the combined angular velocity ω, which is the vector sum of the rotational angular velocity ω _R (see FIG. 3A) and the rotational angular velocity ω _D (see FIG. 3B), and then the following equations (5) and (6). ) May be used to decompose the combined angular velocity ω _X in the x-axis direction and the combined angular velocity ω _Y in the y-axis direction.

ω _X = ω × sin (θ) ・ ・ ・ (5)
ω _Y = ω × cos (θ) ・ ・ ・ (6)

The arithmetic unit 33 obtains and refers to the positive numbers m and n indicating the reference pixel positions in the x-axis direction and the y-axis direction by using the following equations (7) and (8) based on the combined angular velocity ω. The pixel positions (x + m, y + n) are output to the image memory 32.

m = (| ω _X | / IFOV) × Δt _{0_X} ... (7)
n = (| ω _Y | / IFOV) × Δt _{0_Y} ... (8)

The positive number m in the above equation (7) indicates the number of pixels in which the object moves in the x-axis direction at the time Δt _{0_X} . Further, the positive number n in the above equation (8) indicates the number of pixels in which the object moves in the y-axis direction at the time Δt _{0_Y} .
The value IFOV in the above equations (7) and (8) is the angle IFOV shown in FIG. For example, a case where the distance between the infrared image pickup device 10 mounted on the vehicle and the object to be imaged by the infrared image pickup device 10 is known will be described. The distance between the infrared imaging device 10 and the object can be measured by using, for example, a laser distance measuring device or the like.
As shown in FIG. 4, the distance between the infrared imaging device 10 and the object is on the ground surface imaged by the infrared imaging device 10 and the infrared imaging device 10 when the infrared imaging device 10 images the ground surface. Corresponds to the distance L [m]. The projection distance L [m] per pixel at the target position can be geometrically calculated from the angle of view IFOV per pixel.
The speed detector 20 detects v [m / s] of the moving speed of the infrared imaging device 10 and outputs it to the conversion unit 37.
The conversion unit 37 converts the moving speed v [m / s] output from the speed detector 20 into the angular velocity ω _D by using the following equation (9).

ω _D = IFOV [rad] / (time for the infrared imaging device 10 to pass the projection distance L) [s]
= IFOV / (L / v) ・ ・ ・ (9)

The values Δt _{0_X} and Δt _{0_Y} in the above equations (7) and (8) can be obtained by the following equations (10) and (11).

Δt _{0_X} = IFOV / ω _X ... (10)
Δt _{0_Y} = IFOV / ω _Y ... (11)

From the above equations (7) and (8), positive numbers m and n for specifying the reference pixel position (x + m, y + n) with respect to the pixel position (x, y) of the object can be specified.

The calculator 33 uses the following equations (12) and (13) to correct a correction coefficient α _X for correcting the brightness value in the x-axis direction and a correction coefficient α for correcting the brightness value in the y-axis direction. _{Find Y.}

α _X = 1-exp ( _{−Δt 0_X} / τ) ・ ・ ・ (12)
α _Y = 1-exp ( _{−Δt 0_Y} / τ) ・ ・ ・ (13)

In the above equations (12) and (13), the value τ is the thermal time constant of the detection element constituting the thermal infrared detector 11.

The subtractor 34, the divider 35, and the adder 36 constituting the correction unit 30 are the brightness data A (x) and A (y) of the correction target pixel specified by the digital image data stored in the image memory 31. Using the brightness data A (x + m) and A (y + n) of the reference pixel specified by the digital image data stored in the image memory 32, the correction target pixel is subjected to the following equations (14) and (15). The corrected brightness data B (x) and B (y) are obtained.

B (x) = {A (x) -A (x + m)} / α _X + A (x + m) ・ ・ ・ (14)
B (y) = {A (y) -A (y + n)} / α _Y + A (y + n) ... (15)

Luminance data B (x) and B (y) after correction of the correction target pixel are output from the adder 36, and these luminance data B (x) and B (y) are rotational motion and translational motion. This is the luminance data in which the image flow generated based on the above is corrected.

FIG. 5 is a flowchart showing the processing of the image flow correction device 100a according to the first embodiment.
First, the speed detector 20 detects v [m / s] of the moving speed of the infrared imaging device 10 (step S1).
Next, the conversion unit 37 geometrically obtains the projection distance L [m] (see FIG. 4) on the ground plane per pixel of the infrared detector 11 from the angle of view IFOV per pixel (step). S2).
Next, the conversion unit 37 is based on the projection distance L [m] per pixel at the target position, the image angle IFOV per pixel, and the movement speed v [m / s] detected by the speed detector 20. The angular velocity ω _D is calculated using the above equation (9) (step S3).
Next, the calculation unit 33 uses the above equations (3) and (4) from the installation angle θ2 (see FIG. 3B) of the infrared image pickup apparatus 10 to two directions based on translational motion (that is, the x-axis direction). And the angular velocity ω _{D_X in the} y-axis direction) and the angular velocity ω _{D_Y} are calculated (step S4).

On the other hand, the arithmetic unit 33 uses the above equations (1) and (2) based on the angular velocity ω _R based on the angular velocity detected by the angular velocity detector 13, and has two directions based on the rotational motion (that is, the x-axis). The angular velocity ω _{R_X in} the direction and the y-axis direction) and the angular velocity ω _{R_Y} are calculated (step S5).
Next, the calculation unit 33 obtains the combined angular velocities ω _X and ω _Y (step S6). Specifically, the calculation unit 33 _obtains the combined angular velocity ω _X in the x-axis direction by synthesizing the angular velocity ω _{D_X} calculated in step S4 and the angular velocity ω _{R_X} calculated in step S5. Further, the calculation unit 33 _obtains the combined angular velocity ω _Y in the y-axis direction by synthesizing the angular velocity ω _{D_Y} calculated in step S4 and the angular velocity ω _{R_Y} calculated in step S5.

Next, the arithmetic unit 33 obtains the time for the object to be imaged by the infrared image pickup device 10 to move by one pixel of the infrared detector 11 (step S7). Specifically, the arithmetic unit 33 uses the above equation (10) based on the combined angular velocity ω _X obtained in step S6, so that the object is one pixel of the infrared detector 11 in the x-axis direction. _Find the time Δt _{0_X} to move to. Further, the arithmetic unit 33 moves the object in the y-axis direction by one pixel of the infrared detector 11 by using the above-mentioned equation (11) based on the combined angular velocity ω _Y obtained in step S6. _{Find the} time Δt _{0_Y} .

Next, the arithmetic unit 33 calculates the positive numbers m and n for specifying the reference pixel position (x + m, y + n) (step S8). Specifically, the arithmetic unit 33 _obtains a positive number m by using the above equation (7) based on the ω _X obtained in step S6 and the time Δt _{0_X} obtained in step S7. Further, the arithmetic unit 33 _obtains a positive number n by using the above equation (8) based on the ω _Y obtained in step S6 and the time Δt _{0_Y} obtained in step S7.

Next, the arithmetic unit 33 calculates the correction coefficient (step S9). Specifically, the calculation unit 33 uses the above equation (12) based on the time Δt _{0_X} obtained in step S7 to obtain an image flow generated in the x-axis direction in the image captured by the infrared imaging apparatus 10. Find the correction coefficient α _X used for correction. Further, the calculation unit 33 corrects the image flow generated in the y-axis direction in the image captured by the infrared imaging device 10 by using the above-mentioned equation (13) based on the time Δt _{0_Y} obtained in step S7. The correction coefficient α _Y used for is obtained.

Next, the arithmetic unit 33 determines the direction in which the image flow is occurring from the combined angular velocities ω _X and ω _Y obtained in step S6 (step S10). For example, the arithmetic unit 33 determines that image flow due to the rotational motion and translational motion of the infrared image pickup apparatus 10 occurs in the directions of the combined angular velocity ω _X in FIG. 3C and the angular velocity vector ω determined by the combined angular velocity ω _Y. To do.

Next, the subtractor 34, the divider 35, and the adder 36 calculate the corrected luminance value (step S11). Specifically, the subtractor 34 calculates the value of {A (x) -A (x + m)}, and the divider 35 calculates the value of {A (x) -A (x + m)} / α _X. Then, the adder 36 calculates the value of {A (x) −A (x + m)} / α _X + A (x + m), thereby calculating the corrected x-axis direction of the correction target pixel of the above equation (14). The brightness data B (x) of Further, the subtractor 34 calculates the value of {A (y) −A (y + n)}, and the divider 35 calculates the value of {A (y) −A (y + n)} / α _Y and adds them. The device 36 calculates the value of {A (y) −A (y + n)} / α _Y + A (y + n) to obtain the corrected brightness data in the y-axis direction of the correction target pixel of the above equation (15). Find B (y).
Next, the adder 36 outputs the corrected luminance data B (x) and B (y) of the correction target pixel obtained in step S11 in the x-axis direction and the y-axis direction.

By processing the flowchart of FIG. 5, the image flow correction device 100a uses the brightness of the reference pixel located at the tip of the angular velocity vector ω (see FIG. 3C) to use the brightness of the reference pixel located at the tip of the angular velocity vector ω, and the object located at the base of the angular velocity vector ω By correcting the brightness, it is possible to correct the image flow caused by the translational motion and the rotational motion.

The image flow based on the rotational motion is generated by the rotation and shaking of the infrared imaging device 10. On the other hand, the image flow based on the translational motion is generated by the movement of the infrared imaging device 10 and the movement of the object (subject). When the distance between the infrared image pickup device 10 and the object is sufficient, the effect of the speed correction is small. On the other hand, the closer the infrared imaging device 10 is to the object, the greater the effect of speed correction. This is because the shorter the distance, the larger the movement of the object on the infrared image due to traveling or the like. For example, for a vehicle-mounted infrared image pickup device 10 in which the distance between the infrared image pickup device 10 and an object is several meters to several hundred meters, the effect of speed correction can be sufficiently increased.

The motion of the object consists of a combination of rotational motion and translational motion. With the image flow correction process of the present embodiment, it is possible to correct the image flow in any combination of motions.

[Second Embodiment]
Next, a second embodiment of the present invention will be described.
In the second embodiment, the parts that adopt the same configuration as the above-mentioned first embodiment or the parts that perform the same processing as the above-mentioned first embodiment are designated by the same reference numerals and are assigned the same reference numerals. The explanation is omitted.

FIG. 6 is a block diagram showing a configuration of the image flow correction device 100b according to the second embodiment. The image flow correction device 100b is different from the configuration of the image flow correction device 100a according to the first embodiment (see FIG. 1) in that the image flow correction device 100b includes the distance detector 50 and the control unit 51.
The distance detector 50 is composed of a laser type distance measuring device or the like. The distance detector 50 measures the distance between the image flow correction device 100b and the object imaged by the infrared image pickup device 10, and outputs the measurement result to the control unit 51.
The control unit 51 controls the infrared detector 11, the angular velocity detector 13, and the speed detector 20 based on the measurement result output by the distance detector 50.

FIG. 7 is a flowchart showing processing of the image flow correction device 100b according to the second embodiment. The flowchart shown in FIG. 7 is different from the flowchart of the first embodiment (FIG. 5) in that the processes of steps S20 and S21 are executed.
First, the distance detector 50 measures the distance between the image flow correction device 100b and the object imaged by the infrared image pickup device 10 (step S20).

Next, the control unit 51 determines whether or not the distance measured in step S20 is less than a predetermined distance (step S21). If it is determined that the distance measured in step S20 is not less than the predetermined distance, the control unit 51 determines YES in step S21, and powers the infrared detector 11, the angular velocity detector 13, and the speed detector 20. Is stopped, and the processing of the flowchart shown in FIG. 7 is terminated. In this case, the control unit 51 may cause the distance detector 50 to perform the process of step S20 again after a predetermined time (for example, after 1 second) has elapsed.

On the other hand, when it is determined that the distance measured in step S20 is less than a predetermined distance, the control unit 51 determines NO in step S21 and goes to the infrared detector 11, the angular velocity detector 13, and the speed detector 20. The image flow correction device 100b is made to perform the processes of steps S1 to S12 described in the flowchart (FIG. 5) of the first embodiment by continuing to supply the power of.
The predetermined distance used for the determination in step S21 of FIG. 7 is preferably set to several hundred meters, for example, 300 m, more preferably 200 m.

According to the second embodiment, when the distance between the image flow correction device 100b and the object imaged by the infrared imaging device 10 is not less than a predetermined distance, that is, the image flow correction device 100b and the object If the distance is too long, image flow correction can be prevented.
If the distance between the image flow correction device 100b and the object imaged by the infrared imaging device 10 becomes too large, the moving distance of the object in the image captured by the infrared imaging device 10 becomes small, and the image flow correction is performed. Although the correction effect is also small, in such a case, since the processes of steps S1 to S12 of FIG. 7 are not performed in the second embodiment, the power consumption of the image flow correction device 100b can be suppressed.

FIG. 8 is a block diagram showing the configuration of the image flow correction device 100c having the minimum configuration. The image flow correction device 100c includes a conversion unit 61 and a correction unit 62.

FIG. 9 is a flowchart showing the processing of the image flow correction device 100c.
The conversion unit 62 converts the moving speed v [m / s] of the object in the captured image into the first rotation angular velocity ω _D (step S31), and outputs it to the correction unit 62. The correction unit 62 calculates the combined rotational angular velocity ω by synthesizing the second rotational angular velocity ω _R of the object in the captured image and the first rotational angular velocity ω _D converted by the conversion unit 61 ( Step S32). The correction unit 62 corrects the image flow of the object based on the combined rotational angular velocity ω (step S33).

In the above-described embodiment, the case where the image flow correction device 100a or the like including the infrared imaging device 10 is mounted on a vehicle such as a car has been described, but the present invention is not limited to this. For example, by mounting an image flow correction device 100a including the infrared image pickup device 10 on a turntable such as a radar or an unmanned aircraft such as a drone, the rotational motion generated by the infrared image captured by the infrared image pickup device 10 And the image flow based on the translational motion may be corrected.

In the above-described embodiment, the time when the infrared detector 11 captures the infrared image, the time when the angular velocity detector 13 detects the angular velocity ω _R, and the time when the speed detector 20 detects the angular velocity ω _D are all. It is preferable to set the same time T. By doing so, it is possible to correct the image flow that occurs at the same time based on the rotational motion and the translational motion. For example, by setting the time T to the current time, the image flow correction device simultaneously performs the rotation angle velocity correction for the image flow based on the rotational motion and the velocity correction for the image flow based on the translational motion in real time. It can be carried out.

In the above-described embodiment, the case where the infrared imaging device 10 is used as the device for detecting the image of the object has been described. When an uncooled sensor of an infrared camera is used as the infrared imaging device 10, the response speed is slow and the image flow is remarkable. Therefore, by applying this embodiment, the image flow can be appropriately corrected. ..
However, if the rotation angle speed is very large not only in the infrared imaging device 10 but also in other sensors, image flow may occur as long as the response speed is finite. Therefore, by applying this embodiment, , Image flow can be corrected.

In FIG. 4, the case where the infrared imaging device 10 images the ground surface has been described as an example, but the present invention is not limited to this. For example, when the distance between the infrared imaging device 10 and the object (subject) or the imaging distance of the infrared imaging device 10 is known, the present embodiment can be applied. For example, when the infrared imaging device 10 is mounted on an unmanned aerial vehicle such as a drone, it is possible to correct the image flow that occurs when imaging another flying object.

In recent years, automatic driving techniques using various sensors have been developed, but according to the above-described embodiment of the present invention, both the rotational angular velocity and the traveling speed can be determined by using the infrared imaging device 10 mounted on the vehicle. The visibility of the infrared image can be improved by correcting the image flow in the coexisting scene.

The speed detector 20 is generally mounted on a vehicle, but according to the above-described embodiment of the present invention, it can be obtained in a small size and inexpensively due to the development of MEMS (Micro Electro Mechanical Systems) technology. By simply using the angular velocity sensor (gyro sensor, etc.) in combination with the infrared imaging device 10, it can be realized without mounting other equipment.

In addition, since the image flow correction processing by software can be realized by simple four arithmetic operations or trigonometric function operations, complicated processing is not required, and image flow correction based on rotational motion and translational motion can be performed in real time. It can be carried out.

In addition, the above-described embodiment also applies to a general infrared camera, a security and surveillance camera, a tracking used in tracking, missile homing, etc., an infrared camera for object detection, etc., which uses an uncooled infrared sensor. Can be applied.

The image flow correction devices 100a, 100b, and 100c described above have a computer system inside. The processes of FIGS. 5, 7, and 9 described above are stored in a computer-readable recording medium in the form of a program, and the process is performed by the computer reading and executing this program. Here, the computer-readable recording medium refers to a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, or the like. Further, this computer program may be distributed to a computer via a communication line, and the computer receiving the distribution may execute the program.

A program for realizing the functions of each part in FIGS. 1, 6 and 8 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed. You may process each part according to. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. Further, the "computer system" shall also include a WWW system provided with a homepage providing environment (or display environment). Further, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, or a CD-ROM, or a storage device such as a hard disk built in a computer system. Furthermore, a "computer-readable recording medium" is a volatile memory (RAM) inside a computer system that serves as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, it shall include those that hold the program for a certain period of time.

Further, the program may be transmitted from a computer system in which this program is stored in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the "transmission medium" for transmitting a program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. Further, the above program may be for realizing a part of the above-mentioned functions. Further, a so-called difference file (difference program) may be used, which can realize the above-mentioned functions in combination with a program already recorded in the computer system.

The present invention can be applied to an image flow correction device, an image flow correction method, a program, etc., which need to simultaneously correct not only an image flow generated based on rotational motion but also an image flow generated based on translational motion. ..

10 ... Infrared image pickup device 11 ... Infrared detector 12 ... Processing circuit 13 ... Angle speed detector 20 ... Speed detector 30 ... Correction unit 31 ... Image memory 32 ... Image memory 33 ... Arithmetic unit 34 ... Subtractor 35 ... Divider 36 ... Adder 37 ... Conversion unit 50 ... Distance detector 51 ... Control unit 61 ... Conversion Unit 62 ... Correction unit 100a, 100b, 100c ... Image flow correction device

Claims (13)

A speed detector that detects the moving speed of the image pickup device,
The first movement of the object in the captured image is based on the angle formed by the moving speed detected by the speed detection unit between the predetermined direction in the captured image and the moving direction of the object in the captured image. A converter that converts to rotational angular velocity,
The combined rotational angular velocity is calculated by synthesizing the second rotational angular velocity of the object in the captured image and the first rotational angular velocity converted by the conversion unit, and is based on the combined rotational angular velocity. , A correction unit that corrects the image flow of the object,
Image flow correction device including. The image flow according to claim 1, wherein the correction unit obtains a reference pixel in the captured image by using the combined rotational angular velocity, and corrects the brightness of the pixel of the object by using the brightness of the reference pixel. Correction device. The correction unit determines the direction in which the image flow of the object is generated in the captured image based on the combined rotation angle velocity, and uses the direction to correct the image flow according to claim 1 or 2. The image flow correction device according to. The correction unit synthesizes the first rotation angular velocity converted based on the movement speed detected at a predetermined time and the second rotation angular velocity detected at the predetermined time, thereby. The image flow correction device according to any one of claims 1 to 3, which calculates the combined rotational angular velocity. The correction unit calculates the combined velocity by synthesizing the first rotational angular velocity and the second rotational angular velocity, and uses the combined velocity as the first angular velocity in the first direction in the captured image. The image flow correction device according to any one of claims 1 to 4, wherein the combined rotational angular velocity is calculated by decomposing the combined rotational angular velocity into a second angular velocity in a second direction orthogonal to the first direction. A speed detector that detects the moving speed of the image pickup device,
A conversion unit that decomposes the moving speed detected by the speed detection unit into a first angular velocity in the first direction in the captured image and a second angular velocity in the second direction orthogonal to the first direction. ,
The second rotational angular velocity of the object in the captured image is decomposed into the third angular velocity in the first direction and the fourth angular velocity in the second direction, and the first angular velocity and the third angular velocity are decomposed. The combined angular velocity is calculated by synthesizing the second angular velocity and the fourth angular velocity, and the image flow correction of the object is performed based on the combined rotational angular velocity. Correction part and
Image flow correction device including. A speed detector that detects the moving speed of the image pickup device,
A conversion unit that converts the moving speed detected by the speed detection unit into the first rotation angular velocity of the object in the captured image, and a conversion unit.
When the distance to the object in the captured image is less than a predetermined distance, the second rotation angular velocity of the object in the captured image and the first rotation angular velocity converted by the conversion unit. And, the combined rotation angular velocity is calculated, the image flow correction of the object is performed based on the combined rotation angular velocity, and when the distance is equal to or more than the predetermined distance, the image flow is corrected. A correction unit that does not correct and
Image flow correction device including. Detects the moving speed of the image pickup device and
The detected moving speed is converted into the first rotation angular velocity of the object in the captured image based on the angle formed by the predetermined direction in the captured image and the moving direction of the object in the captured image. ,
The combined rotational angular velocity is calculated by synthesizing the second rotational angular velocity of the object in the captured image and the converted first rotational angular velocity.
An image flow correction method for correcting the image flow of the object based on the combined rotational angular velocity. Detects the moving speed of the image pickup device and
The detected moving speed is decomposed into a first angular velocity in the first direction and a second angular velocity in the second direction orthogonal to the first direction in the captured image.
The second rotational angular velocity of the object in the captured image is decomposed into the third angular velocity in the first direction and the fourth angular velocity in the second direction, and the first angular velocity and the third angular velocity are decomposed. The combined angular velocity is calculated by synthesizing the second angular velocity and the fourth angular velocity, and the image flow correction of the object is performed based on the combined rotational angular velocity. Image flow correction method. Detects the moving speed of the image pickup device and
The detected moving speed is converted into the first rotation angular velocity of the object in the captured image.
When the distance to the object in the captured image is less than a predetermined distance, the second rotation angle velocity of the object in the captured image and the converted first rotation angle velocity are combined. By doing so, the combined rotation angle velocity is calculated, the image flow correction of the object is performed based on the combined rotation angle velocity, and when the distance is equal to or greater than the predetermined distance, the image flow correction is not performed. Image flow correction method. The computer of the image flow correction device,
A speed detecting means for detecting the moving speed of the image pickup device and
The first movement of the object in the captured image is based on the angle formed by the moving speed detected by the speed detecting means between a predetermined direction in the captured image and the moving direction of the object in the captured image. Conversion means to convert to rotational angular velocity,
The combined rotation angle velocity is calculated by synthesizing the second rotation angle velocity of the object in the captured image and the first rotation angle velocity converted by the conversion means, and based on the combined rotation angle velocity. , A correction means for correcting the image flow of the object,
A program that functions as. The computer of the image flow correction device,
Speed detecting means for detecting the moving speed of the image pickup device,
A conversion means for decomposing the moving speed detected by the speed detecting means into a first angular speed in the first direction and a second angular speed in the second direction orthogonal to the first direction in the captured image.
The second angular velocity of the Target contained in the captured image, and a third angular velocity in the first direction, to decompose the fourth angular in the second direction, the said first angular velocity the The combined rotational angular velocity is calculated by synthesizing the angular velocity of 3 and the second angular velocity and the fourth angular velocity, and the image flow correction of the object is performed based on the combined rotational angular velocity. Correction means to perform,
A program that functions as. The computer of the image flow correction device,
Speed detecting means for detecting the moving speed of the image pickup device,
A conversion means that converts the moving speed detected by the speed detecting means into a first rotation angle speed of an object in a captured image.
When the distance to the object in the captured image is less than a predetermined distance, the second rotation angle velocity of the object in the captured image and the first rotation angle velocity converted by the conversion means. And, the combined rotation angle velocity is calculated, the image flow correction of the object is performed based on the combined rotation angle velocity, and when the distance is equal to or more than the predetermined distance, the image flow is corrected. Correction means without correction,
A program that functions as.