JP2021026622A - Image processing apparatus - Google Patents

Abstract

To provide an image processing apparatus with improved accuracy of correlating a camera image and a dot group.SOLUTION: An image processing apparatus generating a fusion image by using a camera image picked up with a camera capable of keeping a shutter open at all times and a dot group acquired by a rider, includes: an estimation part for estimating, on the basis of information obtained by the camera and a clock time of the rider measuring, a camera image at the clock time of the rider measuring; an intermediate generation part for generating plural intermediate fusion images on the basis of a dot group corresponding to a resolution of the rider and the camera image at the clock time of the rider measuring; and a synthesis part for generating a fusion image by synthesizing the plural intermediate fusion images.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to an image processing apparatus.

Patent Document 1 discloses a device that combines information of a plurality of types of sensors by sensor fusion. This device uses a lidar (LIDAR: Laser Imaging Detection and Ranging) and a camera as sensors.

JP-A-2018-048949

As an example of sensor fusion, it is conceivable to generate a fusion image by using a camera image captured by a camera and a point cloud generated by a rider. However, the measurement cycle (frame rate) of the camera and the measurement cycle of the rider are different. Furthermore, the camera is a two-dimensional measurement, while the rider is a three-dimensional measurement. Therefore, not only the measurement timing but also the measurement range is different between the two. Therefore, it is difficult to generate an appropriate fusion image by combining both.

The present disclosure provides an image processing device with improved accuracy of association between a camera image and a point cloud.

One form of the present disclosure is an image processing device that generates a fusion image by using a camera image taken by a camera capable of constantly opening the shutter and a point group acquired by a rider, and is acquired by the camera. Multiple intermediate fusions based on an estimater that estimates the camera image at the rider's measurement time based on the information and the rider's measurement time, and a group of points corresponding to the rider's resolution and the camera image at the rider's measurement time. It includes an intermediate generation unit that generates an image and a composition unit that synthesizes a plurality of intermediate fusion images to generate a fusion image.

According to the present disclosure, it is possible to improve the accuracy of associating a camera image with a point cloud.

It is a block diagram explaining the configuration outline of the image processing apparatus which concerns on embodiment. It is a figure explaining the sensor installation position, the measurement range of a camera, and the measurement range of a rider. It is a figure explaining the measurement time of a camera and the measurement time of a rider. It is a figure explaining the integration of the measurement result of a camera and a rider for a stationary object. It is a figure explaining the measurement result of a camera and a rider for a moving body. It is a figure explaining the flow of generating an intermediate fusion image from the measurement result of a camera and a rider, and generating a fusion image. It is a flowchart which shows the image generation operation of arbitrary time. It is a flowchart which shows the generation operation of a fusion image.

Hereinafter, various exemplary embodiments will be described. In the following description, the same or equivalent elements are designated by the same reference numerals, and duplicate description will not be repeated.

(Configuration of image processing device)
FIG. 1 is a block diagram illustrating an outline of a configuration of an image processing apparatus according to an embodiment. As shown in FIG. 1, the image processing device 1 is a device mounted on a vehicle 2 and generates a fusion image in which a camera image which is an image taken by a camera and a point cloud acquired by a rider are integrated. is there.

The image processing device 1 includes an ECU (Electronic Control Unit) 10, a time correlation image sensor 11, a rider 12, and a clock 13. The ECU is an electronic control unit having a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), a CAN (Controller Area Network) communication circuit, and the like. The ECU 10 is connected to the time correlation image sensor 11 and the rider 12. The time correlation image sensor 11 and the rider 12 receive the time t from the clock 13 and synchronize their internal times.

ｆ（ｘ，ｙ，ｔ）は時刻ｔにおける画素（ｘ，ｙ）の明度であり、ｖは画素（ｘ，ｙ）の速度である。 The time correlation image sensor 11 is an example of a camera capable of constantly opening the shutter. Opening the shutter all the time means that there is no time gap in the exposure time and the light intensity can be measured at all times. One pixel of the imager of the time-correlated image sensor 11 has a configuration of one light diode and a configuration in which the electric charge generated from the light diode is distributed to three capacitors according to a reference signal from the outside. The reference signal is generally three-phase alternating current. The photocurrent for each pixel that produces an image follows Equation 1 below.

f (x, y, t) is the brightness of the pixel (x, y) at time t, and v is the velocity of the pixel (x, y).

数式（２）に示されるように、画像は、明度ｆ（ｘ，ｙ，ｔ）に複素数の参照信号e^-inΔ _ω ^ｔを乗じて１フレーム時間の積分を演算した値となる。撮像された画像ｇ（ｘ，ｙ）は以下の数式（３）を満たすとする。

数式（３）の第二項は積分境界値である。数式（３）は連立方程式をなしているため、例えば２つの画像ｇ_０（ｘ，ｙ）、ｇ_１（ｘ，ｙ）を用いて連立方程式を解くことで、積分境界値を消去することができる。時間相関イメージセンサは、実部のみからなる強度画像ｇ_０（ｘ，ｙ）と、複素数の相関画像ｇ_ｎ（ｘ，ｙ）の実部と虚部を出力することができる。このため、センサ検出結果を数式（３）の連立方程式に代入して解くことで各画素における速度ｖ、つまりオプティカルフローを得ることができる。時間相関イメージセンサ１１は、上述した強度画像、相関画像、及びフレーム端点時刻を出力する。以下では、時間相関イメージセンサ１１により出力される画像又は画素をカメラ画像という。 The imaging time (shutter opening time) for acquiring an image of one frame is T, and the image g (x, y) is expressed by the following mathematical formula (2).

As shown in the formula (2), the image is a value obtained by multiplying the brightness f (x, y, t) by the complex reference signal e ^{-in Δ} _ω ^t and calculating the integral for one frame time. It is assumed that the captured image g (x, y) satisfies the following mathematical formula (3).

The second term of equation (3) is the integration boundary value. Since the equation (3) is a simultaneous equation, it is possible to eliminate the integration boundary value by solving the simultaneous equations using _{, for example, two images g 0} (x, y) and g _{1 (x, y).} it can. The time correlation image sensor can output the real part and the imaginary part of the _{intensity image g 0} (x, y) consisting only of the real part and the complex number correlation image g _{n (x, y).} Therefore, the velocity v at each pixel, that is, the optical flow can be obtained by substituting the sensor detection result into the simultaneous equations of the equation (3) and solving the equation. The time correlation image sensor 11 outputs the above-mentioned intensity image, correlation image, and frame endpoint time. Hereinafter, the image or pixel output by the time correlation image sensor 11 is referred to as a camera image.

The rider 12 is a detection device that detects an object around the vehicle 2 by using light. The rider 12 transmits light to the periphery of the vehicle 2 and detects the object by receiving the light reflected by the object. The rider 12 outputs a rider measurement signal. The rider measurement signal includes a distance, a direction, and a measurement time tL.

FIG. 2 is a diagram illustrating a sensor installation position, a camera measurement range, and a rider measurement range. As shown in FIG. 2A, the vehicle 2 is provided with the time correlation image sensor 11 and the rider 12 at different positions, and the measurement origin positions are different. Further, the time correlation image sensor 11 measures the measurement range R1 which is a fixed range in front of the vehicle 2. On the other hand, the rider 12 performs measurement while rotating the measurement range R2, which is a very narrow range (arrow in the figure). Therefore, as shown in FIG. 2B, the fusion image can be generated only when the measurement range R2 of the rider 12 overlaps with the measurement range R1 of the time correlation image sensor 11.

FIG. 3 is a diagram for explaining the measurement time of the camera and the measurement time of the rider. As shown in FIG. 3, the time correlation image sensor 11 can acquire information about an image at a frame period T1 interval (1/30 sec), and there is no time gap in the exposure time. Therefore, the measurement time T2 of the rider 12 always overlaps with the measurement time of the time correlation image sensor 11.

Returning to FIG. 1, the ECU 10 includes a time calculation unit 14, a calibration information storage unit 15, an estimation unit 16, an intermediate generation unit 17, and a synthesis unit 18.

The time calculation unit 14 calculates the corresponding pixel based on the rider measurement signal of the rider 12 and the calibration information stored in the calibration information storage unit 15. The calibration information storage unit 15 is a storage medium that stores calibration information in advance. The calibration information is information that corrects the spatial difference between the time correlation image sensor 11 and the rider 12, and is acquired in advance. As shown in FIG. 2A, the vehicle 2 is provided with the time correlation image sensor 11 and the rider 12 at different positions, and the calibration information is provided by the time correlation image sensor 11 and the rider 12. This is information for correcting the difference in the measurement origin position.

FIG. 4 is a diagram illustrating integration of measurement results between a camera and a rider for a stationary object. As shown in FIG. 4, there is a spatial deviation between the measurement result D1 of the time correlation image sensor 11 and the measurement result D2 of the rider 12. The time calculation unit 14 aligns the measurement result D1 and the measurement result D2 (coordinate conversion, etc.) using the calibration information, and positions the pixels corresponding to the distance and direction measured by the rider measurement signal (time correlation). The position of the image sensor 11 on the image) is calculated. As a result, the measurement result D1 of the time correlation image sensor 11 and the measurement result D2 of the rider 12 are associated with each other.

The estimation unit 16 estimates the camera image at the measurement time tL of the rider 12 based on the information acquired by the time correlation image sensor 11 and the measurement time tL of the rider 12. Specifically, the estimation unit 16 reproduces the brightness value of the corresponding pixel at the measurement time tL based on the corresponding pixel and the measurement time tL calculated by the time calculation unit 14. As a time correlation imaging technique, a method of reproducing an arbitrary image within one frame time is known. Although the formula (3) holds for any n> = 0, only g ₀ (x, y) and g ₁ (x, y) can be obtained by observation. From these two images, the velocity vector v and the integration boundary value (integration boundary value image F ₀ (x, y)) which is the second term of the mathematical formula (3) can be obtained. Therefore, the mathematical formula (3) for n> = 2 _{is regarded as a partial differential equation having a known coefficient vector v and a known driving term F 0} (x, y), and this is mathematically solved to obtain a higher-order Fourier. The coefficient image g _n (x, y) is estimated. Thereby, the brightness value at the measurement time tL of the corresponding pixel can be reproduced.

The above description is for the case where the measurement target is a stationary object. When the measurement target is a moving body, the measurement result D1 of the time correlation image sensor 11 and the measurement result D2 of the rider 12 change. FIG. 5 is a diagram for explaining the measurement results of the camera and the rider for the moving body. As shown in FIG. 5, the measurement result D1 of the time correlation image sensor 11 causes a time lag when a moving object is targeted, so that blurring (so-called ghost) occurs. Further, the measurement result of the rider 12 is displaced by the amount of movement in the scanning direction, so that the arrangement of the point cloud is distorted.

In order to eliminate the deviation caused by such a moving body, the correspondence between the measurement result D1 of the time correlation image sensor 11 and the measurement result D2 of the rider 12 is devised. Specifically, the deviation caused by the moving body is alleviated by associating with the resolution unit of the rider 12.

The intermediate generation unit 17 generates a plurality of intermediate fusion images based on the point cloud corresponding to the resolution of the rider 12 and the camera image at the measurement time of the rider 12. FIG. 6 is a diagram illustrating a flow of generating an intermediate fusion image from the measurement results of the camera and the rider to generate the fusion image. As shown in FIG. 6A, the measurement result D2 of the rider 12 is a vertically arranged point cloud, and the vertically one row of point clouds is the resolution of the rider. A camera image at the measurement time tL of the point cloud is estimated and associated with each point cloud according to the resolution of the rider. Subsequently, as shown in FIG. 6B, at each measurement time tL, the intermediate generation unit 17 obtains the image GD2 corresponding to the point cloud corresponding to the resolution of the rider from the measurement result D1 of the time correlation image sensor 11. break the ice. As a result, the image GD2 corresponding to the resolution of the rider, that is, the intermediate fusion image GD2 can be obtained in chronological order.

The synthesizing unit 18 synthesizes a plurality of intermediate fusion images GD2 to generate a fusion image F. As shown in FIG. 6C, the compositing unit 18 aligns and integrates a plurality of intermediate fusion images GD2. The compositing unit 18 generates the fusion image F according to, for example, the narrower of the angle of view of the time correlation image sensor 11 and the angle of view of the rider 12.

(Operation of image processing device)
FIG. 7 is a flowchart showing an image generation operation at an arbitrary time. The flowchart shown in FIG. 7 is executed at the timing when the user receives the operation of instructing the generation of the fusion image.

First, the time correlation image sensor 11 generates an intensity image g ₀ (x, y, t) and a correlation image g ₁ (x, y, t) (step S10). Next, the estimation unit 16 calculates the optical flow field v (x, y) by combining the mathematical formula (3) with n = 0 and the mathematical formula (3) with n = 1 (step S12). Subsequently, the estimation unit 16 _{substitutes the intensity image g 0} (x, y, t), the correlation image g ₁ (x, y, t) and the optical flow field v (x, y) into the equation (3). _{The integration boundary value (integration boundary value image F 0} (x, y)), which is the second term of the mathematical formula (3), is reproduced (step S14). Then, the estimation unit 16 regards the equation (3) relating to n> = 2 _{as a partial differential equation having a known coefficient vector v and a known driving term F 0} (x, y), and mathematically solves this. , Higher-order Fourier coefficient image g _n (x, y) is estimated (step S16). When step S16 ends, the flowchart shown in FIG. 7 ends.

After the flowchart is completed, the image processing device 1 determines whether or not the user has accepted the end operation, and if the user has not accepted the end operation, the image processing device 1 executes the flowchart shown in FIG. 7 from the beginning. By executing the flowchart shown in FIG. 7, the luminance function Y (x, y, t) at the time t of the pixel (x, y) can be obtained.

FIG. 8 is a flowchart showing a fusion image generation operation. The flowchart shown in FIG. 8 is executed at the timing when the user receives the operation of instructing the generation of the fusion image.

As shown in FIG. 8, the time calculation unit 14 calculates the corresponding pixel based on the rider measurement signal of the rider 12 and the calibration information stored in the calibration information storage unit 15 (step S20). The time calculation unit 14 calculates the pixels (x, y) of the time correlation image sensor 11 corresponding to the measurement direction of the rider 12.

Subsequently, the estimation unit 16 converts the measurement time tL into the time within the frame of the time correlation image sensor 11 in order to calculate the luminance value at the measurement time tL of the rider 12 obtained in step S20 (step S22). The in-frame time is a time with the frame start time as 0. The estimation unit 16 converts the measurement time tL into the in-frame time C (x, y) based on the frame end point time output by the time correlation image sensor 11.

Subsequently, the estimation unit 16 uses the luminance function Y (x, y, t) acquired by the flowchart of FIG. 7 to obtain the luminance value Y (x, y, C) at the time C (x, y) in the frame. x, y)) is calculated (step S24). As a result, the brightness value at the measurement time tL can be obtained.

Subsequently, the intermediate generation unit 17 performs a plurality of intermediate fusions based on the point cloud corresponding to the resolution of the rider 12 and the brightness value Y (x, y, C (x, y)) at the measurement time tL of the rider 12. An image is generated (step S26). As shown in FIG. 6B, the intermediate fusion image GD2 corresponding to the point cloud is cut out from the camera image.

Subsequently, the synthesis unit 18 synthesizes a plurality of intermediate fusion images GD2 to generate a fusion image F (step S28). As shown in FIG. 6C, the compositing unit 18 obtains one fusion image F by aligning and integrating a plurality of intermediate fusion images GD2. When step S28 ends, the flowchart shown in FIG. 8 ends.

After the flowchart is completed, the image processing device 1 determines whether or not the user has accepted the end operation, and if the user has not accepted the end operation, the image processing device 1 executes the flowchart shown in FIG. 8 from the beginning. By executing the flowchart shown in FIG. 8, one fusion image F can be obtained.

(Summary of Embodiment)
According to the image processing device 1, the camera image of the time correlation image sensor 11 at the measurement time tL of the rider 12 is estimated based on the information acquired by the time correlation image sensor 11 and the measurement time tL of the rider 12. Then, a plurality of intermediate fusion images are generated based on the point cloud (measurement result D2) corresponding to the resolution of the rider 12 and the camera image at the measurement time tL of the rider 12. Then, a plurality of intermediate fusion images GD2 are combined to generate a fusion image F. In this way, the time-correlation image sensor 11 is used to estimate the image according to the measurement time tL of the rider 12, and the images are sequentially cut out according to the resolution of the rider 12 to generate an intermediate fusion image, and a plurality of intermediate fusion images are generated. By integrating the above to generate the final fusion image, it is possible to alleviate the temporal and spatial deviation between the sensors. Therefore, according to the image processing device 1, the accuracy of associating the camera image with the point cloud can be improved.

The present invention is not limited to the above embodiment. The present invention can be modified in various ways without departing from the gist thereof.

Although an example of adopting the time correlation image sensor 11 as a camera has been described, the present disclosure is not limited to the time correlation image sensor 11, and a CMOS camera capable of constantly opening the shutter may be used. When a CMOS camera is adopted, the brightness of the frame start time (for example, the brightness of the image of the frame immediately before the current time) and the brightness of the frame end time (the brightness of the image of the current frame) are linearly interpolated. , The image in the frame according to the measurement time tL of the rider 12 can be reproduced.

1 ... image processing device, 12 ... rider, 16 ... estimation unit, 17 ... intermediate generation unit, 18 ... synthesis unit, GD2 ... intermediate fusion image, F ... fusion image, tL ... measurement time.

Claims (1)

An image processing device that generates a fusion image using a camera image taken by a camera that can always open the shutter and a group of points acquired by the rider.
An estimation unit that estimates the camera image at the measurement time of the rider based on the information acquired by the camera and the measurement time of the rider.
An intermediate generation unit that generates a plurality of intermediate fusion images based on the point cloud corresponding to the resolution of the rider and the camera image at the measurement time of the rider.
A compositing unit that synthesizes the plurality of intermediate fusion images to generate the fusion image, and
An image processing device comprising.

Images