JP5285487B2 - Image recording system and image recording method - Google Patents

Description

The present invention relates to an image recording system and an image recording method for recording three-dimensional shape information including a color image and a distance image of a subject that are continuously captured while moving at high speed.

Development of a device that continuously measures three-dimensional shapes over long distances is expected, such as damage inspection of road surfaces such as roads and bridges, wall surfaces of tunnels and buildings, and detection of obstacles. In addition, a method for displaying a three-dimensional shape at an arbitrary position in combination with the preceding and following three-dimensional shapes from such a vast amount of three-dimensional shape information continuously measured over a long distance is also expected.
As a conventional three-dimensional shape detection method for the purpose of inspecting a road surface or a tunnel wall surface, a method of measuring a three-dimensional shape of a road surface or a wall surface by irradiating a laser beam and scanning two-dimensionally is a typical representative method. Method. This laser beam scanning method takes time to measure a three-dimensional shape, and takes a lot of time to continuously measure the three-dimensional shape over a long distance. In addition, since there is almost no example of continuously measuring such a large amount of three-dimensional shape information, neither a recording method nor a display method is considered.

In recent years, a high-speed three-dimensional image sensor that combines light-emitting diode intensity-modulated light irradiation and a high-speed gain modulation television camera has been developed (Patent Document 1). According to this high-speed three-dimensional image sensor, it is possible to correct the color image and the distance image simultaneously captured based on the data of the zoom lens to obtain three-dimensional information with high speed and high accuracy. It has become possible to acquire three-dimensional shape information of a subject (Patent Document 2). Furthermore, by replacing light-emitting diode intensity-modulated light irradiation with ultrashort laser pulse light irradiation, it has become possible to continuously measure the three-dimensional shape of an object that moves at high speed, which has been impossible until now. It can be used in a wide range of industrial fields such as high-speed measurement of measured objects.

This high-speed three-dimensional image sensor is mounted on a car, and it is possible to measure three-dimensional shapes such as road surfaces, bridges, and wall surfaces continuously over a long distance at a TV frame rate while traveling. Color images and distance images can be obtained.

However, the color image and the distance image captured in this way are translated and rotated for each frame due to the running and shaking of the car, so that the situation of the three-dimensional shape over a plurality of frames can be accurately grasped. Is difficult.

JP 2008-249431 A JP 2008-249430 A

The present invention relates to a color image and a distance image continuously captured by a three-dimensional image sensor mounted on a traveling automobile (high-speed moving body), traveling (moving) speed, acceleration, and acceleration of the automobile (high-speed moving body) as auxiliary data. A method of detecting the translation and rotation applied to each frame from the angular velocity, a method of correcting the detected translation and rotation and combining a color image and a distance image between adjacent frames, a road surface, a bridge, It is an object of the present invention to provide a method for grasping a three-dimensional shape such as a wall surface and an apparatus therefor.

The present invention is an image recording system for recording an image continuously captured, a 3-dimensional image sensor for continuously capturing a color image and a distance image of the object at a predetermined frame rate while moving the movable body, wherein Identical by converting a sensor that detects acceleration, velocity, and angular velocity during movement of the moving body as auxiliary data, and a color image and a distance image obtained by the three-dimensional image sensor according to one of the coordinate systems. In the same coordinate system, the translation and rotation applied to each frame due to the movement and shaking of the moving body are corrected, and the corrected color image or distance image between adjacent frames is combined and recorded. possess a device, the recording apparatus, the relationship between the adjacent frame and the rotation matrix of the camera coordinate system with respect to the 3-dimensional image sensor translation vector The coordinate system is transformed using the rotation matrix and the translation vector, and the rotation matrix and the translation vector are distances of a three-dimensional shape of a distance image of an overlapping region between the adjacent frames. Are evaluated in the same coordinate system, and the rotation matrix and the translation vector are obtained so that the distance to the overlapping region of the plane image between the color image or the distance image of the adjacent frame estimated from the auxiliary data obtained from the sensor is minimized. It is characterized by that.
Further, the present invention combines a color image and a distance image continuously captured at a high speed frame rate for a long time with a three-dimensional image sensor mounted on a moving automobile (high-speed moving body), for example. traveling body) (moving) speed, acceleration, and method of recording a color image and the distance image data and recording in synchronization with angular velocity to the frame signal as auxiliary data, said road, bridges, color, such as a wall The data format that combines the movement amount data of the camera coordinate system between adjacent frames, the color image, and the distance image, and the color image and the distance image are combined. A method for creating data for combining a color image and a distance image, wherein only a distance image between frames is used, and A method for generating data for combining a color image and a distance image, wherein the auxiliary image is used to combine a color image and a distance image to combine the color image and the distance image; and In order to combine a color image and a distance image, the method includes a method of combining a color image and a distance image using the combined data, and an apparatus incorporating these methods is also included.

Specifically, image processing means for estimating the translation and rotation of the camera coordinate system between adjacent frames by evaluating and minimizing the distance between the distance images between adjacent frames is used. By using the auxiliary data together, it is also possible to efficiently perform image processing that minimizes the distance between the distance images. The camera coordinate system between adjacent frames is converted using the estimated translation and rotation, and means for combining the color image and the distance image is used.

According to the present invention, it is possible to grasp a three-dimensional shape such as a road surface such as a road and a bridge extending from a short distance to a long distance, a ground surface, a tunnel, or a wall surface of a building.

FIG. 1 is a diagram showing the configuration of a color image and distance image recording system for road surfaces, bridges, wall surfaces, and the like. Example 1 FIG. 2 is an image diagram of a color image and a distance image that are captured and recorded. Example 1 FIG. 3 is a diagram of the coordinate system and distance image (rotation and translation) of each frame. (Example 2) FIG. 4 is a diagram of a distance image (overlapping portion between adjacent frame distance images) of the overlapping area S between the coordinate system of each frame and the adjacent frame. (Example 2)

The outline of the recording system of the road surface / wall surface distance image data according to the present invention will be described with reference to FIG. The three-dimensional image sensor 10, the sensor 11, and the color image / distance image / auxiliary data recording device 12 are mounted on an automobile (moving body), and color images, distance images, and auxiliary data (road surface, bridge, wall surface, etc.) while traveling The speed (acceleration, angular velocity) of the automobile (moving body) is continuously recorded in the color image / distance image / auxiliary data recording device 12 at the frame rate.

The data recorded in the distance image / auxiliary data recording device 12 is a combination of the following data.
Color images for each frame C ₀ , C ₁ , C ₂ , ... C _i-1 , C _i , C _{i + 1} , ...
Distance images for each frame D ₀ , D ₁ , D ₂ , ... D _i-1 , D _i , D _{i + 1} , ...
Speed for each frame… (v _x ) _i-1 , (v _x ) _i , (v _x ) _{i + 1} ,…
Three-axis acceleration for each frame… (a _x , a _y , a _z ) _i−1 , (a _x , a _y , a _z ) _i , (a _x , a _y , a _z ) _{i + 1} ,…
Triaxial angular velocity for each frame ... (w _x , w _y , w _z ) _i-1 , (w _x , w _y , w _z ) _i , (w _x , w _y , w _z ) _{i + 1} , ...

The distance image captured by the running and shaking of the automobile (moving body) is translated and rotated for each frame, and the images of the color image 13-2 and the distance image 14-2 captured and recorded are as shown in FIG. .

As shown in FIG. 3, the distance images between adjacent frames captured with the lens parameters of the three-dimensional image sensor fixed are respectively represented by D _i-1 (X _i-1 , Y _{i- 1} ), D _i (X _i , Y _i ) of the current (i) frame and D _{i + 1} (X _{i + 1} , Y _{i + 1} ) of the next (i + 1) frame. (O _i-1 X _i-1 Y _i-1 Z _i-1 ) coordinate system, (O _i X _i Y _i Z _i ) coordinate system and (O _{i + 1} X _{i + 1} Y _{i + 1} Z _{i +) 1} ) The position vectors u _i−1 , u _i and u _{i + 1} representing the three-dimensional shape of the distance image in the coordinate system can be expressed as follows.


Incidentally, each similarly color image between adjacent frames, prior _{(i-1) C i-} 1 of the frame _{(X i-1, Y i} -1), C i (X i of the current (i) frames, Y _i ) and C _{i + 1} (X _{i + 1} , Y _{i + 1} ) of the next (i + 1) frame.
The camera coordinate system between adjacent frames (for example, the (O _i-1 X _i-1 Y _i-1 Z _i-1 ) coordinate system of the previous (i-1) frame and the current (i) frame and the (O _i X _i The relationship of the Y _i Z _i ) coordinate system can be expressed by equation (1) using the rotation matrix R _i and the translation vector T _i .

(1)
However,
,

By calculating the rotation matrix R _i and the translation vector T _i , the (O _i-1 X _i-1 Y _i-1 Z _i-1 ) coordinate system is changed to the (O _i X _i Y _i Z _i ) coordinate system, or Conversely, it can be converted. Therefore, by converting the color image and the distance image between adjacent frames according to one of the coordinate systems, it is possible to handle them in the same coordinate system. That is, it is possible to combine the distance images between adjacent frames by correcting the translation and rotation applied to each frame. Similarly, color images between adjacent frames can be combined.

Therefore, the data format for enabling the color images C _i-1 and C _i and the distance images D _i-1 and D _i between adjacent frames to be combined is a color image and a distance image as shown in equation (2). And a camera coordinate system movement amount between adjacent frames (rotation matrix R _i and translation vector T _i ) are arranged in frame order.
..., (C _i-1 , D _i-1 , R _i-1 , T _i-1 ), (C _i , D _i , R _i , T _i ), (C _{i + 1} , D _{i + 1} , R _{i + 1} , T _{i + 1} ),… (2)
As a specific example, binding before the (i-1) the distance of the frame image _{D i-1 (X i-} 1, Y i-1) and current (i) the frame of the distance image _{D i (X i, Y i} ) First, coordinates are transformed by equation (1), and a position vector u _i representing the three-dimensional shape of the distance image D _i (X _i , Y _i ) of the current (i) frame is converted to the previous (i-1) frame. In the (O _i-1 X _i-1 Y _i-1 Z _i-1 ) coordinate system, the following expression is used.


Since u _i-1 and u _i = R _i u _i-1 + T _i are expressed in a common (O _i-1 X _i-1 Y _i-1 Z _i-1 ) coordinate system, these two positions By superimposing the vectors, D _i-1 (X _i-1 , Y _i-1 ) and D _i (X _i , Y _i ) can be combined. Conversely, by converting u _i-1 to a distance image in the (O _i X _i Y _i Z _i ) coordinate system and superimposing it with u _i , D _i-1 (X _{i- 1} , Y _i-1 ) and D _i (X _i , Y _i ) can be combined.

In order to obtain the rotation matrix R _i and the translation vector T _i of the coordinate system between adjacent frames, the distance image D ′ _i−1 of the overlapping region S of the previous (i−1) frame and the current ((i)) frame The distance d between the three-dimensional shapes u _i−1 and u _i of D ′ _i is evaluated in the same coordinate system, and R _i and T _i that minimize this distance d may be obtained.
That is, in the (O _i-1 X _i-1 Y _i-1 Z _i-1 ) coordinate system

(3)
Or in (O _i X _i Y _i Z _i ) coordinate system
(4)
The rotation matrix R _i and the translation vector T _i in equation (1) are determined so that the distance d expressed by
For example, as one calculation method, the distance image D ′ of the overlap area S of the X _i-1 Y _i-1 plane or the X _i Y _i plane between the distance images of adjacent frames estimated from the traveling speed (v _x ) _i , _{For i-1} (X _i-1 , Y _i-1 ) and D ' _i (X _i , Y _i ), the distance d expressed by equation (3) or (4) is minimized (1 ) Of the rotation matrix R _i and the translation vector T _i , the Iterated Closest Point (ICP) algorithm (PJ Best, et al., IEEE Trans., Pattern analysis and machine intelligence, vol.14, No. 2, pp.239-256).

Since the rotation matrix R _i and the translation vector T _i obtained by iterative calculation of the ICP algorithm are not necessarily guaranteed to be global optimal solutions, they converge to the local solution and translate to the incorrect rotation matrix R _i The vector T _i may be required. Therefore, using the auxiliary data velocity (v _x ) _i , triaxial acceleration (a _x , a _y , a _z ) _i and triaxial angular velocity (w _x , w _y , w _z ) _i in combination, the direction of rotation and translation And the accuracy of the required rotation matrix R _i and translation vector T _i are improved.

In order to reduce the amount of calculation, the X and Y coordinates are slightly reduced during stable running when the triaxial acceleration (a _x , a _y , a _z ) _i and the triaxial angular velocity (w _x , w _y , w _z ) _i are small. The amount of parallel movement is applied by applying a simple pattern matching method in the x and y directions to find the amount of change in X and Y that minimizes the distance d between the distance images of the overlapping portion S of adjacent frames by changing each Calculate (DX, DY) and apply R _i = I, T _i = (DX, DY, 0).

By using the data in the form of (2) and the coordinate transformation of (1), the distance images between adjacent frames can be combined by developing and superimposing the distance images between adjacent frames in the same coordinate system. . By sequentially combining the distance images in the order of frames, it is possible to grasp a three-dimensional shape such as a road surface, a bridge, and a wall surface over a plurality of frames.

Similarly, by using the data in the form of equation (2) and the coordinate transformation of equation (1) for color images, the color images between adjacent frames are developed in the same coordinate system and superimposed. Color images between frames can be combined. Then, by superimposing the combined color image on the distance image combined by the above method, it is possible to add a texture to the combined distance image.

According to the present invention, it is possible to grasp a three-dimensional shape such as a road surface of a long distance, a road surface of a bridge, and a tunnel wall surface.

10 three-dimensional image sensor 11 sensor 12 color image / distance image / auxiliary data recording device 13 color image data C (x, y)
14 Distance image data D (x, y)
15 Auxiliary (acceleration, velocity, angular velocity) data ((a _x , a _y , a _z ), (v _x ), (w _x , w _y , w _z )

Claims (4)

In an image recording system for recording continuously captured images,
A three-dimensional image sensor that continuously captures a color image and a distance image of a subject at a predetermined frame rate while being moved by a moving body;
A sensor for detecting acceleration, velocity, and angular velocity during movement of the moving body as auxiliary data;
The color image obtained by the three-dimensional image sensor and the distance image are converted to match one of the coordinate systems to be the same coordinate system, and each frame is moved by the movement and shaking of the moving body in the same coordinate system. took corrected translation and rotation, have a recording device and for recording by combining a color image or distance image between corrected adjacent frames,
The recording apparatus represents a relationship between the adjacent frames using a rotation matrix and a translation vector of a camera coordinate system for the three-dimensional image sensor, and converts the coordinate system using the rotation matrix and the translation vector. And
The rotation matrix and the translation vector are obtained by evaluating the distance of the three-dimensional shape of the distance image of the overlapping region between the adjacent frames in the same coordinate system, and the color image of the adjacent frame estimated from the auxiliary data obtained from the sensor or An image recording system characterized in that the rotation matrix and the translation vector are obtained so that a distance to a planar overlapping region between distance images is minimized . The recording device comprises:
Based on the auxiliary data obtained the sensors or al, and wherein the translating and obtains the direction and magnitude of the rotation, obtained in correspondence with the direction and magnitude of obtaining the translation vector and the rotation matrix The image recording system according to claim 1 . The recording device comprises:
The image recording system according to claim 1 or 2, characterized in that recorded by synchronizing the auxiliary data in the color image and the frame signal of the range image obtained from the 3-dimensional image sensor. In an image recording method for recording continuously captured images,
Taking a color image and a distance image of a subject continuously at a predetermined frame rate while moving a 3D image sensor by a moving body,
Detecting acceleration, velocity, and angular velocity during movement of the moving body from the sensor as auxiliary data;
The color image obtained by the three-dimensional image sensor and the distance image are converted to match one of the coordinate systems to be the same coordinate system, and each frame is moved by the movement and shaking of the moving body in the same coordinate system. The translation and rotation applied to the image are corrected, and the corrected color image or distance image between adjacent frames is combined and recorded on the recording device.
The recording apparatus represents a relationship between the adjacent frames using a rotation matrix and a translation vector of a camera coordinate system for the three-dimensional image sensor, and converts the coordinate system using the rotation matrix and the translation vector. And
The rotation matrix and the translation vector are obtained by evaluating the distance of the three-dimensional shape of the distance image of the overlapping region between the adjacent frames in the same coordinate system and estimating the color image of the adjacent frame estimated from auxiliary data obtained from the sensor or The image recording method, wherein the rotation matrix and the translation vector are such that the distance to the overlapping region of the planes between the distance images is minimized .