JP2020035158A - Attitude estimation device and calibration system - Google Patents

Abstract

To provide an attitude estimation device for estimating attitude information of an imaging camera from three-dimensional coordinate information using an SLAM technology.SOLUTION: According to the present invention, an attitude estimation device 100 includes: an imaging camera 110 for imaging a vehicle periphery; an SLAM processing unit 210 for converting image data from the imaging camera 110 into three-dimensional coordinate information; an attitude estimation unit 220 for estimating an attitude of the imaging camera on the basis of the three-dimensional coordinate information; and a calibration unit 230 for performing calibration of the imaging camera on the basis of an estimation result of the attitude estimation unit 220.SELECTED DRAWING: Figure 2

Description

  The present invention relates to calibration of an imaging camera mounted on a vehicle, and more particularly, to estimation and calibration of the orientation of an imaging camera using an image processing technique such as SLAM (Simultaneous Localization and Mapping).

  2. Description of the Related Art In recent years, an imaging camera is often attached to a vehicle such as an automobile. In-vehicle cameras include cameras that image the inside of a vehicle and cameras that image the front, rear, and sides of the vehicle. The image captured by the on-board camera is displayed on a display to provide a front image, a rear image, a side image, an overhead image, etc. of the own vehicle, and is used for detecting a vehicle, an obstacle, and the like around the own vehicle. You. As described above, the in-vehicle camera has been increasingly recognized as being indispensable to the in-vehicle apparatus from the conventional standpoint that it can be mounted as an option.

  With the mounting of on-vehicle cameras becoming more common, there is a growing demand for the accuracy of images captured by on-vehicle cameras. As a technique for calibrating an error of an in-vehicle camera, Patent Literature 1 discloses a calibration device for a camera mounted on a vehicle. The calibration device can calibrate the internal parameters of the camera by extracting feature points from a plurality of captured images and analyzing the locus of the feature points.

  In addition, SLAM technology is used as a means for estimating the self-position. The SLAM technique is a technique that is performed simultaneously with self-position estimation and environment map creation, and is used for a robot or the like that performs independent traveling. As a method of grasping the self-position, the GPS technology is generally used, but in an environment such as an indoor space or a tunnel, the accuracy may be reduced. Even in such a situation, the SLAM technique can perform the self-position estimation. In particular, the one that performs the self-position estimation using the image information of the camera is referred to as a visual SLAM. Patent Literature 2 and Patent Literature 3 disclose technologies for performing traveling control and obtaining the attitude of a camera using the SLAM technology.

JP-A-2017-139600 JP 2013-045298 A JP 2014-032666 A

  FIGS. 8A and 8B show a conventional on-vehicle camera calibration method. For example, a plurality of calibration sheets 3, 4, and 5 are arranged in front of the vehicle 1, and a plurality of markers on the sheets 3, 4, and 5 are imaged by a vehicle-mounted camera 2 attached to the vehicle 1. The captured image is provided to the calibration device, where the attitude of the in-vehicle camera 2 is determined, and the in-vehicle camera 2 or the image data is calibrated.

  However, the conventional calibration method requires preparing a plurality of seats 3 to 5, and requires a large space for arranging the plurality of seats 3 to 5 around the vehicle. It is desired that the road surface 6 on which to 5 are arranged is flat. For this reason, in the conventional calibration, it is necessary to secure a flat and wide area, and the time required for the calibration has been long.

  An object of the present invention is to solve such a conventional problem, and an object of the present invention is to provide a posture estimation device and a calibration system that can efficiently estimate the posture of an imaging camera and perform calibration. .

  A posture estimating apparatus according to the present invention includes: an imaging camera attached to a moving object; a conversion unit that receives imaging data captured by the imaging camera and converts the imaging data into three-dimensional coordinate information; First calculating means for calculating road surface information from the information, second calculating means for calculating movement information of the imaging camera from the three-dimensional coordinate information, and an attitude of the imaging camera based on the road surface information and the movement information Estimating means for estimating.

In one embodiment, the conversion unit generates the three-dimensional coordinate information based on a plurality of frame images received from the imaging camera, and the second calculation unit calculates a point group included in the three-dimensional coordinate information. The road surface information is calculated by the processing. In one embodiment, the estimating unit estimates the attitude of the imaging camera by calculating the following equation (1). Here, Vx, Vy, and Vz are movement vectors of the imaging camera, a, b, and c are coefficients of a plane equation of a road surface calculated by the first calculation unit, and R is a rotation matrix of the imaging camera. Is the inverse matrix.

  In one embodiment, the posture estimating device further includes a calibrating means for calibrating the image data based on the posture of the imaging camera estimated by the estimating means. In one embodiment, the imaging camera includes a plurality of imaging cameras, the estimating unit estimates a position of each of the imaging cameras, and the configuration unit calibrates imaging data between the plurality of imaging cameras.

  A calibration system according to the present invention includes an imaging camera attached to a moving body, an in-vehicle device including a processing unit that processes imaging data captured by the imaging camera, and a computer device that is separated from the in-vehicle device. An acquisition unit for acquiring image data captured by the imaging camera; a conversion unit configured to convert the image data acquired by the acquisition unit into three-dimensional coordinate information; First calculating means for calculating road surface information, second calculating means for calculating movement information of the imaging camera from the three-dimensional coordinate information, and estimating the attitude of the imaging camera based on the road surface information and the movement information Estimating means, and calibrating means for calibrating imaging data based on the attitude of the imaging camera estimated by the estimating means. To.

  In one embodiment, the acquisition unit acquires the imaging data from the in-vehicle device by wire or wirelessly. In one embodiment, the calibrating means calibrate the processing means of the on-vehicle device.

  According to the present invention, since the posture of the imaging camera is estimated using the three-dimensional coordinate information of the imaging data captured by the imaging camera, a calibration sheet or a flat and wide space is required as in the related art. Therefore, the posture and the calibration of the imaging camera can be efficiently performed.

It is a figure explaining the precondition of a posture estimating device concerning an example of the present invention. It is a block diagram showing composition of a posture estimating device concerning an example of the present invention. It is an illustration of an imaging range of an imaging camera. FIG. 3 is a diagram illustrating a functional configuration of a posture estimating unit according to the embodiment. It is a figure explaining conversion to three-dimensional coordinate information by SLAM processing. It is a flowchart which shows the attitude | position estimation operation | movement which concerns on a present Example. FIG. 7 is a diagram illustrating a calibration system according to a first modification of the present invention. It is a figure explaining the 2nd modification of the present invention. FIG. 9 is a diagram illustrating a conventional calibration method.

  Next, embodiments of the present invention will be described in detail with reference to the drawings. In one embodiment of the present invention, the posture estimating device includes an imaging camera that images the periphery of the moving object, and an electronic device or an in-vehicle device mounted on the moving object having a function of processing imaging data from the imaging camera. including. The posture estimating device acquires imaging data from an imaging camera, sequentially restores a three-dimensional shape of a scene (landscape) around the vehicle by using SLAM technology, and additionally estimates an attitude of the imaging camera with respect to the scene. . Further, the posture estimation device can calibrate the imaging camera or the imaging data based on the estimated posture of the imaging camera. The calibration may be, for example, adjustment of internal parameters (for example, an imaging range, a trimming range of a captured image, an angle correction value, etc.) of an imaging camera or a processing circuit for processing imaging data, or an external parameter ( The position of the imaging camera, the mounting angle, etc.) may be adjusted.

  Further, the posture estimation device of the present invention may be temporarily attached to a vehicle when performing calibration, or an in-vehicle device such as a navigation device mounted on the vehicle performs its function. It may be something. The in-vehicle device may have a function of displaying the front, side, rear, and bird's-eye views of the vehicle based on image data captured by the imaging camera, a function of detecting a nearby obstacle or another vehicle based on the image data, and the like. it can. Further, the in-vehicle device may be provided with a navigation function, a function of reproducing audio / video data, a function of receiving television / radio broadcasting, a function of executing application software, and the like.

  In still another embodiment, the posture estimation function of the present invention may be executed by an external computer device separately from the on-vehicle device mounted on the vehicle. The imaging data captured by the imaging camera is provided to an external computer device by wire or wirelessly.

  Next, examples of the present invention will be described. FIG. 1 is a diagram illustrating preconditions of a posture estimation device according to an embodiment of the present invention. In FIG. 1, a vehicle M exists on a road surface (ground) R and travels on the road surface R. An imaging camera 110 is mounted in front of the vehicle M. As a precondition, it is assumed that the imaging camera 110 advances parallel to the road surface R. Vx, Vy, and Vz represent the movement vectors or speeds of the vehicle M or the imaging camera 110 in the X, Y, and Z directions when the vehicle M travels, and a, b, and c represent the normal vectors of the road surface R. . Since it is assumed that the imaging camera 110 advances parallel to the road surface R, the unit vector of the normal to the road surface (ground) R is (a, b, c) = (0, 0, 1), and the imaging camera Is a unit vector of (Vx, Vy, Vz) = (0, 1, 0). As the XYZ coordinates of the imaging camera 110, an actual measurement value is input.

  FIG. 2 is a block diagram illustrating a schematic configuration of the posture estimation device according to the embodiment of the present invention. The posture estimation device 100 according to the present embodiment includes an imaging camera 110, an input unit 120, a display unit 130, an audio output unit 140, a communication unit 150, a storage unit 160, and a control unit 170. The function of the attitude estimation device 100 may be executed by an in-vehicle device that executes a navigation function or the like.

  The image capturing camera 110 is a camera that captures an image around the vehicle. Here, an example is shown in which the image capturing camera 110 is mounted in front of the vehicle M as shown in FIG. The imaging camera 110 is mounted in front of the vehicle M as shown in FIG. 2A, and captures an image of the periphery around the optical axis C in a range θr in the left-right direction and a range θh in the up-down direction. The imaging data captured by the imaging camera 110 is sequentially provided to the control unit 170. For example, the imaging camera 110 captures an image of 30 frames per second.

  The input unit 120 receives an external instruction such as a user and provides the instruction to the control unit 170. For example, the user can input an actual measurement value of the XYZ coordinates of the imaging camera 110, input an instruction to start posture estimation, and the like. The display unit 130 can display image data captured by the image capturing camera 110, or display an estimation result of the posture estimating unit 176 described later, and the like. The audio output unit 140 can output, for example, the estimation result of the posture estimating unit 176 as audio. The communication unit 150 enables communication with an external device via wireless or wired communication means.

  The storage unit 160 stores, for example, imaging data from the imaging camera 110, a program to be executed by the control unit 170, other necessary data, and the like. The control unit 170 controls each unit of the posture estimation device 100. The control unit 170 includes an image processing unit 200 that processes image data from the imaging camera 110, a SLAM processing unit 210, a posture estimation unit 220, and a calibration unit 230. These functions are performed by software and / or hardware. The posture estimating unit 220 further includes a movement vector calculating unit 222, a road surface calculating unit 224, and a posture calculating unit 226, as shown in FIG.

  The image processing unit 200 receives image data from the image capturing camera 110 and processes the received image data. The image processing unit 200 processes, for example, image data to appropriately display an image ahead of the vehicle on the display unit 130, or processes image data to detect a vehicle ahead or the like. The image processing section 200 is operated not when determining the attitude of the imaging camera but exclusively when using imaging data.

  When determining the attitude of the imaging camera 110, the SLAM processing unit 210 receives imaging data from the monocular imaging camera 110, and converts a scene (landscape) in a three-dimensional space ahead of the vehicle based on a plurality of temporally changed image frames. Restore. The data processed by the SLM processing unit 172 includes coordinate data of a plurality of point groups for restoring a three-dimensional space.

  FIG. 4 is a diagram illustrating the conversion of the imaging data into three-dimensional coordinate information by the SLAM processing unit 210. FIG. 4A is an example of a one-frame image of the imaging data, in which a road surface R and a white line W are shown. As shown in FIG. 4B, such a frame image is converted, for example, into countless point group information having a contour of an object as a feature point. In addition, since the frame images are sequentially input to the SLAM processing unit 210, as a result, point group information in a three-dimensional space is generated. Each of the point groups has three-dimensional coordinate information in an XYZ three-dimensional space.

  The posture estimating unit 220 receives the three-dimensional coordinate information processed by the SLAM processing unit 210, and estimates the posture of the imaging camera 110. The posture estimating unit 220 includes the movement vector calculating unit 222, the road surface calculating unit 224, and the posture calculating unit 226, as described above.

  The movement vector calculation unit 222 calculates a movement vector of the imaging camera 110 (or the vehicle) based on the three-dimensional coordinate information processed by the SLAM processing unit 210. The imaging data provided from the imaging camera 110 is, for example, 30 frames per second, and the movement vector calculation unit 222 extracts feature points from the point group information as shown in FIG. The amount of movement and the direction of movement of the imaging camera 110 per unit time can be calculated by capturing the temporal change of the point. For example, comparing the feature point P at the time T1 with the feature point P at the time T2 one second after the time T1 elapses, in which direction and how much the imaging camera 110 has moved during one second. Can be calculated. The movement vectors calculated by the movement vector calculation unit 222 are Vx, Vy, and Vz shown in FIG.

The road surface calculation unit 224 calculates the widest plane, that is, the road surface, from the three-dimensional point group information shown in FIG. 4B, which is the processing result of the SLAM processing unit 210. Specifically, the road surface calculation unit 224 calculates a plane equation of the road surface defined by the following equation (2) based on the point cloud information in the three-dimensional space. a, b, and c are coefficients of x, y, and z, and d is a constant.
ax + by + cz + d = 0 (2)

  The posture calculation unit 226 uses the movement vectors Vx, Vy, and Vz calculated by the movement vector calculation unit 222 and the coefficients a, b, and c of the plane equation of the road surface calculated by the road surface calculation unit 224, using the following equation. According to (3), the attitude of the imaging camera 110 is calculated.

  The matrix of the element r in the equation (3) is a rotation matrix of the imaging camera 110, the matrix of Vx, Vy, Vz, a, b, and c is a vector in the camera coordinate system, and the matrix on the right side is FIG. Is a vector in the world coordinate system that reflects the precondition described in. Equation (3) is converted to the following equation (1).

  The matrix of the element R in the equation (1) is an inverse matrix of the rotation matrix of the imaging camera 110. Here, the rotation matrix of the imaging camera 110 will be described. In an XYZ three-dimensional space in which the imaging camera 110 is installed, assuming that a rotation angle around the Z axis is rz, a rotation angle around the X axis is rx, and a rotation angle around the Y axis is ry, a rotation matrix of the imaging camera 110 Is expanded as in the following equation (4).

  The right side of Expression (4) indicates a rotation matrix of the Z axis, a rotation matrix of the X axis, and a rotation matrix of the Y axis from the left. When the equation (4) is converted into an inverse matrix, the order of multiplication is reversed, and the matrix is transformed as in the following equation (5).

  Since the inverse of the rotation matrix is the same as the transposed matrix, equation (5) can be converted to the following equation (6).

  When the right side of Expression (6) is expanded, Expression (7) is obtained, and each element of the inverse matrix of the rotation matrix of the imaging camera 110 is calculated.

  Thereby, the relationship of Expression (8) holds.

  By solving for rx, ry, and rz based on Expression (8), Expression (9) representing the attitude of the imaging camera can be obtained.

  The values of a, b, c, and Vy are known. Therefore, if rx is obtained first, ry and rz can be obtained. In this way, it is possible to estimate the angle at which the imaging camera 110 is tilted with respect to each axis by using Expression (9). Information about the posture of the imaging camera 110 estimated by the posture estimation unit 220 is provided to the calibration unit 230.

  The calibration unit 230 performs calibration of the imaging camera 110 based on the estimation result of the posture estimation unit 220. In one embodiment, the calibration unit 230 electronically adjusts the internal parameters of the imaging camera 110 set in the image processing unit 220, and adjusts the parameters for adjusting, for example, the position correction, the trimming range, and the distortion correction of the imaging data. Optimize. Further, the calibration unit 230 may give a calibration instruction to the user while displaying the imaging data on the display unit 130. Further, when an actual measurement value of the position information (XYZ coordinates) of the imaging camera 110 is input from the input unit 120, the calibration unit 230 can perform calibration in consideration of the actual measurement value.

  Next, the operation of the posture estimation device 100 according to the present embodiment is shown in the flowchart of FIG. The start of the posture estimation operation is not particularly limited, but is started, for example, when a user's instruction is given from the input unit 120 or the like. First, the imaging by the imaging camera 110 is started while the vehicle is traveling forward (S100). Next, the image data captured by the image capturing camera 110 is sequentially supplied to the SLAM processing unit 210 (S102), and the SLAM processing unit 210 converts the image data into three-dimensional coordinate information composed of point clouds (S104). Next, the posture estimating unit 220 calculates a movement vector of the imaging camera based on the three-dimensional coordinate information (S106), calculates a plane (road surface) (S108), and calculates the posture (expression) of the imaging camera using the above-described calculation formula. (9)) is calculated (S110). Next, the calibration unit 230 performs calibration on the imaging camera based on the estimated posture of the imaging camera (S112).

  As described above, according to the present embodiment, the posture of the imaging camera is estimated from the imaging data captured by the imaging camera 110, so that a calibration sheet is prepared or a flat wide No space is required, and the time required for calibration can be reduced, and the work efficiency can be improved.

  Next, another embodiment of the present invention will be described. SLAM technology requires a computing device or computer resource with a high degree of computing power. In the above-described embodiment, the example in which the in-vehicle device implements the SLAM technology has been described. Therefore, the calibration system according to the present embodiment executes the SLAM technology using the resources of the external computer device, and performs the calibration of the vehicle-mounted camera.

  FIG. 6 is a diagram illustrating a calibration system according to the present embodiment. The calibration device 300 is a computer device having a high degree of arithmetic capability. The calibration device 300 includes the functions of the SLAM processing unit 210, the attitude estimation unit 220, and the calibration unit 230 shown in FIG. It has a function of performing wired or wireless data communication with the device. The calibration device 300 receives the imaging data captured by the imaging camera 110 via the communication unit 150 of the in-vehicle device, performs SLAM processing on the received imaging data, estimates the attitude of the imaging camera, and responds to the estimation result. A control signal for calibration is transmitted to the vehicle-mounted device. Upon receiving the control signal, the control unit 170 electronically adjusts internal parameters of the image processing unit 200 regarding the imaging data.

  As described above, according to the present embodiment, when the processing capability of the in-vehicle device, that is, the posture estimation device 100, is low, image processing such as SLAM or calculation processing can be performed at high speed by an external calibration device. On the in-vehicle device side, only calibration based on the estimated attitude of the imaging camera need be performed.

  Next, a first modified example of the present invention will be described. In the above embodiment, the posture of the imaging camera 110 is estimated on the assumption that the vehicle travels linearly. However, in this modification, even when the vehicle travels while curving, the posture of the imaging camera 110 is A method for more accurately calculating will be described.

  Based on Equation (1), the determinant when considering the curve running is as shown in the following Equation (10).

  Here, the matrix represented by the element of A is a rotation matrix related to the movement of the imaging camera 110, and can be acquired as output data of the SLAM technique. The matrix having R 'as an element is a composite matrix of a matrix having R as an element and a matrix having A as an element. If such a composite matrix is used, it can be formally calculated in the same manner as Expression (1), so that the following Expression (11) holds.

  Here, rx ′, ry ′, and rz ′ have the same formal meaning as rx, ry, and rz in Equation (9). However, since the matrix of R ′ is a composite matrix, the following equation (12) is used. ) To return to the rotation matrix of the attitude of the imaging camera 110.

  When Expression (11) is converted based on Expression (12), Expression (13) is obtained.

  As described above, according to the first modified example, even when the vehicle is running meandering or curved, appropriate calibration can be performed by calculating the attitude of the imaging camera 110 based on Expression (13). .

  Next, a second modified example of the present invention will be described. In the above-described embodiment, the posture estimation and the calibration of the imaging camera when one imaging camera is mounted on the in-vehicle apparatus are illustrated. However, the second modified example includes a plurality of imaging cameras mounted on a vehicle. If so, about calibration.

  As shown in FIG. 7A, a vehicle M includes an imaging camera 110A for imaging a front view, an imaging camera 110B for imaging a left view, an imaging camera 110C for imaging a rear view, and an imaging camera 110D for imaging a right view. Four imaging cameras are mounted. The imaging data captured by the four imaging cameras 110A to 110D is provided to the image processing unit 200, and the image processing unit 200 combines the imaging data from the four imaging cameras 110A to 110D, and obtains the image shown in FIG. It is possible to generate overhead images Pa to Pd (top view images) that look down on the vehicle M as shown, and display them on the display unit 130.

  On the other hand, when performing calibration, the imaging data captured by each of the imaging cameras 110 </ b> A to 110 </ b> D is supplied to the SLAM processing unit 210, as in the above-described embodiment, and the attitude estimation unit 220 determines the orientation of each imaging camera. Is estimated. In one embodiment, the posture estimating unit 220 aligns the coordinate information of the point group subjected to the SLAM processing corresponding to each imaging camera by an ICP algorithm as shown in FIG. The position may be estimated. Adjusting the relative positional relationship of the imaging cameras is equivalent to the generation of discontinuity or deviation of the boundaries between the images Pa, Pb, Pc, and Pd when generating a top-view image as shown in FIG. Suppress.

  In the above-described embodiment, an example has been described in which the control unit 170 includes the attitude estimation unit 220 and the calibration unit 230 separately from the SLAM processing unit 210. However, this is merely an example. It may have the function of the unit 220 and / or the calibration unit 230.

  As described above, the preferred embodiments of the present invention have been described in detail, but the present invention is not limited to the specific embodiments, and various modifications and changes can be made within the scope of the invention described in the claims. Changes are possible.

100: posture estimation device 110: imaging camera 120: input unit 130: display unit 140: audio output unit 150: communication unit 160: storage unit 170: control unit 200: image processing unit 210: SLAM processing unit 220: posture estimation unit 230 ： Calibration unit

Claims (9)

An imaging camera attached to a moving object,
A conversion unit that receives imaging data captured by the imaging camera and converts the imaging data into three-dimensional coordinate information;
First calculating means for calculating road surface information from the three-dimensional coordinate information;
Second calculating means for calculating movement information of the imaging camera from the three-dimensional coordinate information;
Estimating means for estimating the attitude of the imaging camera based on the road surface information and the movement information,
A posture estimating device having: The conversion unit generates the three-dimensional coordinate information based on a plurality of frame images received from the imaging camera,
The posture estimating apparatus according to claim 1, wherein the second calculating means calculates road surface information by processing a point group included in the three-dimensional coordinate information. The posture estimating apparatus according to claim 1, wherein the estimating unit estimates the posture of the imaging camera by calculating the following equation (1).
Here, Vx, Vy, and Vz are movement vectors of the imaging camera, a, b, and c are coefficients of a plane equation of a road surface calculated by the first calculation unit, and R is a rotation matrix of the imaging camera. Is the inverse matrix.

The posture estimating device according to any one of claims 1 to 3, wherein the posture estimating device further includes a calibration unit configured to calibrate imaging data based on the posture of the imaging camera estimated by the estimating unit. 5. The imaging camera according to claim 1, wherein the imaging camera includes a plurality of imaging cameras, the estimation unit estimates a position of each imaging camera, and the configuration unit calibrates imaging data between the plurality of imaging cameras. 6. A posture estimating apparatus according to any one of the preceding claims. An imaging camera attached to a moving body, and an in-vehicle device including a processing unit that processes imaging data captured by the imaging camera;
The in-vehicle device and a computer device separated from each other,
The computer device includes:
An acquisition unit configured to acquire imaging data captured by the imaging camera;
Conversion means for converting the image data acquired by the acquisition means into three-dimensional coordinate information;
First calculating means for calculating road surface information from the three-dimensional coordinate information;
Second calculating means for calculating movement information of the imaging camera from the three-dimensional coordinate information;
Estimating means for estimating the attitude of the imaging camera based on the road surface information and the movement information,
Calibration means for calibrating the imaging data based on the attitude of the imaging camera estimated by the estimation means,
A calibration system having: The calibration system according to claim 6, wherein the acquisition unit acquires the imaging data from the in-vehicle device by wire or wirelessly. The calibration system according to claim 6, wherein the calibration unit calibrates the processing unit of the vehicle-mounted device. The calibration system according to claim 1, wherein the estimating unit estimates the attitude of the imaging camera by calculating the following equation (1).
Here, Vx, Vy, and Vz are movement vectors of the imaging camera, a, b, and c are coefficients of a plane equation of a road surface calculated by the first calculation unit, and R is a rotation matrix of the imaging camera. Is the inverse matrix.

Images