WO2019156072A1

WO2019156072A1 - Attitude estimating device

Info

Publication number: WO2019156072A1
Application number: PCT/JP2019/004065
Authority: WO
Inventors: 謙二岡野
Original assignee: 株式会社デンソー
Priority date: 2018-02-06
Filing date: 2019-02-05
Publication date: 2019-08-15
Also published as: JP2019139290A; JP6947066B2

Abstract

An attitude estimating device according to one aspect of the present disclosure is provided with an image acquiring unit (S110), a feature point extracting unit (S120), a base estimating unit (S130), and an attitude estimating unit (S140). The feature point extracting unit is configured to extract a plurality of feature points as a plurality of pre-movement points, from within at least one captured image that is acquired repeatedly, and to extract a plurality of feature points corresponding to the plurality of pre-movement points, as a plurality of post-movement points, from within a captured image acquired chronologically later than the captured image from which the plurality of pre-movement points were extracted. The base estimating unit is configured to estimate a base matrix, which is a matrix for causing the plurality of pre-movement points to migrate to the plurality of post-movement points, and which represents a matrix when a moving object is assumed to move along a preset trajectory without rotating. The attitude estimating unit is configured to estimate the attitude of one or a plurality of image capturing units in accordance with the base matrix.

Description

Posture estimation device

Cross-reference of related applications

This international application claims priority based on Japanese Patent Application No. 2018-019011 filed with the Japan Patent Office on February 6, 2018. The entire contents are incorporated herein by reference.

This disclosure relates to a posture estimation device that estimates the posture of an imaging unit.

For example, in Patent Document 1 below, as a technique for estimating the posture of an imaging unit mounted on a vehicle while the vehicle is running, an optical flow of feature points on a road surface is obtained, and a homography matrix is estimated from the optical flow. Thus, a technique for estimating the posture of the imaging unit is disclosed.

JP 2014-101075 A

However, as a result of detailed studies by the inventor, it is necessary to estimate many parameters in order to obtain a homography matrix, and the technique for estimating a homography matrix by a single process as described above provides stable accuracy. The problem that it was not possible was found.

One aspect of the present disclosure is to provide a technique for improving accuracy in a posture estimation device that estimates the posture of an imaging unit.

The posture estimation apparatus according to an aspect of the present disclosure includes an image acquisition unit, a feature point extraction unit, a basic estimation unit, and a posture estimation unit. The image acquisition unit is configured to repeatedly acquire each captured image obtained by one or a plurality of imaging units mounted on the moving object.

The feature point extraction unit extracts a plurality of feature points as a plurality of pre-movement points from at least one of the repeatedly acquired captured images, and follows a time series more than a captured image obtained by extracting the plurality of pre-movement points. A plurality of feature points corresponding to a plurality of pre-movement points are extracted as a plurality of post-movement points from the captured image acquired later.

The basic estimator is a matrix for transitioning a plurality of pre-movement points to a plurality of post-movement points, and represents a matrix when it is assumed that the moving object moves according to a preset trajectory and does not rotate. Configured to estimate a matrix. The posture estimation unit is configured to estimate the posture of one or a plurality of imaging units according to a basic matrix.

According to such a configuration, since the moving direction of the image pickup unit can be estimated by obtaining a simple basic matrix, the rough posture of the image pickup unit can be estimated with a simple process.

It is a block diagram which shows the structure of a display system. It is a flowchart (first half) of attitude | position estimation processing. It is a flowchart (second half) of posture estimation processing. It is explanatory drawing which shows a coordinate system.

Hereinafter, embodiments of one aspect of the present disclosure will be described with reference to the drawings.

[1. Embodiment]
[1-1. Constitution]
The display system 1 according to the present embodiment is a system that is mounted on a vehicle such as a passenger car and that combines images captured by a plurality of cameras and displays the images on a display unit. The camera posture is estimated and an image is displayed according to the posture. It has a function to correct. As shown in FIG. 1, the display system 1 includes a control unit 10. The display system 1 may include a front camera 21F, a rear camera 21B, a right camera 21R, a left camera 21L, various sensors 22, a display unit 26, and the like.

Note that a vehicle equipped with the display system 1 is also referred to as a host vehicle. Further, the front camera 21F, the rear camera 21B, the right camera 21R, and the left camera 21L are collectively referred to as a plurality of cameras 21.

The front camera 21F and the rear camera 21B are attached to the front part and the rear part of the own vehicle, respectively, in order to image the road ahead and behind the own vehicle. The right camera 21R and the left camera 21L are attached to the right side surface and the left side surface of the host vehicle, respectively, in order to image the right and left roads of the host vehicle. That is, the plurality of cameras 21 are respectively arranged at different positions on the host vehicle.

Further, the plurality of cameras 21 are set so that a part of the imaging regions overlap each other, and a portion where the imaging regions overlap in the captured image is defined as an overlapping region.

The various sensors 22 include, for example, a vehicle speed sensor, a yaw rate sensor, a rudder angle sensor, and the like. The various sensors 22 are used to detect whether or not the host vehicle is traveling in a steady manner such as a straight movement or a constant turning movement.

The control unit 10 includes a microcomputer having a CPU 11 and a semiconductor memory (hereinafter, memory 12) such as a RAM or a ROM. Each function of the control unit 10 is realized by the CPU 11 executing a program stored in a non-transitional physical recording medium. In this example, the memory 12 corresponds to a non-transitional tangible recording medium that stores a program. Also, by executing this program, a method corresponding to the program is executed. Note that the non-transitional tangible recording medium means that the electromagnetic waves in the recording medium are excluded. The control unit 10 may include one microcomputer or a plurality of microcomputers.

As shown in FIG. 1, the control unit 10 includes a camera posture estimation unit 16 and an image drawing unit 17. The method of realizing the functions of the respective units included in the control unit 10 is not limited to software, and some or all of the functions may be realized using one or a plurality of hardware. For example, when the function is realized by an electronic circuit that is hardware, the electronic circuit may be realized by a digital circuit, an analog circuit, or a combination thereof.

In the functions as the camera posture estimation unit 16 and the image drawing unit 17, posture estimation processing described later is performed according to a program. In particular, the camera posture estimation unit 16 estimates the postures of the plurality of cameras 21.

Also, the function as the image drawing unit 17 generates a bird's-eye image obtained by looking down the road around the vehicle from the vertical direction from images captured by the plurality of cameras 21. Then, the generated bird's-eye view image is displayed on the display unit 26 that is configured with a liquid crystal display or the like and is disposed in the vehicle interior. However, in the function as the image drawing unit 17, when generating the bird's-eye view image, the coordinates that serve as the boundaries of the captured images according to the postures of the plurality of cameras 21 so that the boundaries of the images to be combined are in appropriate positions. Is set as appropriate. The control unit 10 temporarily stores captured images obtained from the plurality of cameras 21 in the memory 12.

[1-2. processing]
Next, posture estimation processing executed by the control unit 10 will be described with reference to the flowcharts of FIGS. 2A and 2B. The posture estimation process is started, for example, when the host vehicle is performing steady running, and is repeatedly performed while performing steady running.

In particular, the posture estimation processing in the present embodiment is performed when the host vehicle is moving straight as steady running. The straight-ahead movement indicates a state in which the traveling speed of the host vehicle is a speed equal to or higher than a preset threshold and the steering angle or the yaw rate is less than a preset threshold.

In the posture estimation process, first, in S110, the control unit 10 acquires images captured by the plurality of cameras 21, respectively. At this time, a captured image captured in the past, for example, a captured image captured 10 frames before is also acquired from the memory 12.

Subsequently, in S120, the control unit 10 extracts a plurality of feature points from each captured image, and detects an optical flow for the plurality of feature points. In addition, the process after S120 is implemented for each of the plurality of cameras 21.

Here, the feature point represents an edge that becomes a boundary between luminance and chromaticity in the captured image. In the present embodiment, a point indicating a part that is relatively easy to track in image processing, such as a corner of a structure, among these edges, is adopted as a feature point. It is sufficient that at least two feature points can be extracted. However, when the host vehicle is traveling in a group of urban buildings, the control unit 10 uses corners of structures such as buildings and corners of window glass as feature points. Many may be extracted.

In addition, the control unit 10 extracts a plurality of feature points as a plurality of pre-movement points from each of the repeatedly acquired captured images, and follows a time series more than the captured image from which the plurality of pre-movement points are extracted. A plurality of feature points corresponding to a plurality of pre-movement points are extracted from a captured image acquired later as a plurality of post-movement points. And the control part 10 detects the line segment which connects the point after a movement from the point before a movement as an optical flow.

Subsequently, in S <b> 130, the control unit 10 estimates a basic matrix E on the premise of linear motion. The basic matrix E is a matrix for transitioning a plurality of pre-movement points U ₁ to a plurality of post-movement points U ₂ , and the moving object moves according to a preset trajectory, in this embodiment, a linear motion, and Represents a matrix assuming no rotation.

More specifically, for example, the following formula is used.

In the description of this embodiment, the vehicle coordinate system is defined as shown in FIG. That is, the road surface at the center of the vehicle is the origin, the right direction of the vehicle is positive on the X axis, the rear direction of the vehicle is positive on the Y axis, and the downward direction of the vehicle is positive on the Z axis.

In the above formula, the control unit 10 obtains the basic matrix E for each of the plurality of pre-movement points U ₁ , and performs the optimization operation such as the least square method on the plurality of basic matrices E, so that the most probable. A basic matrix E is obtained.

Subsequently, in S140, the control unit 10 estimates the camera posture. This process is configured to estimate the postures of the plurality of cameras 21 according to the basic matrix E. For example, the control unit 10 obtains the rotation component of the camera as follows.

The control unit 10 first converts the translation vector V included in the basic matrix E into the world coordinate system using the rotation matrix R.

Subsequently, a deviation angle dx about the X axis between the translation vector V _W in the world coordinate system and the translation vector V in the vehicle coordinate system is calculated, and a rotation matrix Rx for correcting the deviation angle dx is obtained.

Then, rotate the _{V W} in the _Rx, determine the _{V W2.}

Subsequently, a deviation angle dz about the Z axis between V _W2 and the translation vector V in the vehicle coordinate system is calculated, and a rotation matrix Rz for correcting the deviation angle dz is obtained.

Subsequently, Rx, correct the rotation matrix R in Rz, and determines a rotation matrix R ₂ corresponding to the camera angle after correction.

However, the rotational component around the Y axis is a value before correction. If such a rotation matrix R ₂ is multiplied by the basic matrix E and then an operation for changing a plurality of pre-movement points U ₁ to a plurality of post-movement points U _{2 is performed} , a rotation component around the Y axis is taken into consideration. It is possible to estimate the camera posture.

The rotation component around the Y axis may be ignored, a preset rotation matrix Ry may be used, or a calculation may be obtained as follows.

When obtaining the rotation matrix Ry, first, in S150, the control unit 10 extracts the road surface flow. That is, the control unit 10 is configured to extract a feature point on the road surface on which the moving object travels as a road surface front point from at least one of the repeatedly acquired captured images. In addition, the control part 10 recognizes the pixel which has the brightness | luminance or chromaticity within the preset range as a pixel which shows a road surface in the position in the preset captured image.

Further, the control unit 10 extracts feature points corresponding to the road surface front point as a plurality of road surface back points from the captured image acquired later in time series than the captured image from which the road surface front point is extracted. Composed. And the control part 10 detects the line segment which connects a road surface back point from a road surface front point as a road surface flow which is an optical flow.

Subsequently, in S160, the control unit 10 estimates the camera angle and the camera height from the vehicle movement amount. In this process, the basic matrix E, the road surface front point, and the road surface rear point are used to estimate the height of one or a plurality of cameras 21 with respect to the road surface and the rotation angle with respect to the road surface.

In this process, the control unit 10 performs the following process, for example. That is, using a rotation matrix R _2, obtaining the homography matrix H.

Here, a rotation matrix Ry about the Y axis that optimally satisfies the following expression is obtained using a known nonlinear optimization method such as Newton's method. In the following example, d = 1. The road surface normal vector Nw is obtained by specifying the position of a plane showing the road surface by a plurality of road surface flows and obtaining a vertical line with respect to the plane.

In this way, by identifying the Ry, it is possible to obtain a rotation matrix R ₃ for correcting the three-axis angle.

In the present embodiment, as described above, after obtaining the two axes of the rotation matrices Rx and Rz, one axis of the rotation matrix Ry is obtained separately from this processing. The process of obtaining the previous two axes does not require a road surface flow, and the two axes Rx and Rz can be obtained using the structure flow. For this reason, in this process, robustness can be improved as compared with the case of using road surface feature points that are less likely to produce contrast differences.

Subsequently, in S <b> 170, the control unit 10 associates feature points in the overlapping area by the plurality of cameras 21. That is, the control unit 10 determines whether or not the feature points located in the overlap region among the feature points are feature points that represent the same object, and when the feature points represent the same object, Is associated with each captured image. The camera postures obtained in this way are accumulated and recorded in the memory 12.

Subsequently, in S210, the control unit 10 estimates the relative position between the cameras. That is, the control unit 10 estimates the inter-camera relative position by recognizing a shift between the coordinates of the feature points located in the overlapping area and the coordinates corresponding to the corrected camera posture.

Subsequently, in S220, the control unit 10 acquires the cumulative camera posture as the cumulative camera posture. Subsequently, in S230, the control unit 10 integrates the cumulative camera posture. In this process, the control unit 10 is configured to perform filtering on a plurality of posture estimation results and output the filtered values as the postures of the plurality of cameras 21. As the filtering, any filtering method such as simple average, weighted average, and least square method can be adopted.

Subsequently, in S240, the control unit 10 performs image conversion. Image conversion is processing for generating a bird's-eye view image in consideration of the posture of the camera. Since the boundary for combining multiple captured images is set in consideration of the camera posture, one object in the imaging area may be displayed in two in the bird's eye view image, or no object may be displayed Can be suppressed.

Subsequently, in S250, the control unit 10 displays the bird's-eye view image as a video on the display unit 26, and then ends the posture estimation process of FIGS. 2A and 2B.

[1-3. effect]
According to the first embodiment described in detail above, the following effects are obtained.

(1a) In the display system 1 described above, the control unit 10 is configured to repeatedly acquire each captured image obtained by the one or more cameras 21 mounted on the moving object in S110. In S120, the control unit 10 extracts a plurality of feature points as a plurality of pre-movement points from at least one of the repeatedly acquired captured images, and more than the captured image obtained by extracting the plurality of pre-movement points. A plurality of feature points corresponding to a plurality of pre-movement points are extracted as a plurality of post-movement points from a captured image acquired later in time series.

Furthermore, the control unit 10 is a matrix for transitioning a plurality of pre-movement points to a plurality of post-movement points in S130, assuming that the moving object moves according to a preset trajectory and does not rotate. Is configured to estimate a base matrix representing a matrix of

Further, the control unit 10 is configured to estimate the posture of one or a plurality of cameras 21 according to the basic matrix in S140.

According to such a configuration, since the movement direction of the camera 21 can be estimated by obtaining a simple basic matrix, the rough posture of the camera 21 can be estimated by simple processing. Further, according to such a configuration, even when the feature points on the road surface cannot be accurately extracted as in the case where the contrast of the road surface is small, the feature points can be extracted from the structure or the like. The accuracy when estimating the camera posture can be improved.

(1b) In the display system 1 described above, in step S150, the control unit 10 uses, as a road surface front point, a feature point on the road surface on which the moving object travels from at least one of the repeatedly acquired captured images. A feature point corresponding to a road surface front point is extracted as a plurality of road surface rear points from a captured image acquired later in time series than the captured image from which the road surface front point is extracted. .

In S160, the control unit 10 is configured to estimate the height of the one or more cameras 21 relative to the road surface and the rotation angle relative to the road surface using the basic matrix, the road surface front point, and the road surface rear point. .

According to such a configuration, after obtaining the basic matrix, the height with respect to the road surface and the rotation angle with respect to the road surface are estimated using the feature points on the road surface. The estimation accuracy can be improved.

(1c) In the display system 1 described above, the control unit 10 repeatedly repeats at least the processes of S120 to S140 three or more times to change the captured image from which the feature points are extracted, and repeatedly the posture of one or more cameras 21 Is configured to estimate

In S230, the control unit 10 is configured to perform filtering on a plurality of posture estimation results and output the filtered values as the postures of one or a plurality of cameras 21.

According to such a configuration, since the posture of the camera 21 is estimated using a plurality of posture estimation results, even if a temporary erroneous estimation occurs, the influence can be reduced.

[2. Other Embodiments]
As mentioned above, although embodiment of this indication was described, this indication is not limited to the above-mentioned embodiment, and can carry out various modifications.

(2a) In the above embodiment, the configuration includes the plurality of cameras 21. However, the configuration may include only one camera.

(2b) In the above embodiment, the postures of the plurality of cameras 21 are individually estimated using the captured images obtained from the plurality of cameras 21, but the present invention is not limited to this. For example, the height of a camera whose posture is to be estimated may be estimated using the height of another camera or the like. This may be because the positional relationship between the camera whose posture is to be estimated and other cameras is often set in advance.

In the case of such a configuration, the control unit 10 uses the height from the road surface for the cameras 21 other than the camera 21 to estimate the height with respect to the road surface in S160. You may be comprised so that the height with respect to a road surface may be estimated. In particular, in S160, the control unit 10 determines the height from the road surface for the camera 21 having the largest ratio of the road surface in the captured image among the cameras 21 other than the camera 21 that is to estimate the height with respect to the road surface. It may be configured to estimate the height of one or more cameras 21 with respect to the road surface.

According to such a configuration, if the positional relationship between the plurality of cameras 21 is known, the height of the other camera 21 from the road surface can be used. It can be simplified.

Moreover, according to such a structure, the height from the road surface about the camera 21 with the largest ratio of the road surface in the captured image among the cameras 21 other than the camera 21 to estimate the height with respect to the road surface is obtained. Therefore, it is possible to accurately estimate the height from the road surface using the height from the camera 21 that is highly likely to have been obtained.

(2c) A plurality of functions of one constituent element in the embodiment may be realized by a plurality of constituent elements, or a single function of one constituent element may be realized by a plurality of constituent elements. . Further, a plurality of functions possessed by a plurality of constituent elements may be realized by one constituent element, or one function realized by a plurality of constituent elements may be realized by one constituent element. Moreover, you may abbreviate | omit a part of structure of the said embodiment. In addition, at least a part of the configuration of the above embodiment may be added to or replaced with the configuration of the other embodiment.

(2d) In addition to the display system 1 described above, non-transitional actual recording such as a device that is a component of the display system 1, a program for causing a computer to function as the display system 1, and a semiconductor memory that records the program The present disclosure can also be realized in various forms such as a medium and a posture estimation method.

[3. Correspondence between Configuration of Embodiment and Present Disclosure]
In the above embodiment, the control unit 10 corresponds to the posture estimation device according to the present disclosure. In addition, the process of S110 among the processes executed by the control unit 10 corresponds to an image acquisition unit referred to in the present disclosure, and the process of S120 corresponds to a feature point extraction unit referred to in the present disclosure.

Moreover, the process of S130 among the processes executed by the control unit 10 corresponds to a basic estimation unit in the present disclosure, and the process of S140 corresponds to an attitude estimation unit and a first estimation unit in the present disclosure. Further, the process of S150 corresponds to a travel point extraction unit as referred to in the present disclosure, and the process of S160 corresponds to a second estimation unit referred to in the present disclosure. Further, the process of S230 corresponds to the filtering unit referred to in the present disclosure.

Claims

An image acquisition unit (S110) configured to repeatedly acquire each captured image obtained by one or a plurality of imaging units mounted on a moving object;
A plurality of feature points are extracted as a plurality of pre-movement points from at least one of the repeatedly acquired captured images, and are acquired later in time series than the captured images obtained by extracting the plurality of pre-movement points. A feature point extraction unit (S120) configured to extract a plurality of feature points corresponding to the plurality of pre-movement points from the captured image as a plurality of post-movement points;
A matrix for transitioning the plurality of pre-movement points to the plurality of post-movement points, the basic matrix representing a matrix when the moving object is assumed to move and rotate according to a preset trajectory. A basic estimation unit (S130) configured to estimate;
A posture estimation unit (S140) configured to estimate the posture of the one or more imaging units according to the basic matrix;
A posture estimation device comprising:
The posture estimation apparatus according to claim 1,
A feature point on the road surface on which the moving object travels is extracted as a road front point from at least one of the repeatedly acquired captured images, and is more time-series than the captured image from which the road front point is extracted. A travel point extraction unit (S150) configured to extract a feature point corresponding to the road surface front point as a plurality of road surface rear points from the captured image acquired later along
The posture estimation unit is a first estimation unit, and the height of the one or more imaging units with respect to the road surface and the rotation angle with respect to the road surface are estimated using the basic matrix, the road surface front point, and the road surface rear point. A second estimation unit (S160) configured in
An attitude estimation apparatus further comprising:
The posture estimation apparatus according to claim 1 or 2,
The feature point extraction unit, the basic estimation unit, and the orientation estimation unit are configured to repeatedly estimate the orientation of the one or more imaging units while changing a captured image from which a feature point is extracted,
A filtering unit (S230) configured to perform filtering on the estimation results of a plurality of postures estimated by the posture estimation unit, and to output the filtered values as the postures of the one or the plurality of imaging units,
An attitude estimation apparatus further comprising:
The posture estimation apparatus according to claim 3 quoting claim 2,
The second estimating unit estimates the height of the one or more imaging units with respect to the road surface by using the height from the road surface of the imaging unit other than the imaging unit to estimate the height with respect to the road surface. Configured posture estimation device.
The posture estimation apparatus according to claim 4,
The second estimation unit uses the height from the road surface for the imaging unit that has the largest proportion of the road surface in the captured image, among the imaging units other than the imaging unit that tries to estimate the height relative to the road surface, An attitude estimation device configured to estimate a height of the one or more imaging units with respect to a road surface.