JP2016148956A - Positioning device, positioning method and positioning computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a positioning device that can improve accuracy of association among characteristic points on images created by a plurality of cameras loaded on a vehicle corresponding to the same point on a real space.SOLUTION: A positioning device includes, in a control unit 13,: a characteristic point extraction unit 21 that extracts a first characteristic point from a first image created by a first imaging unit attached to a vehicle at a first time; a projection unit 22 that specifies a first area on a real space including the first characteristic point on the first image and corresponding to a matching range in accordance with a prescribed condition defining a position relation between a vehicle and structures around the vehicle, and projects the first area on a second image created by a second imaging unit attached to the vehicle at a second time to specify a range on the second image corresponding to the matching range; and an association unit 23 that associates with a prescribed point in an area most similar to the matching range on the second image as a second characteristic point corresponding to the first characteristic point.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an alignment apparatus, an alignment method, and an alignment computer program for associating feature points on images captured by a plurality of in-vehicle cameras directed in different directions.

  In recent years, one or more cameras may be attached to a vehicle for the purpose of driving assistance or the like. In such a case, in order to effectively use the image obtained by the camera, it can be used between a plurality of images generated by different cameras or between a plurality of images generated by the same camera at different positions. It has been proposed to associate feature points on an image corresponding to the same point in space (for example, see Patent Document 1 and Non-Patent Documents 1 and 2).

  For example, Patent Document 1 discloses an image in which characteristic points such as the center of a manhole cover and the end of a white line on a road surface are photographed by a camera attached to the moving body before and after moving when the moving body goes straight. In the meantime, it has been proposed to associate using object tracking processing.

  Non-Patent Document 1 describes that SURF features are extracted and collated for two consecutive frames between a camera mounted on a vehicle that captures the front and a camera that captures the left or right side. Further, Non-Patent Document 2 proposes that a region around a feature point on one image is deformed by warping to detect a corresponding point on the other image to be associated.

JP, 2014-101075, A

Gim Hee Lee et al., "Motion Estimation for Self-Driving Cars With a Generalized Camera", Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on. IEEE, 2013 Simon Baker et al., `` Lucas-Kanade 20 Years On: A Unifying Framework: Part 1 '', International journal of computer vision 56.3, pp. 221-255, 2004

  In the technique disclosed in Patent Literature 1, the association between two images is limited to points on the road surface. However, depending on applications, it may be required to associate feature points corresponding to points other than the road surface between images.

  Further, in the technique disclosed in Non-Patent Document 1 or 2, the viewpoints for viewing the positions of the two images taken at the same position in the real space are different. Is different. Therefore, there is a possibility that sufficient accuracy cannot be obtained with respect to the association of the feature points.

  Therefore, the present invention provides an alignment apparatus capable of improving the accuracy of association between feature points on an image corresponding to the same point in real space between images generated by each of a plurality of cameras mounted on the vehicle. An object is to provide an alignment method and an alignment computer program.

According to the first aspect of the present invention, an alignment apparatus is provided as one aspect of the present invention. The alignment device according to the present invention includes at least a first image generated at a first time by a first imaging unit (2-2) attached to the vehicle (10) so as to face the first direction. A feature point extraction unit (21) that extracts one first feature point, and a first area in real space corresponding to a matching range including the first feature point on the first image, 10) and the vehicle (10) are specified in accordance with a predetermined condition that defines the positional relationship between the structures around the vehicle (10), and the first region is directed in a second direction different from the first direction with respect to the vehicle (10). By projecting onto the second image generated at the second time different from the first time by the second imaging unit (2-1) attached in this way, the second corresponding to the matching range A projection unit (22) for specifying a range on the image, and a predetermined search on the second image. Within the range, block matching is performed between the matching range and the second image while changing the relative position of the range on the second image corresponding to the matching range, thereby specifying the region most similar to the matching range. And an associating unit (23) that sets a predetermined point in the most similar region as a second feature point corresponding to the first feature point.
The alignment device according to the present invention has the above-described configuration, so that the feature points on the image corresponding to the same point in the real space between the images generated by each of the plurality of cameras mounted on the vehicle. The accuracy of matching can be improved.

According to the second aspect of the present invention, the predetermined condition is that the first region corresponding to the matching range is on a plane parallel to the straight traveling direction of the vehicle (10) and perpendicular to the road surface, or on the road surface. The conditions are preferable.
Thereby, the alignment apparatus can appropriately determine the range on the second image corresponding to the matching range.

Alternatively, according to the third aspect, the predetermined condition is that the first region corresponding to the matching range is the position of the vehicle (10) at the first time and the position of the vehicle (10) at the second time. It is preferable that the conditions are such that the vehicle (10) is on a plane parallel to and perpendicular to the road at any position of the road on which the vehicle (10) travels.
As a result, the alignment apparatus appropriately sets the range on the second image corresponding to the matching range even if the vehicle moves in a curved line along the road between the first time and the second time. Can be determined.

According to the fourth aspect of the present invention, the projection unit (22) is obtained based on sensor information acquired from a sensor that detects a movement amount or speed of the vehicle (10) mounted on the vehicle (10). Based on the amount of movement of the vehicle (10) between the first time and the second time, the coordinates of the first region in the coordinate system with the first imaging unit as a reference are used as the reference for the second imaging unit. It is preferable to specify a range on the second image corresponding to the matching range by converting the coordinates of the first region to be converted and projecting the converted coordinates of the first region onto the second image. .
Thereby, even if the vehicle is moving between the production | generation times of two images, the alignment apparatus can specify the range on the 2nd image corresponding to a matching range appropriately.

Furthermore, according to the description of claim 5, the projection unit (22) includes the matching range obtained for the image among the plurality of images generated at different times by the second imaging unit (2-1). When the range corresponding to is included in the image, the image is preferably the second image.
Thereby, the alignment apparatus can appropriately determine the second image for associating the feature points.

Furthermore, according to the description of claim 6, the projection unit (22) includes the time when the image is generated among the plurality of images generated at different times by the second imaging unit (2-1). It is preferable that an image in which the amount of movement of the vehicle (10) during the first time falls within a predetermined range is the second image.
Thereby, the alignment apparatus can appropriately determine the second image for associating the feature points with a simple calculation.

According to the seventh aspect of the present invention, the alignment apparatus further includes a road surface detection unit that detects an area in which the road surface is reflected on the first image, and the projection unit (22) has the first feature. When the point is included in the region where the road surface is reflected, it is preferable to specify that the first region corresponding to the matching range is on the road surface.
Thereby, the alignment apparatus can set the area | region on real space corresponding to a matching range more appropriately.

Moreover, according to the description of Claim 8, the alignment method is provided as another form of this invention. Such an alignment method includes at least a first image generated at a first time by the first imaging unit (2-2) attached to the vehicle (10) so as to face the first direction. The step of extracting one first feature point, and the first region in the real space corresponding to the matching range including the first feature point on the first image are the vehicle (10) and the vehicle (10 ) Is specified according to a predetermined condition that defines the positional relationship of the surrounding structures, and the first region is attached to the vehicle (10) so as to face a second direction different from the first direction. The range on the second image corresponding to the matching range is specified by projecting onto the second image generated at the second time different from the first time by the second imaging unit (2-1) And a match within a predetermined search range on the second image The area most similar to the matching range is identified by performing block matching between the matching range and the second image while changing the relative position of the range on the second image corresponding to the matching range. And a step of setting a predetermined point in the area to be a second feature point corresponding to the first feature point.
The alignment method according to the present invention includes the above-described steps, so that the feature points on the image corresponding to the same point in the real space between the images generated by each of the plurality of cameras mounted on the vehicle. The accuracy of matching can be improved.

According to the eighth aspect of the present invention, an alignment computer program is provided as another embodiment of the present invention. The computer program for alignment is based on the first image generated at the first time by the first imaging unit (2-2) attached so as to face the vehicle (10) in the first direction. A step of extracting at least one first feature point, and a first region in the real space corresponding to the matching range including the first feature point on the first image, the vehicle (10) and the vehicle The first region is attached to the vehicle (10) so as to face a second direction different from the first direction, according to a predetermined condition that defines the positional relationship of the surrounding structures of (10). A range on the second image corresponding to the matching range is projected by the second imaging unit (2-1) onto the second image generated at a second time different from the first time. Identifying a predetermined number on the second image Within the search range, block matching is performed between the matching range and the second image while changing the relative position of the range on the second image corresponding to the matching range, and the region most similar to the matching range is identified. And a step of setting a predetermined point in the most similar area as a second feature point corresponding to the first feature point, and a command for causing the computer to execute.
The computer program for alignment according to the present invention has the above-described instructions, so that the feature on the image corresponding to the same point in the real space between the images generated by each of the plurality of cameras mounted on the vehicle. The accuracy of the correspondence between points can be improved.

  The reference numerals in parentheses attached to the above-described parts are examples that show the correspondence with specific means described in the embodiments described later.

It is a schematic block diagram of the alignment apparatus which concerns on one Embodiment of this invention. It is a functional block diagram of the control part of the alignment apparatus which concerns on one Embodiment of this invention. It is a figure which shows the relationship between the camera coordinate system about each of two cameras, and a vehicle coordinate system. It is a figure which shows the positional relationship between the point in real space, and a vehicle based on the Manhattan-World hypothesis. It is an operation | movement flowchart of a position alignment process. It is a figure which shows the example of the area | region on the real space corresponding to a patch in a modification.

Hereinafter, an alignment apparatus according to an embodiment will be described with reference to the drawings.
This alignment apparatus is mounted on a vehicle and is displayed on an image corresponding to the same point in real space between two images generated at different timings by two cameras directed in different directions. Match feature points to each other. At this time, this alignment device uses the Manhattan-World hypothesis to determine the real space area corresponding to the patch, which is the area for block matching, set around the feature points on the image generated by one camera. By setting the area in the real space and projecting the area in the real space onto the image generated by the other camera and specifying the target area for block matching on the other image, the accuracy of the feature point correspondence can be improved. Improve and reduce the amount of calculation.

  FIG. 1 is a schematic configuration diagram of an alignment apparatus according to one embodiment. As shown in FIG. 1, the alignment device 1 is mounted on a vehicle 10 and includes cameras 2-1 and 2-2, an inertial measurement device (IMU) 3, a wheel speed sensor 4, and an electronic control unit (ECU) 5. Are connected to each other via a control area network (hereinafter referred to as CAN) 6. In FIG. 1, for convenience of explanation, each component mounted on the vehicle 10 such as the alignment device 1 and the shape, size, and arrangement of the vehicle 10 are different from the actual ones.

  Each of the cameras 2-1 and 2-2 is an example of an imaging unit. In the present embodiment, each of the cameras 2-1 and 2-2 includes a two-dimensional detector composed of an array of photoelectric conversion elements having sensitivity to visible light, such as CCD or C-MOS, and the two-dimensional detector. An imaging optical system that forms an image of a region to be photographed on is provided. The camera 2-1 is attached to, for example, the vehicle interior of the vehicle 10 so that the optical axis of the imaging optical system is substantially parallel to the ground and faces the front of the vehicle 10. Then, the camera 2-1 captures a front area of the vehicle 10 at a predetermined shooting period (for example, 1/30 second), and generates an image in which the front area is captured. On the other hand, the camera 2-2 is attached to the rear end of the vehicle 10 so that the optical axis of the imaging optical system is substantially parallel to the ground and faces the rear of the vehicle 10. And the camera 2-2 image | photographs the back area | region of the vehicle 10 for every predetermined | prescribed imaging | photography period, and produces | generates the image which the back area | region was reflected. The images obtained by the cameras 2-1 and 2-2 may be color images or gray images.

  Each time the cameras 2-1 and 2-2 generate an image, they output the generated image to the alignment apparatus 1.

The alignment apparatus 1 includes a storage unit 11, a communication unit 12, and a control unit 13.
The storage unit 11 includes, for example, a semiconductor memory such as an electrically rewritable nonvolatile memory and a volatile memory. The storage unit 11 includes various programs for controlling the alignment apparatus 1, various types of information used in the alignment process, images generated by the camera 2-1 or the camera 2-2, and the control unit 13. Stores temporary calculation results by.

  The communication unit 12 includes a communication interface that communicates with the cameras 2-1, 2-2, the IMU 3, the wheel speed sensor 4, the ECU 5, and the like through the CAN 6 and its control circuit. The communication unit 12 receives images from the cameras 2-1 and 2-2 and passes the images to the control unit 13. In addition, the communication unit 12 acquires odometry information representing the speed and movement amount of the vehicle 10 from the IMU 3 or acquires the wheel speed from the wheel speed sensor 4 for each period of executing the alignment process. It passes to the control unit 13.

The control unit 13 includes one or a plurality of processors (not shown) and their peripheral circuits. And the control part 13 controls the alignment apparatus 1 whole.
FIG. 2 shows a functional block diagram of the control unit 13. As illustrated in FIG. 2, the control unit 13 includes a feature point extraction unit 21, a projection unit 22, and an association unit 23. Each of these units included in the control unit 13 is implemented as, for example, a functional module realized by a computer program executed on a processor included in the control unit 13.

In the present embodiment, the control unit 13 displays the rear of the vehicle 10 at a certain time n (here, the time is expressed in units of frame intervals, and the time in units of seconds, minutes, etc. is referred to as real time). The feature points on the image generated by the camera 2-2 to be captured are the feature points on the image generated by the camera 2-1 that captures the front of the vehicle 10 at the time (nk) k frames before the time. Correlate with. This is because, when the vehicle 10 is moving forward, the range photographed at the time n by the camera 2-2 for photographing the rear of the vehicle 10 is to photograph the front of the vehicle 10 at a time (nk) before that. This is because it is assumed that there is a portion that overlaps the range photographed by the camera 2-1.
Hereinafter, for convenience of explanation, an image generated by the camera 2-1 is referred to as a front image, and an image generated by the camera 2-2 is referred to as a rear image.

Each time the control unit 13 receives a rear image from the camera 2-2, the feature point extracting unit 21 extracts a feature point such as a road surface or a structure in the shooting area of the camera 2-2 from the rear image. To do. For this purpose, the feature point extraction unit 21 detects a corner of an edge on the rear image as a feature point using, for example, a corner detection algorithm. The feature point extraction unit 21 can use, for example, an algorithm disclosed in J. Shi et al., “Good Features to Track”, IEEE CVPR, 1994, pp.593-600 as a corner detection algorithm. Alternatively, the feature point extraction unit 21 may extract a SIFT (Scale Invariant Feature Transform) feature point from the image as the feature point. For details of SIFT feature points and extraction methods, see, for example, David G. Lowe, "Distinctive Image Features from Scale-Invariant Keypoints", Journal of Computer Vision, 2004, vol.60, No.2, pp.91- 110.
For each rear image, the feature point extraction unit 21 passes the acquisition time of the rear image and the coordinates of the feature point extracted from the rear image to the projection unit 22.

  Similarly, whenever the control unit 13 receives a front image from the camera 2-1, the feature point extraction unit 21 extracts a feature point from the front image. Then, the feature point extraction unit 21 stores the coordinates of the extracted feature points in the storage unit 11 together with the acquisition time of the front image.

For each feature point on the rear image, the projection unit 22 sets a block matching patch centered on the feature point, and specifies an area corresponding to the patch in real space. Then, the projection unit 22 projects the corresponding area of the patch in the real space on the front image acquired k frames before the rear image.
This projection processing requires conversion between coordinates on the image and coordinates in the real space. First, a coordinate system used for the conversion will be described.

  FIG. 3 is a diagram illustrating the relationship between the camera coordinate system and the vehicle coordinate system for each of the two cameras. A camera coordinate system (hereinafter referred to as a forward camera coordinate system for convenience) C1, which is a coordinate system in the real space with the camera 2-1 as a reference, uses the image-side focal point of the camera 2-1 as an origin, and a straight traveling direction of the vehicle 10 Is a coordinate system with the z axis, the direction parallel to the road surface and perpendicular to the z axis as the x axis, and the direction perpendicular to the road surface as the y axis. With respect to the z axis, the forward direction of the vehicle 10 is positive, and the positive direction of the x axis is defined by the right-handed coordinate system. Further, a camera coordinate system (hereinafter referred to as a rear camera coordinate system for convenience) C2 which is a coordinate system in the real space with the camera 2-2 as a reference has an image side focal point of the camera 2-2 as an origin, and the vehicle 10 This is a coordinate system in which the straight direction is the z axis, the direction parallel to the road surface and perpendicular to the z axis is the x axis, and the direction perpendicular to the road surface is the y axis. For the z axis, the direction toward the rear of the vehicle 10 is positive, and the positive direction of the x axis is defined by the right-handed coordinate system.

  The vehicle coordinate system V, which is a coordinate system in the real space with respect to the vehicle 10, is an arbitrary point of the vehicle 10, for example, a midpoint between front wheels, a midpoint between rear wheels, or a front side The midpoint between the midpoint between the wheels and the midpoint between the rear wheels is the origin, and the straight direction of the vehicle 10 is the z axis, the direction parallel to the road surface and perpendicular to the z axis is the x axis, and the direction perpendicular to the road surface Is the coordinate system with y-axis. With respect to the z axis, the forward direction of the vehicle 10 is positive, and the positive direction of the x axis is defined by the right-handed coordinate system. In these coordinate systems, the angle formed with the z axis in the xz plane is represented by θ.

ここで、(u,v)は、後方画像上での特徴点の座標を表し、α、βは、それぞれ、後方画像上でのパッチu_pの水平方向及び垂直方向の範囲を表す。例えば、α=[-20,20](pixel)、β=[-20,20](pixel)である。 In this embodiment, the patch is u _p a matching range for performing block matching, for example, is defined as follows:.

Here, (u, v) represents the coordinates of the feature points on the rearward image, alpha, beta respectively represent the horizontal and vertical extent of the patch u _p on the rearward image. For example, α = [-20,20] (pixel) and β = [-20,20] (pixel).

The projection 22 from the patch u _p to be set based on the feature point on the rearward image, corresponding in the real space in accordance with a predetermined condition which defines the positional relationship of the structure in the vicinity of the vehicle 10 and the vehicle 10 Identify the area. Then, the projection unit 22 for the corresponding region, the rear camera coordinate system → the vehicle coordinate system at time n → the world coordinate system that is the absolute coordinate in real space → the vehicle coordinate system at time (nk) → the front camera coordinate system → the front image Project in the order above.
Thus is described first coordinates are derived in the camera coordinate system for the area in the real space corresponding to the patch u _p.

ここで、(c_u,c_v)は後方画像の中心の座標、すなわち、カメラ２−２の光軸上の位置に相当する後方画像上の位置の座標を表す。またf_u、f_vは、それぞれ、後方画像上の画素の水平方向及び垂直方向のサイズを考慮した、水平方向及び垂直方向に相当するカメラ２−２の焦点距離を表す。そして(u_n、v_n)は、特徴点(u,v)に対応する正規化座標であり、U_n=(u_n+α_n,v_n+β_n)は、正規化座標系でのパッチu_p内の各画素の座標である。そして(x,y,z)は、それぞれ、特徴点(u,v)に対応するカメラ座標系上の点の座標である。 In converting the coordinates on the rear image or the front image (two-dimensional coordinates) to the coordinates (three-dimensional coordinates) of the camera coordinate system, a pinhole camera model is applied in the present embodiment. In this case, the coordinates of each pixel in the patch u _p on the rearward image, is converted into the coordinate U _n on the normalized coordinate system in accordance with the following equation.

Here, (c _u , c _v ) represents the coordinates of the center of the rear image, that is, the coordinates of the position on the rear image corresponding to the position on the optical axis of the camera 2-2. Further, f _u and f _v represent the focal lengths of the camera 2-2 corresponding to the horizontal direction and the vertical direction, respectively, in consideration of the size in the horizontal direction and the vertical direction of pixels on the rear image. And (u _n , v _n ) are normalized coordinates corresponding to the feature point (u, v), and U _n = (u _n + α _n , v _n + β _n ) is in the normalized coordinate system the coordinates of each pixel in the patch u _p. (X, y, z) are the coordinates of points on the camera coordinate system corresponding to the feature points (u, v), respectively.

In the present embodiment, the Manhattan-World hypothesis is introduced as a predetermined condition that defines the positional relationship between the vehicle 10 and the structures around the vehicle 10. Based on the Manhattan-World hypothesis, the real space region corresponding to the patch for each feature point is specified. In the Manhattan-World hypothesis, points in the real space corresponding to feature points are located in one of the following three types.
(1) A point on the side wall of the building parallel to the straight direction of the vehicle 10 (2) A point on the side wall of the building orthogonal to the straight direction of the vehicle 10 (3) A point on the road surface or a real space corresponding to a feature point It is assumed that the area around the top point is a plane.

  FIG. 4 is a diagram showing a positional relationship between a point on the real space and the vehicle based on the Manhattan-World hypothesis according to the present embodiment. In the present embodiment, since the camera 2-1 faces the front of the vehicle 10, while the camera 2-2 faces the rear of the vehicle 10, both of the points corresponding to the above (2) are both The feature point on one image is not associated with the feature point on the other image. Therefore, the feature point corresponding to the point 401 on the side wall parallel to the straight traveling direction of the vehicle 10 corresponding to (1) or the feature point corresponding to the point 402 on the road surface corresponding to (3) is Corresponding between the image and the back image.

When a point on the real space corresponding to the feature point is located on the side wall of the building 400 parallel to the straight traveling direction of the vehicle 10 like the point 401, each point included in the real space region corresponding to the patch of the feature point For the point, x = d in the camera coordinate system. In addition, d is the distance to the point in the real space corresponding to the vehicle 10 and the feature point. Therefore, if the above condition (1) is assumed, the coordinates P _p in the camera coordinate system for each point in the area in the real space corresponding to the patch u _p is expressed by the following equation.

On the other hand, when a point in the real space corresponding to the feature point is on the road surface, such as the point 402, each point included in the real space region corresponding to the patch of the feature point is y in the camera coordinate system. = h. Note that h is the height from the road surface to the camera 2-2 and is stored in the storage unit 11 in advance. Therefore, if the above condition (3) is assumed, the coordinates P _p in the camera coordinate system for each point in the area in the real space corresponding to the patch u _p is expressed by the following equation.

Projection 22 (2) in accordance with equation (3), or (2) and (4), the area in the real space corresponding to the patch u _p for each feature point on the rearward image of A coordinate P _p on the camera coordinate system of each point is obtained. Note that d is changed in units of 1 m in an arbitrary predetermined section, for example, a section of [1,20] (m). That is, for each feature point, it is obtained based on the coordinates P _p of 20 regions obtained based on the expressions (2) and (3) for each value of d, and the expressions (2) and (4). The coordinates P _p of one area are obtained.

  Note that the projection unit 22 may set the section d to be narrower than the predetermined section by bundle adjustment. In this case, for example, the projection unit 22 associates a feature point between two rear images obtained by photographing a point in the real space corresponding to the feature point at different times with the camera 2-2. To identify. Then, the projection unit 22 measures the distance from the vehicle 10 to a point on the real space corresponding to the feature point by triangulation using the positions of the two vehicles corresponding to the generation of the two rear images, and the measurement. The interval d may be set so as to include the determined distance.

ここで、u_n'は、座標P_pに対応する、前方画像上の領域の座標である。また関数π_vc1()は、車両座標系からカメラ２-１のカメラ座標系への変換関数であり、関数π_vc2()は、車両座標系からカメラ２-２のカメラ座標系への変換関数である。なお、これらの逆関数は、それぞれ、対応するカメラ座標系から車両座標系への変換関数となる。なお、車両座標系とカメラ座標系間の変換は、それぞれの座標系の原点の差に相当する並進行列と、車両１０の直進方向とカメラの光軸間の傾きに相当する回転行列により表される。 For the patch of each feature point on the rear image, the projection unit 22 projects a warping function for projecting each of the areas on the real space corresponding to the patch onto the front image.

Here, u _n ′ is a coordinate of a region on the front image corresponding to the coordinate P _p . The function π _vc1 () is a conversion function from the vehicle coordinate system to the camera coordinate system of the camera 2-1, and the function π _vc2 () is a conversion function from the vehicle coordinate system to the camera coordinate system of the camera 2-2. It is. Each of these inverse functions is a conversion function from the corresponding camera coordinate system to the vehicle coordinate system. Note that the conversion between the vehicle coordinate system and the camera coordinate system is represented by a parallel progression corresponding to the difference between the origins of the respective coordinate systems and a rotation matrix corresponding to the rectilinear direction of the vehicle 10 and the inclination between the optical axes of the cameras. The

The function π _ci () is a conversion function from the camera coordinate system to the coordinates on the image, and the coordinates (u _n , v _n ) on the image from the camera coordinates (x, y, z) in equation (2). Equivalent to conversion to

The function π ⁿ _wv () is a conversion function from the world coordinate system to the vehicle coordinate system at time n. On the other hand, the inverse function π ⁿ _wv ⁻¹ () is a conversion function from the vehicle coordinate system to the world coordinate system at time n.

ここで、u_nは、オドメトリ情報であり、v_n,right、v_n,leftは、それぞれ、時刻nにおける車両１０の右側後輪の車輪速、及び左側後輪の車輪速を表す。そしてω_n、v_nは、それぞれ、時刻ｎにおける車両１０の角速度及び速度を表す。またd_aは、車両１０の左右の後輪間の間隔である。さらに、Δtは、時刻(n-k)と時刻n間の実時刻の差であり、Δt=k×fで表される。なお、fは、カメラ２−１、２−２のフレーム周期（撮影間隔）である。またε_xは、移動量の誤差を確率分布で表す誤差モデルである。本実施形態では、誤差モデルε_xは、ガウス分布で表されるものとし、その共分散Σ_xは、x軸方向、z軸方向及びθに関して、以下のように定義される。

ここで、σ_x、σ_z及びσ_θは、例えば、オドメトリ情報あるいは車輪速の誤差に基づいて設定され、例えば、σ_x=σ_z=0.1[m]であり、σ_θ=2π/360[rad]である。 Details of the function π ⁿ _wv () will be described below. Let X _n = ^t (x _n , z _n , θ _n ) be a position vector representing the position and orientation of the vehicle 10 at time n in the world coordinate system. In this case, X _n is the time (nk) of the vehicle 10 at the position ^{_{X nk = t (x nk,}} z nk, θ nk) between the relationship is established as represented by the following formula.

Here, u _n is odometry information, and v _{n, right} and v _{n, left} represent the wheel speed of the right rear wheel and the left rear wheel of the vehicle 10 at time n, respectively. Ω _n and v _n represent the angular velocity and speed of the vehicle 10 at time n, respectively. D _a is the distance between the left and right rear wheels of the vehicle 10. Furthermore, Δt is the difference in real time between time (nk) and time n, and is represented by Δt = k × f. Note that f is the frame period (photographing interval) of the cameras 2-1 and 2-2. Further, ε _x is an error model that expresses an error in the movement amount by a probability distribution. In the present embodiment, the error model ε _x is represented by a Gaussian distribution, and the covariance Σ _x is defined as follows with respect to the x-axis direction, the z-axis direction, and θ.

Here, σ _x , σ _z, and σ _θ are set based on, for example, odometry information or wheel speed error. For example, σ _x = σ _z = 0.1 [m], and σ _θ = 2π / 360 [ rad].

したがって、関数πⁿ _wv()は、次式で表される。

ここで、P_wは、世界座標系での着目点の座標を表し、P_vは、P_wに対応する点の車両座標系での座標を表す。なお、（５）式における、π^n-k _wv()では、x_n=x_n-k、z_n=z_n-k、θ_n=θ_n-kとすればよい。 The rotation matrix R ^t and the parallel progression T ^t representing the transformation from the world coordinate system to the vehicle coordinate system at time n are expressed by the following equations using the position vector X _n of the vehicle 10 at time n.

Therefore, the function π ⁿ _wv () is expressed by the following equation.

Here, P _w represents the coordinates of the point of interest in the world coordinate system, and P _v represents the coordinates of the point corresponding to P _w in the vehicle coordinate system. In π ^nk _wv () in equation (5), x _n = x _nk , z _n = z _nk , θ _n = θ _nk may be set.

As described above, corresponding to the patch u _p, a region in the real space by projecting into the forward image, the position of the forward image corresponding to each pixel in the patch u _p, i.e., front patch u _p An area projected on the image is obtained.
Projection unit 22, corresponding to each pixel in the patch u _p for each feature point on the rearward image, and outputs the pixel coordinates to the associated portion 23 of the on k frames before the front image.

Associating unit 23 corresponds to the patch u _p for each feature point on the rearward image, and the region on the k frames before the front image, the area to be similar to each other by performing block matching between the front image By specifying, a feature point on the front image is associated with each feature point on the rear image.

なお、wは、（５）式の右辺を表す。 In the present embodiment, the associating unit 23 sets, for each feature point on the rear image, a search range for searching for a feature point on the corresponding front image as an error ellipse based on the error model. At that time, the error ellipse is determined by a covariance matrix Σ _F calculated by the following equation.

Note that w represents the right side of equation (5).

ここで、χ²は、χ二乗分布であり、誤差楕円の信用区間を表す。また、前方画像上の水平方向に対して誤差楕円の長径がなす角θ_εは、次式で表される。

Associating unit 23 calculates eigenvalues lambda ₁ of the covariance matrix sigma _F, and lambda _2, the eigenvector v _1, v _2. However, the larger eigenvalue is λ ₁ . At this time, the major axis a and minor axis b of the error ellipse are expressed by the following equations.

Here, χ ² is a χ square distribution and represents the confidence interval of the error ellipse. Further, the angle θ _ε formed by the major axis of the error ellipse with respect to the horizontal direction on the front image is expressed by the following equation.

ここで、(u'_c,v'_c)は、後方画像上の着目する特徴点に対応する前方画像上での画素の座標を表す。そして(u',v')は、前方画像上で検出された特徴点の座標である。 The associating unit 23 detects, for each feature point on the rear image, a feature point within an error ellipse that is a search range on the front image before k frames as a candidate to be associated with the feature point on the rear image. Note that the associating unit 23 determines that a feature point on the forward image that satisfies the following expression is included in the search range.

Here, (u ′ _c , v ′ _c ) represents the coordinates of the pixel on the front image corresponding to the feature point of interest on the rear image. (U ′, v ′) is the coordinates of the feature point detected on the front image.

  For each feature point on the rear image, the associating unit 23 performs block matching on each feature point in the search range on the front image before k frames to calculate a normalized cross-correlation value. At this time, the associating unit 23 determines the center of the area on the front image on which the patch is projected, that is, the position where the feature point on the rear image is projected on the front image, and the front image on which the block matching is performed. By translating the position of each pixel in the patch projected on the front image so as to cancel out the difference in position between the feature points on the front image, on the front image corresponding to each pixel in the patch on the back image What is necessary is just to specify a pixel. The association unit 23 can calculate a normalized cross-correlation value based on the luminance value of each pixel in the patch on the rear image and the luminance value of the pixel on the corresponding front image. The associating unit 23 associates the feature point on the front image having the maximum normalized cross-correlation value with the feature point of interest on the rear image. Note that the association unit 23 may associate the two feature points only when the maximum value of the normalized cross-correlation value is equal to or greater than a predetermined threshold (for example, 0.6 to 0.7).

  The association unit 23 sets the search range to a predetermined range that is centered on the coordinates on the front image corresponding to the feature points on the rear image in order to reduce the amount of calculation. Also good. In this case, the predetermined range may be, for example, 2 to 3 times the horizontal size of the patch in the horizontal direction and 2 to 3 times the vertical size of the patch in the vertical direction.

  According to the modification, the associating unit 23 normalizes each feature point on the rear image by translating the patch so that each pixel in the corresponding search range corresponds to the center of the patch. A cross-correlation value is calculated, and a pixel having the maximum normalized cross-correlation value may be a feature point on the front image corresponding to a feature point of interest on the rear image. In this case, the feature point extraction unit 21 may not extract feature points from the front image.

  The associating unit 23 associates a set of coordinates of the feature point on the front image associated with the feature point on the back image, and a set of the acquisition time of the corresponding back image and the acquisition time of the front image, Store in the storage unit 11.

  FIG. 5 shows an operation flowchart of the alignment process controlled by the control unit 13. In addition, the control part 13 performs the alignment process according to this operation | movement flowchart for every back image.

  The feature point extraction unit 21 extracts feature points from each of the rear image and the front image (step S101). Then, the feature point extraction unit 21 stores the coordinates of each feature point extracted from the front image in the storage unit 11. The feature point extraction unit 21 passes the coordinates of each feature point extracted from the rear image to the projection unit 22.

  For each feature point extracted from the rear image, the projection unit 22 sets a patch centered on the feature point, and sets one or more regions on the real space corresponding to the patch according to the Manhattan-World hypothesis (step S22). S102). Then, the projection unit 22 projects each region in the real space corresponding to the patch on the front image acquired k frames before the acquisition of the rear image for each feature point extracted from the rear image (step). S103).

  The associating unit 23 sets a search range for each feature point extracted from the rear image, centered on the projection position on the front image at the center of the area in the real space corresponding to the patch (step S104). Then, the associating unit 23 performs the most block matching between the area around the feature point on the front image within the search range and the projection range of the area on the real space corresponding to the patch on the front image. The matching feature point is associated with the feature point on the rear image (step S105). Then, the control unit 13 ends the alignment process.

  As described above, the alignment device uses the Manhattan-World hypothesis when matching feature points corresponding to the same point in real space between two images generated at different timings by different cameras. Based on this, an area in the real space corresponding to a patch around a feature point on one image is specified, thereby improving the accuracy in projecting the patch onto the other image. Thereby, this position alignment apparatus can improve the precision of matching of a feature point. In addition, this alignment apparatus can reduce the number of areas in the real space corresponding to the patches, and thus the amount of calculation can be reduced.

  The preferred embodiments of the present invention have been described above, but the present invention is not limited to these embodiments. For example, the control unit 13 selects a plurality of front images to be associated with feature points on one rear image, performs the processing of the projection unit 22 and the association unit 23 on the plurality of front images, The feature points may be associated with each other. At that time, the projection unit 22 selects only the forward image including the projection range of the patch set for the feature point of interest on the rear image among the plurality of front images generated at different times. You may select as a front image which matches about. Alternatively, the projection unit 22 has a movement range of the vehicle 10 between the time when the rear image is acquired and the time when the front image is acquired, among the plurality of front images generated at different times. Only a forward image included in a predetermined moving distance range that is assumed to overlap may be selected as a forward image for which feature points are associated. Note that the movement amount of the vehicle 10 is calculated based on odometry information or wheel speed of the acquisition time of the rear image and the acquisition time of the front image. Furthermore, only when the control unit 13 determines that the vehicle 10 is traveling straight between the acquisition of the rear image and the acquisition of the front image to be associated based on the odometry information. The feature points on the rear image may be associated with the feature points on the front image.

  According to another modification, the control unit 13 may further include a road surface detection unit that detects a road surface on the rear image. In this case, the projection unit 22 corresponds to the patch according to the equations (2) and (4) for the feature points within the detected road surface range among the feature points extracted on the rear image. An area in real space may be set. On the other hand, among the feature points extracted on the rear image, the projection unit 22 applies the actual points corresponding to the patches according to the equations (2) and (3) for the feature points outside the detected road surface range. A region on the space may be set.

  The road surface detection unit can use various methods for detecting a region where the road surface is shown on the image in order to detect the road surface from the rear image. For example, the road surface detection unit detects a line representing a lane such as a white line on both sides of the host vehicle by, for example, template matching, and uses a method in which an area sandwiched between lines representing the lanes on both sides is used as a road surface. it can. Alternatively, the road surface detection unit generates images respectively generated by two cameras whose shooting ranges overlap (in this case, another camera for shooting the rear of the vehicle 10 is required in addition to the camera 2-2). Alternatively, a method for detecting a road surface by three-dimensional measurement using two images taken at different times by a single camera (for example, Okutomi et al., “Flat part for visual guidance using stereo video images”). Continuous estimation ”, IPSJ Journal, Vol.43, No.4, pp.1061-1069, 2002) may be used. In this case, since the alignment apparatus can further limit the area in the real space corresponding to the patch, the accuracy of association between feature points can be further improved.

  According to yet another modification, the associating unit 23 calculates the absolute value of the difference in luminance value between corresponding pixels instead of calculating a normalized cross-correlation value for block matching between corresponding regions on the patch and the front image. The sum of the values may be calculated as the evaluation value. In this case, the associating unit 23 may associate the feature point on the front image with the smallest evaluation value with the feature point on the rear image.

Furthermore, according to another modification, the projecting unit 22 determines that the area in the real space corresponding to the patch set for each feature point on the rear image is the position of the vehicle 10 and the front of the vehicle 10 when the rear image is generated. It may be on a plane parallel to the road at any position of the road on which the vehicle 10 travels and perpendicular to the road surface between the position of the vehicle 10 when the image is generated.
FIG. 6 is a diagram showing an example of an area in the real space corresponding to the patch in this modification. In this case, since the coordinates of the feature point on the rear image can determine the angle between the direction toward the point on the real space corresponding to the feature point and the optical axis of the camera 2-2 when the rear image is generated, The direction from the vehicle 10 toward the point on the real space corresponding to the feature point at the time of generating the square image is known. Therefore, for example, the projection unit 22 refers to the trajectory information of the vehicle 10 obtained from the odometry information stored in the storage unit 11 and refers to the feature point from the position of the vehicle 10 when the rear image is generated. The search range of the distance d from the straight line 601 in the direction toward the point to the point 602 in the real space corresponding to the vehicle 10 and the feature point is the length of the perpendicular line 603 dropped from the straight line 601 to the locus 602 on which the vehicle 10 has traveled. For example, as in the above embodiment, the position on the line 601 is specified, which can be 1 m to 20 m). Then, the projection unit 22 indicates that the point 604 in the real space corresponding to the feature point is parallel to the tangential direction of the trajectory 602 on the foot of the perpendicular line 603 descending from the identified position to the trajectory 602 of the vehicle 10 and on the road surface. Assume that it lies on a vertical plane 605. Since it can be assumed that the vehicle 10 is traveling in a direction parallel to the road, the tangential direction of the trajectory of the perpendicular foot drawn from the specified position to the trajectory of the vehicle 10 is parallel to the road. It can be considered. In this case, the area in the real space corresponding to the patch set for the feature point is the angle of the vehicle 10 by the angle between the road extension direction when the rear image is generated and the road extension direction at the specified position. The coordinates of each point in the area in the real space corresponding to each pixel in the patch may be obtained by assuming that it is inclined from the straight direction and is in a plane perpendicular to the road surface.

  Furthermore, according to another modification, the alignment device sets the region on the real space corresponding to the patch for each feature point extracted from the forward image based on the Manhattan-World hypothesis, The same processing as described above may be performed to associate with feature points on the rear image. Alternatively, the two cameras that generate the images for associating the feature points are not limited to those facing the front and the rear of the vehicle, but are directed in different directions, and one of the cameras as the vehicle travels. It is only necessary that the area included in the shooting range is attached so as to be included in the shooting range of the other camera.

  For example, it is assumed that one camera is attached to the vehicle 10 so as to face the front of the vehicle 10, and the other camera is attached so as to face the side of the vehicle 10. In this case, a plane perpendicular to the traveling direction of the vehicle 10 and perpendicular to the road surface is also included in the shooting range of the two cameras. Therefore, in such a case, the feature point corresponding to the point (for example, the point 403 shown in FIG. 4) on the side wall of the building orthogonal to the straight traveling direction of the vehicle 10 shown in the above (2). It becomes possible to detect in each image obtained by each camera.

Therefore, the projection 22 (2) in accordance with equation (3), or (2) and (4), corresponding to the patch u _p for each feature point on the image obtained by one camera not only determine the coordinates P _p on the camera coordinate system of each point in the area in the real space, real space each feature point corresponds to the patch u _p in the case where a point corresponding to the (2) The coordinates P _p on the camera coordinate system of each point in the upper area are also obtained.

When the point on the real space corresponding to the feature point is on the side wall of the building orthogonal to the straight traveling direction of the vehicle 10, each point included in the region on the real space corresponding to the patch of the feature point is Since the distance d ′ from the camera in the traveling direction of the vehicle 10 is equal, z = d ′. Therefore, if the above condition (1) is assumed, the coordinates P _p in the camera coordinate system for each point in the area in the real space corresponding to the patch u _p is expressed by the following equation.

Therefore, the projection unit 22 uses one camera in the case where the point in the real space corresponding to the feature point is on the side wall of the building orthogonal to the straight traveling direction of the vehicle 10 according to the equations (2) and (14). may be obtained coordinates P _p on the camera coordinate system of each point in the area in the real space corresponding to the patch u _p for each feature point on the images obtained. Note that d ′ may be changed in units of 1 m in an arbitrary predetermined section, for example, a section of [1,20] (m).

A set of feature points associated with different images according to the above embodiment or modification can be used for, for example, self-location estimation of the vehicle 10 and creation of an environment map (Simultaneous Localization and Mapping, SLAM). In this case, since the distance between the camera positions when two images corresponding to the feature point set are generated, that is, the base line length becomes long, the point in the real space corresponding to the feature point set is The accuracy when specifying is improved.
As described above, those skilled in the art can make various modifications in accordance with the embodiment to be implemented within the scope of the present invention.

1 Positioning device 2-1, 2-2 Camera 3 IMU
4 Wheel speed sensor 5 ECU
6 CAN
DESCRIPTION OF SYMBOLS 11 Storage part 12 Communication part 13 Control part 21 Feature point extraction part 22 Projection part 23 Correlation part

Claims (9)

At least one first feature from the first image generated at the first time by the first imaging unit (2-2) attached to the vehicle (10) so as to face the first direction. A feature point extraction unit (21) for extracting points;
The first region in the real space corresponding to the matching range including the first feature point on the first image is the positional relationship between the vehicle (10) and structures around the vehicle (10). And a second imaging unit (2) attached to the vehicle (10) so as to face a second direction different from the first direction. -1), the projection unit that specifies the range on the second image corresponding to the matching range by projecting onto the second image generated at the second time different from the first time (22)
Block matching is performed between the matching range and the second image while changing a relative position of the range on the second image corresponding to the matching range within a predetermined search range on the second image. An association unit (23) that identifies an area most similar to the matching range and sets a predetermined point in the most similar area as a second feature point corresponding to the first feature point;
An alignment device having   The predetermined condition is a condition that the first region corresponding to the matching range is on a plane parallel to a straight traveling direction of the vehicle (10) and perpendicular to a road surface, or on a road surface. 2. The alignment apparatus according to 1.   The predetermined condition is that the first region corresponding to the matching range is between the position of the vehicle (10) at the first time and the position of the vehicle (10) at the second time. The alignment apparatus according to claim 1, wherein the alignment device is on a plane parallel to the road and perpendicular to the road surface at any position of the road on which the vehicle (10) travels.   The projection unit (22) is configured to obtain the first time determined based on sensor information acquired from a sensor that detects a moving amount or speed of the vehicle (10) mounted on the vehicle (10) and the Based on the amount of movement of the vehicle (10) during a second time, the coordinates of the first region in the coordinate system with the first imaging unit as a reference are used with the second imaging unit as a reference. A range on the second image corresponding to the matching range is specified by converting the coordinates of the first region onto the second image by converting the converted coordinates of the first region onto the second image. The alignment apparatus according to any one of claims 1 to 3.   The projection unit (22) includes a plurality of images generated at different times by the second imaging unit (2-1) so that a range corresponding to the matching range obtained for the image is within the image. The alignment apparatus according to any one of claims 1 to 4, wherein the image is the second image if included in the image.   The projection unit (22) includes the vehicle between the time when the image is generated and the first time among the plurality of images generated at different times by the second imaging unit (2-1). The alignment apparatus according to any one of claims 1 to 4, wherein an image in which the movement amount of (10) is within a predetermined range is the second image. A road surface detection unit for detecting a region in which the road surface is reflected on the first image;
The said projection part (22) specifies that the said 1st area | region corresponding to the said matching range exists on a road surface, when the said 1st feature point is contained in the area | region where the said road surface is reflected. The alignment apparatus as described in any one of -6. At least one first feature from the first image generated at the first time by the first imaging unit (2-2) attached to the vehicle (10) so as to face the first direction. Extracting a point;
The first region in the real space corresponding to the matching range including the first feature point on the first image is the positional relationship between the vehicle (10) and structures around the vehicle (10). And a second imaging unit (2) attached to the vehicle (10) so as to face a second direction different from the first direction. -1), specifying a range on the second image corresponding to the matching range by projecting onto a second image generated at a second time different from the first time; ,
Block matching is performed between the matching range and the second image while changing a relative position of the range on the second image corresponding to the matching range within a predetermined search range on the second image. Performing a step of specifying a region most similar to the matching range and setting a predetermined point in the most similar region as a second feature point corresponding to the first feature point;
Including an alignment method. At least one first feature from the first image generated at the first time by the first imaging unit (2-2) attached to the vehicle (10) so as to face the first direction. Extracting a point;
The first region in the real space corresponding to the matching range including the first feature point on the first image is the positional relationship between the vehicle (10) and structures around the vehicle (10). And a second imaging unit (2) attached to the vehicle (10) so as to face a second direction different from the first direction. -1), specifying a range on the second image corresponding to the matching range by projecting onto a second image generated at a second time different from the first time; ,
Block matching is performed between the matching range and the second image while changing a relative position of the range on the second image corresponding to the matching range within a predetermined search range on the second image. Performing a step of specifying a region most similar to the matching range and setting a predetermined point in the most similar region as a second feature point corresponding to the first feature point;
A computer program for alignment for causing a computer to execute.

Images