JP6886136B2 - Alignment device, alignment method and computer program for alignment - Google Patents

Description

The present invention relates to an alignment device, an alignment method, and an alignment computer program for associating line segments on images taken by a plurality of in-vehicle cameras oriented in different directions.

In recent years, one or more cameras may be attached to a vehicle for the purpose of driving assistance or the like. In such a case, in order to effectively utilize the image obtained by the camera, it is actually between a plurality of images generated by different cameras or between a plurality of images generated by the same camera at different positions. It has been proposed to associate feature points or line segments on an image corresponding to the same point or line in space with each other (see, for example, Patent Document 1 and Non-Patent Documents 1 to 3).

For example, in Patent Document 1, a feature point on a road surface, such as the center of a manhole cover and the end of a white line, is captured by a camera attached to the moving body before and after the moving body when the moving body goes straight. It has been proposed to associate between them using object tracking processing.

Further, Non-Patent Document 1 describes that SURF features are extracted and collated for two consecutive frames between a camera mounted on a vehicle that captures the front and a camera that captures the left or right side. Further, Non-Patent Document 2 proposes that a region around a feature point on one image is deformed by warping to detect a corresponding point on the other image to be associated. Furthermore, Non-Patent Document 3 proposes associating line segments detected from each of different frames with each other in order to execute Simultaneous Localization and Mapping (SLAM).

Japanese Unexamined Patent Publication No. 2014-101075

Gim Hee Lee et al., "Motion Optimization for Self-Driving Cars With a Generalized Camera", Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on. IEEE, 2013 Simon Baker et al., "Lucas-Kanade 20 Years On: A Unifying Framework: Part 1", International journal of computer vision 56.3, pp.221-255, 2004 Keisuke Hirose et al., "Fast Line Description for Line-based SLAM" BMW C 2012, 2012

In the technique disclosed in Patent Document 1, the correspondence between the two images is limited to the point on the road surface. However, depending on the application, it may be required to associate feature points corresponding to points other than the road surface between images.

Further, in the techniques disclosed in Non-Patent Documents 1 to 3, each of two images taken at the same position in the real space has different viewpoints for viewing the position, so that the position can be seen on the image. Is different. Therefore, there is a possibility that sufficient accuracy cannot be obtained for associating feature points or line segments.

Therefore, the present invention can improve the accuracy of associating line segments on an image corresponding to lines of the same structure in real space between images generated by each of a plurality of cameras mounted on a vehicle. It is an object of the present invention to provide an alignment device, an alignment method, and a computer program for alignment.

According to the description of claim 1, as one embodiment of the present invention, an alignment device is provided. The alignment device is at least from the first image generated at the first time by the first imaging unit (2-2) mounted so as to face the vehicle (10) in the first direction. A vehicle (a line segment extraction unit (21) for extracting one first line segment and a first region in real space corresponding to a matching range including the first line segment on the first image are used as a vehicle (a vehicle (21). The first region is oriented in a second direction different from the first direction with respect to the vehicle (10) by specifying according to a predetermined condition that defines the positional relationship between the structure 10) and the structure around the vehicle (10). By projecting onto the second image generated at the second time different from the first time by the second image pickup unit (2-1) attached as described above, the second image corresponding to the matching range is supported. The matching range and the projection unit (22) that specifies the range on the image and the matching range while changing the relative position of the range on the second image corresponding to the matching range within a predetermined search range on the second image. By performing block matching between the second images, the area most similar to the matching range is specified, and the line segment in the most similar area is set as the second line segment corresponding to the first line segment. It has an attachment portion (23).
By having the above configuration, the alignment device according to the present invention has an image corresponding to a line segment of the same structure in real space between images generated by each of a plurality of cameras mounted on a vehicle. The accuracy of associating line segments with each other can be improved.

Further, according to the description of claim 2, the predetermined condition is that the first region corresponding to the matching range is on a plane parallel to the straight-ahead direction of the vehicle (10) and perpendicular to the road surface, or on the road surface. It is preferable that the conditions are met.
Thereby, the alignment device can appropriately determine the range on the second image corresponding to the matching range.

Alternatively, according to the description of claim 3, the predetermined condition is that the first region corresponding to the matching range is the position of the vehicle (10) at the first time and the position of the vehicle (10) at the second time. It is preferable that the condition is parallel to the road at any position of the road on which the vehicle (10) travels and is on a plane perpendicular to the road surface.
As a result, the alignment device appropriately covers the range on the second image corresponding to the matching range even if the vehicle moves in a curved line along the road between the first time and the second time. Can be decided.

Further, according to the description of claim 4, the projection unit (22) is obtained based on the sensor information acquired from the sensor that detects the movement amount or the speed of the vehicle (10) mounted on the vehicle (10). , Based on the amount of movement of the vehicle (10) between the first time and the second time, the coordinates of the first region in the coordinate system based on the first imaging unit are referred to the second imaging unit. It is preferable to specify the range on the second image corresponding to the matching range by converting to the coordinates of the first region to be processed and projecting the converted coordinates of the first region onto the second image. ..
Thereby, the alignment device can appropriately identify the range on the second image corresponding to the matching range even if the vehicle is moving between the generation times of the two images.

Further, according to the description of claim 5, the projection unit (22) has a matching range obtained for the image among a plurality of images generated by the second image pickup unit (2-1) at different times. When the range corresponding to is included in the image, it is preferable to use the image as the second image.
As a result, the alignment device can appropriately determine the second image in which the line segments are associated with each other.

Further, according to the description of claim 6, the projection unit (22) is set to the time when the image is generated among the plurality of images generated by the second imaging unit (2-1) at different times. It is preferable that the second image is an image in which the amount of movement of the vehicle (10) during the first time is within a predetermined range.
As a result, the alignment device can appropriately determine the second image in which the line segments are associated with each other by a simple calculation.

Further, according to the description of claim 7, the alignment device further includes a road surface detection unit that detects a region in which the road surface is reflected on the first image, and the projection unit (22) is a first line segment. When the line segment is included in the area where the road surface is reflected, it is preferable to specify that the first area corresponding to the matching range is on the road surface.
As a result, the alignment device can more appropriately set the area in the real space corresponding to the matching range.

Further, according to the description of claim 8, the alignment device includes a map creation unit (24) that creates a map by writing the line segments corresponding to the first line segment and the second line segment on the map. It is preferable to have more.
As a result, the alignment device can realize SLAM.

Further, according to the description of claim 9, as another embodiment of the present invention, an alignment method is provided. Such an alignment method is at least from the first image generated at the first time by the first imaging unit (2-2) mounted so as to face the vehicle (10) in the first direction. The vehicle (10) and the vehicle (10) are the first regions in real space that correspond to the step of extracting one first line and the matching range including the first line on the first image. ) Is specified according to a predetermined condition that defines the positional relationship of the surrounding structures, and the first region is attached so as to face the vehicle (10) in a second direction different from the first direction. By projecting onto a second image generated at a second time different from the first time by the imaging unit (2-1) of 2, the range on the second image corresponding to the matching range is specified. And within a predetermined search range on the second image, block matching is performed between the matching range and the second image while changing the relative position of the range on the second image corresponding to the matching range. This includes a step of identifying a region most similar to the matching range and making the line segment in the most similar region a second line segment corresponding to the first line segment.
The alignment method according to the present invention has the above steps so that between the images generated by each of the plurality of cameras mounted on the vehicle, on the image corresponding to the line of the same structure in the real space. The accuracy of associating line segments with each other can be improved.

Further, according to claim 10, as another embodiment of the present invention, a computer program for alignment is provided. The alignment computer program is derived from the first image generated at the first time by the first imaging unit (2-2) mounted so as to face the vehicle (10) in the first direction. , The vehicle (10) and the vehicle the first region in real space corresponding to the step of extracting at least one first line segment and the matching range including the first line segment on the first image. It is specified according to a predetermined condition that defines the positional relationship of the structures around (10), and the first region is attached so as to face the vehicle (10) in a second direction different from the first direction. The range on the second image corresponding to the matching range is projected on the second image generated at the second time different from the first time by the second imaging unit (2-1). Block matching between the matching range and the second image while changing the relative position of the range on the second image corresponding to the matching range within the predetermined search range on the second image and the step of specifying To make the computer execute the step of identifying the region most similar to the matching range and setting the line segment in the most similar region as the second line segment corresponding to the first line segment. Includes instructions for.
By having the above instructions, the alignment computer program according to the present invention has an image corresponding to a line segment of the same structure in real space between images generated by each of a plurality of cameras mounted on a vehicle. The accuracy of associating the above line segments with each other can be improved.

The reference numerals in parentheses attached to the above parts are examples showing the correspondence with the specific means described in the embodiments described later.

It is a schematic block diagram of the alignment apparatus which concerns on one Embodiment of this invention. It is a functional block diagram of the control part of the alignment apparatus which concerns on one Embodiment of this invention. It is a figure which shows the relationship between the camera coordinate system and the vehicle coordinate system for each of two cameras. (A) to (c) are diagrams showing an example of a patch which is a matching range for performing block matching, respectively. It is a figure which shows the positional relationship between a point in real space and a vehicle based on the Manhattan-World hypothesis. It is an operation flowchart of the alignment process. (A) is a diagram showing an example of a road on which a vehicle has traveled, and (b) is a map created by using the alignment device according to the present embodiment corresponding to the road shown in (a). It is a figure which shows an example. It is a figure which shows the example of the region in the real space corresponding to a patch in the modification.

Hereinafter, the alignment device according to one embodiment will be described with reference to the drawings.
This alignment device is a line segment (structure) of the same structure in real space between two images generated at different timings by two cameras mounted on a vehicle and pointed in different directions. The line segments on the image corresponding to (including not only the lines based on the shape of) but also the lines drawn on the structure such as figures or characters are associated with each other. At that time, this alignment device sets a real space area corresponding to a patch, which is a block matching area, set around a line segment extracted from an image generated by one camera. By setting based on the World hypothesis and projecting the area in the real space onto the image generated by the other camera to specify the target area for block matching on the other image, the line segments can be associated with each other. The accuracy of the camera is improved and the amount of calculation is reduced.

FIG. 1 is a schematic configuration diagram of an alignment device according to one embodiment. As shown in FIG. 1, the alignment device 1 is mounted on the vehicle 10 and includes cameras 2-1 and 2-2, an inertial measurement unit (IMU) 3, a wheel speed sensor 4, and an electronic control unit (ECU) 5. , Control area networks (hereinafter referred to as CAN) 6 are connected to each other. In FIG. 1, for convenience of explanation, each component mounted on the vehicle 10 such as the alignment device 1 and the shape, size, and arrangement of the vehicle 10 are different from the actual ones.

Cameras 2-1 and 2-2 are examples of imaging units, respectively. In the present embodiment, the cameras 2-1 and 2-2 are a two-dimensional detector composed of an array of photoelectric conversion elements having sensitivity to visible light such as CCD or C-MOS, respectively, and the two-dimensional detector thereof. It has an imaging optical system on top that forms an image of the region to be imaged. The camera 2-1 is mounted in, for example, the interior of the vehicle 10 so that the optical axis of the imaging optical system is substantially parallel to the ground and faces the front of the vehicle 10. Then, the camera 2-1 captures the front region of the vehicle 10 at predetermined shooting cycles (for example, 1/30 second), and generates an image in which the front region is captured. On the other hand, the camera 2-2 is attached to the rear end of the vehicle 10 so that the optical axis of the imaging optical system is substantially parallel to the ground and faces the rear of the vehicle 10. Then, the camera 2-2 photographs the rear region of the vehicle 10 at predetermined imaging cycles, and generates an image in which the rear region is captured. The images obtained by the cameras 2-1 and 2-2 may be color images or gray images.

Each time the cameras 2-1 and 2-2 generate an image, the cameras 2-1 and 2-2 output the generated image to the alignment device 1.

The positioning device 1 includes a storage unit 11, a communication unit 12, and a control unit 13.
The storage unit 11 has, for example, a semiconductor memory such as an electrically rewritable non-volatile memory and a volatile memory. Then, the storage unit 11 contains various programs for controlling the alignment device 1, various information used in the alignment process, an image generated by the camera 2-1 or the camera 2-2, and the control unit 13. Stores temporary calculation results by.

The communication unit 12 has a communication interface and a control circuit thereof that communicate with cameras 2-1 and 2-2, IMU3, wheel speed sensor 4, ECU5 and the like through CAN6. Then, the communication unit 12 receives an image from the cameras 2-1 and 2-2 and passes the image to the control unit 13. Further, the communication unit 12 acquires odometry information indicating the speed and the amount of movement of the vehicle 10 from the IMU 3 or acquires the wheel speed from the wheel speed sensor 4 for each cycle of executing the alignment process. Pass it to the control unit 13.

The control unit 13 includes one or a plurality of processors (not shown) and peripheral circuits thereof. Then, the control unit 13 controls the entire alignment device 1.
FIG. 2 shows a functional block diagram of the control unit 13. As shown in FIG. 2, the control unit 13 includes a line segment extraction unit 21, a projection unit 22, an association unit 23, and a map creation unit 24. Each of these units included in the control unit 13 is implemented as, for example, a functional module realized by a computer program executed on the processor included in the control unit 13.

In the present embodiment, the control unit 13 sets the rear of the vehicle 10 at a certain time n (here, the time in units of frame intervals is represented. The time in units of seconds, minutes, etc. is referred to as real time). The line segment on the image generated by the camera 2-2 to be photographed is the line segment on the image generated by the camera 2-1 that photographs the front of the vehicle 10 at a time (nk) k frames before that time. Correspond with. This is because when the vehicle 10 is moving forward, the range photographed at time n by the camera 2-2 that photographs the rear of the vehicle 10 photographs the front of the vehicle 10 at the time (nk) before that. This is because it is assumed that there is a portion that overlaps the range captured by the camera 2-1.
In the following, for convenience of explanation, the image generated by the camera 2-1 is referred to as a front image, and the image generated by the camera 2-2 is referred to as a rear image.

Each time the control unit 13 receives a rear image from the camera 2-2, the line segment extraction unit 21 refers to a road surface, a road sign, a signboard, or a building in the photographing area of the camera 2-2 from the rear image. Extract the line segment corresponding to the line of the structure. Therefore, the line segment extraction unit 21 detects a line segment on the rear image using, for example, a line segment detector. As a line segment detector, for example, RG von Gioi et al., "A fast line segment detector with a false detection control", IEEE Transactions on Pattern Analysis & Machine Intelligence, (4), 2008, pp. The algorithms disclosed in .722-732 are available. Alternatively, the line segment extraction unit 21 may use another line segment detector capable of detecting the line segment from the image.
For each rear image, the line segment extraction unit 21 obtains a vector representing the acquisition time of the rear image and a vector representing the coordinates, orientation, and length of the midpoint of the line segment on the rear image representing the line segment extracted from the rear image. It is passed to the projection unit 22.

Similarly, the line segment extraction unit 21 extracts a line segment from the front image each time the control unit 13 receives the front image from the camera 2-1. Then, the line segment extraction unit 21 stores the vector representing the coordinates, direction, and length of the midpoint of the extracted line segment in the storage unit 11 together with the acquisition time of the front image.

The projection unit 22 sets a patch for block matching centered on each line segment on the rear image, and specifies an area corresponding to the patch in the real space. Then, the projection unit 22 projects the corresponding area of the patch in the real space onto the front image acquired k frames before the rear image.
This projection process requires conversion between the coordinates on the image and the coordinates in the real space. Therefore, first, the coordinate system used for the conversion will be described.

FIG. 3 is a diagram showing the relationship between the camera coordinate system and the vehicle coordinate system for each of the two cameras. The camera coordinate system (hereinafter referred to as the front camera coordinate system for convenience) C1, which is a coordinate system in real space with reference to the camera 2-1 as the origin, is the straight-ahead direction of the vehicle 10 with the image-side focus of the camera 2-1 as the origin. Is the z-axis, the direction parallel to the road surface and orthogonal to the z-axis is the x-axis, and the direction perpendicular to the road surface is the y-axis. Then, with respect to the z-axis, the direction toward the front of the vehicle 10 is positive, and the positive direction of the x-axis is defined by the right-hand coordinate system. Further, the camera coordinate system (hereinafter, referred to as the rear camera coordinate system for convenience) C2, which is a coordinate system in the real space with reference to the camera 2-2, has the image side focus of the camera 2-2 as the origin, and the vehicle 10 It is a coordinate system in which the straight direction is the z-axis, the direction parallel to the road surface and orthogonal to the z-axis is the x-axis, and the direction perpendicular to the road surface is the y-axis. With respect to the z-axis, the direction toward the rear of the vehicle 10 is positive, and the positive direction of the x-axis is defined by the right-hand coordinate system.

Further, the vehicle coordinate system V, which is a coordinate system in real space with reference to the vehicle 10, is an arbitrary point of the vehicle 10, for example, a midpoint between front wheels, a midpoint between rear wheels, or a front side. With the midpoint between the midpoint between the wheels and the midpoint between the wheels on the rear side as the origin, the straight direction of the vehicle 10 is the z-axis, the direction parallel to the road surface and orthogonal to the z-axis is the x-axis, and the direction perpendicular to the road surface. Is a coordinate system whose y-axis is. Then, with respect to the z-axis, the direction toward the front of the vehicle 10 is positive, and the positive direction of the x-axis is defined by the right-hand coordinate system. Further, in these coordinate systems, the angle formed by the z-axis in the xz plane is represented by θ.

4 (a) to 4 (c) are diagrams showing an example of a patch which is a matching range for performing block matching, respectively. In the example shown in FIG. 4A, the patch 401 has the same length as the line segment 410 along the direction parallel to the line segment 410 of interest, and has a predetermined width in the direction orthogonal to the line segment 410. Set as a rectangular area with (eg 10-20 pixels). Further, in the example shown in FIG. 4B, the patch 402 has a predetermined width (for example, 10 to 20 pixels) in a direction orthogonal to the line segment 410 of interest, and has a direction parallel to the line segment 410. Along the line segment 410, it is set as a rectangular area longer than the line segment 410 by a predetermined length (for example, 10 to 20 pixels). Further, in the example shown in FIG. 4C, the patch 403 is set as a region where the distance to the line segment 410 of interest is a predetermined value (for example, 10 to 20 pixels) or less.

ここで、(u_m,2,v_m,2)は、後方画像上での着目する線分の中点の座標を表し、α、βは、それぞれ、後方画像上でのパッチu_p内の画素の線分の中点からの水平方向及び垂直方向の距離を表す。なお、αの取り得る値の範囲[α₀, α₁]、及び、βの取り得る値の範囲[β₀, β₁]は、それぞれ、垂直方向の位置及び水平方向の位置についての関数として表される。そしてその関数は、線分の向き、及び、図４（ａ）〜図４（ｃ）に示される、パッチの形状に応じて決定される。 In this embodiment, the patch is u _p a matching range for performing block matching, for example, is defined as follows:.

Here, (um _{, 2} , v _{m, 2} ) represents the coordinates of the midpoint of the line segment of interest on the rear image, and α and β are respectively in the patch u _p on the rear image. Represents the horizontal and vertical distances from the midpoint of the line segment of the pixel. The range of possible values of α [α ₀ , α ₁ ] and the range of possible values of β [β ₀ , β ₁ ] are functions of the vertical position and the horizontal position, respectively. expressed. The function is determined according to the direction of the line segment and the shape of the patch shown in FIGS. 4 (a) to 4 (c).

The projection 22 from the patch u _p to be set on the basis of the line segment on the rearward image, corresponding in the real space in accordance with a predetermined condition which defines the positional relationship of the structure in the vicinity of the vehicle 10 and the vehicle 10 Identify the area. Then, the projection unit 22 has a rear camera coordinate system → a vehicle coordinate system at time n → a world coordinate system which is an absolute coordinate in real space → a vehicle coordinate system at time (nk) → a front camera coordinate system → a front image. Project in the order of the top.
Thus is described first coordinates are derived in the camera coordinate system for the area in the real space corresponding to the patch u _p.

ここで、(c_u,c_v)は後方画像の中心の座標、すなわち、カメラ２−２の光軸上の位置に相当する後方画像上の位置の座標を表す。またf_u、f_vは、それぞれ、後方画像上の画素の水平方向及び垂直方向のサイズを考慮した、水平方向及び垂直方向に相当するカメラ２−２の焦点距離を表す。そして(u_n、v_n)は、線分の中点(u_m,2,v_m,2)に対応する正規化座標であり、U_n=(u_n+α_n,v_n+β_n)は、正規化座標系でのパッチu_p内の各画素の座標である。そして(x,y,z)は、U_nに対応するカメラ座標系上の点の座標である。 In the present embodiment, a pinhole camera model is applied when converting the coordinates (two-dimensional coordinates) on the rear image or the front image into the coordinates (three-dimensional coordinates) of the camera coordinate system. In this case, the coordinates of each pixel in the patch u _p on the rearward image, is converted into the coordinate U _n on the normalized coordinate system in accordance with the following equation.

Here, ( _cu , c _v ) represents the coordinates of the center of the rear image, that is, the coordinates of the position on the rear image corresponding to the position on the optical axis of the camera 2-2. Further, f _u and f _v represent the focal lengths of the cameras 2-2 corresponding to the horizontal and vertical directions in consideration of the horizontal and vertical sizes of the pixels on the rear image, respectively. Then (u _n, v _n) is the normalized coordinates corresponding to the line segment of the mid-point _{(u m, 2, v m} , 2), U n = (u n + α n, v n + β n ) are the coordinates of each pixel in the patch u _p in a normalized coordinate system. The (x, y, z) are the coordinates of points on the camera coordinate system corresponding to U _n.

In this embodiment, the Manhattan-World hypothesis is introduced as a predetermined condition that defines the positional relationship between the vehicle 10 and the structures around the vehicle 10. Then, based on the Manhattan-World hypothesis, identify the real-space region corresponding to the patch for each line segment. According to the Manhattan-World hypothesis, the line of the structure in the real space corresponding to the line segment on the image is one of the following three types.
(1) Line on the side wall of the building parallel to the straight line direction of vehicle 10 (2) Line on the side wall of the building orthogonal to the straight line direction of vehicle 10 (3) Line on the road surface Also, it corresponds to the line segment on the image. It is assumed that the perimeter of the line of the structure in the real space is flat.

FIG. 5 is a diagram showing the positional relationship between the line of the structure in the real space and the vehicle based on the Manhattan-World hypothesis according to the present embodiment. In the present embodiment, the camera 2-1 faces the front of the vehicle 10, while the camera 2-2 faces the rear of the vehicle 10, so that both lines corresponding to (2) above are both. It is not visible to the camera and the line segment on one image is not associated with the line segment on the other image. Therefore, the line segment corresponding to the line 501 on the side wall parallel to the straight line direction of the vehicle 10 corresponding to (1) or the line segment corresponding to the line 502 on the road surface corresponding to (3) is forward. It is associated between the image and the rear image.

When the line of the structure in the real space corresponding to the line segment on the image is on the side wall of the building 500 parallel to the straight line direction of the vehicle 10, as in line 501, on the real space corresponding to the patch of the line segment. For each point included in the area of, x = d in the camera coordinate system. Note that d is the distance to the vehicle 10 and the line in the real space corresponding to the line segment thereof. Therefore, when the above condition (1) is assumed _{, the coordinates P p} in the camera coordinate system for each point in the real space region corresponding to the _{patch u p} are expressed by the following equations.

On the other hand, when the line of the structure in the real space corresponding to the line segment on the image is on the road surface as in the point 502, for each point included in the area in the real space corresponding to the patch of the line segment. , In the camera coordinate system, y = h. Note that h is the height from the road surface to the camera 2-2, and is stored in the storage unit 11 in advance. Therefore, when the above condition (3) is assumed _{, the coordinates P p} in the camera coordinate system for each point in the real space region corresponding to the _{patch u p} are expressed by the following equations.

Projection 22 (2) and (3), or (2) and (4) according to wherein each of the regions in the real space corresponding to the patch u _p of each line segment on the rearward image _{Find the coordinates P p} of the points on the camera coordinate system. Note that d is changed in 1 m units in an arbitrary predetermined section, for example, a section of [1,20] (m). _{That is, for each line segment, the coordinates P p} of 20 regions obtained based on Eqs. (2) and (3) for each value of d, and 1 obtained based on Eqs. (2) and (4). The coordinates P _p of the regions are obtained.

The projection unit 22 may set the section d narrower than the above-mentioned predetermined section by bundle adjustment. In this case, the projection unit 22 captures, for example, the lines of the structure in real space corresponding to the line segments on the image between the two rear images obtained by shooting the lines of the structure with the camera 2-2 at different times. It is specified by associating line segments with. Then, the projection unit 22 is a point in real space corresponding to a point (for example, a midpoint) on the line segment from the vehicle 10 by a triangular measurement using the positions of the two vehicles corresponding to the generation of the two rear images. The distance to is measured, and the interval d may be set so as to include the measured distance.

ここで、u_n'は、座標P_pに対応する、前方画像上の領域の座標である。また関数π_vc1()は、車両座標系からカメラ２-１のカメラ座標系への変換関数であり、関数π_vc2()は、車両座標系からカメラ２-２のカメラ座標系への変換関数である。なお、これらの逆関数は、それぞれ、対応するカメラ座標系から車両座標系への変換関数となる。なお、車両座標系とカメラ座標系間の変換は、それぞれの座標系の原点の差に相当する並進行列T_vc1,T_vc2と、車両１０の直進方向とカメラの光軸間の傾きに相当する回転行列R_vc1,R_vc2により表される。 For each line segment patch on the rear image, the projection function 22 projects each of the real-space regions corresponding to the patch onto the front image, and the warping function is expressed by the following equation.

Here, u _n 'corresponds to the coordinates P _p, the coordinates of the region of the forward image. The function π _vc1 () is a conversion function from the vehicle coordinate system to the camera coordinate system of the camera 2-1 and the function π _vc2 () is a conversion function from the vehicle coordinate system to the camera coordinate system of the camera 2-2. Is. Each of these inverse functions is a conversion function from the corresponding camera coordinate system to the vehicle coordinate system. The conversion between the vehicle coordinate system and the camera coordinate system corresponds to the _{parallel traveling matrices T vc1} and T vc2 corresponding to the difference in the origins of the respective coordinate systems, and the inclination between the straight direction of the vehicle 10 and the optical axis of the camera. It is represented by the rotation matrix R _vc1 and R _vc2.

The function [pi _ci () is a conversion function to the coordinates on the image from the camera coordinate system, (2) of the camera coordinate (x, y, z) from the image coordinates (u _n, v _n) Corresponds to conversion to.

The function π ⁿ _wv () is a conversion function from the world coordinate system to the vehicle coordinate system at time n. On the other hand, its inverse function π ⁿ _wv ^-1 () is a conversion function from the vehicle coordinate system to the world coordinate system at time n.

ここで、u_nは、オドメトリ情報であり、v_n,right、v_n,leftは、それぞれ、時刻nにおける車両１０の右側後輪の車輪速、及び左側後輪の車輪速を表す。そしてω_n、v_nは、それぞれ、時刻ｎにおける車両１０の角速度及び速度を表す。またd_aは、車両１０の左右の後輪間の間隔である。さらに、Δtは、時刻(n-k)と時刻n間の実時刻の差であり、Δt=k×fで表される。なお、fは、カメラ２−１、２−２のフレーム周期（撮影間隔）である。またε_xは、移動量の誤差を確率分布で表す誤差モデルである。本実施形態では、誤差モデルε_xは、ガウス分布で表されるものとし、その共分散Σ_xは、x軸方向、z軸方向及びθに関して、以下のように定義される。

ここで、σ_x、σ_z及びσ_θは、例えば、オドメトリ情報あるいは車輪速の誤差に基づいて設定され、例えば、σ_x=σ_z=0.1[m]であり、σ_θ=2π/360[rad]である。 The details of the function π ⁿ _wv () will be described below. Let X _n = ^t (x _n , z _n , θ _n ) be a position vector representing the position and orientation of the vehicle 10 at time n in the world coordinate system. In this case, X _n holds the relationship expressed by the following equation with _{the position X nk} = ^t (x _nk , z _nk , θ _nk ) of the vehicle 10 at the time (nk).

Here, u _n is odometry information, v _{n,. Right,} v _{n, left,} respectively, the wheel speed of the right rear wheel of the vehicle 10 at time n, and represents the wheel speed of the left rear wheel. And ω _n and v _n represent the angular velocity and the speed of the vehicle 10 at the time n, respectively. The d _a is the distance between the left and right rear wheels of the vehicle 10. Further, Δt is the difference between the time (nk) and the real time between the time n, and is represented by Δt = k × f. Note that f is the frame period (shooting interval) of the cameras 2-1 and 2-2. Ε _x is an error model that expresses the error of the amount of movement by a probability distribution. In this embodiment, the error model ε _x is represented by a Gaussian distribution, and its covariance Σ _x is defined as follows with respect to the x-axis direction, the z-axis direction, and θ.

Here, σ _x , σ _z and σ _θ are set based on, for example, odometry information or wheel speed error, for example, σ _x = σ _z = 0.1 [m], and σ _θ = 2π / 360 [. rad].

したがって、関数πⁿ _wv()は、次式で表される。

ここで、P_wは、世界座標系での着目点の座標を表し、P_vは、P_wに対応する点の車両座標系での座標を表す。なお、（５）式における、π^n-k _wv()では、x_n=x_n-k、z_n=z_n-k、θ_n=θ_n-kとすればよい。 ^{The rotation matrix R t} and the parallel ^{traveling matrix T t} representing the conversion from the world coordinate system to the vehicle coordinate system at time n are expressed by the following equations using _{the position vector X n} of the vehicle 10 at time n.

Therefore, the function π ⁿ _wv () is expressed by the following equation.

Here, P _w represents the coordinates of the point of interest in the world coordinate system, and P _v represents the coordinates of the point corresponding to P _w in the vehicle coordinate system. ^{In π nk} _wv () in Eq. (5) _{, x n} = x _nk , z _n = z _nk , and θ _n = θ _nk may be set.

ここでRⁱ _vwは、時刻iにおける、変換式πⁱ _wvに含まれる回転行列Rの逆行列であり、（８）式及び（９）式に従って算出される。 Further, in order to more accurately identify the corresponding line segment on the front image with reference to the line segment on the rear image, the line segment on the front image corresponding to the direction of the line segment on the rear image. It is preferable that the orientation of is required. A line segment in real space is a three-dimensional coordinate p (px, py, pz) of a point (for example, a midpoint) on the line segment and a three-dimensional vector r (rx, rx,) representing the direction of the line segment. It is represented by ry, rz). Therefore, the conversion formula for converting the three-dimensional vector r representing the direction of the line segment in the world coordinate system to the vector A ₁ in the camera coordinate system based on the camera 2-1 and the camera 2-2 are used as the reference. The conversion formula for converting to the vector A _{2 in the} camera coordinate system is expressed by the following formula.

Here, R ⁱ _{v w} is the inverse matrix of the rotation matrix R included in the transformation equation π ^{i w} _v at time i, and is calculated according to equations (8) and (9).

ここで、q₁(x₁,y₁,z₁)は、カメラ２−１を基準とするカメラ座標系での線分上の点pの座標を表し、q₂(x₂,y₂,z₂)は、カメラ２−２を基準とするカメラ座標系での線分上の点pの座標を表し、それぞれ、（５）〜（９）式に従って算出される。したがって、投影部２２は、後方画像上の各線分について、その線分の向きを表すベクトルを、カメラ２−２に関する（１０）式及び（１１）式の逆変換に従って世界座標系でのベクトルに変換し、その後、カメラ２−１に関する（１０）式及び（１１）式に従って、世界座標系でのベクトルから、前方画像上での対応ベクトルを求めることができる。 In addition, the equation _{for projecting the vector A 1 of the camera coordinate system onto the front image to calculate the correspondence vector a 1} on the front image, and the correspondence on the rear image by projecting the vector A ₂ of the camera coordinate system onto the rear image. The formula for calculating the vector a ₂ is expressed by the following formula.

Here, q ₁ (x ₁ , y ₁ , z ₁ ) represents the coordinates of the point p on the line segment in the camera coordinate system with respect to the camera 2-1 and q ₂ (x ₂ , y ₂ ,, z 1). z ₂ ) represents the coordinates of the point p on the line segment in the camera coordinate system with respect to the camera 2-2, and is calculated according to the equations (5) to (9), respectively. Therefore, for each line segment on the rear image, the projection unit 22 converts the vector representing the direction of the line segment into a vector in the world coordinate system according to the inverse conversion of equations (10) and (11) with respect to the camera 2-2. After conversion, the corresponding vector on the front image can be obtained from the vector in the world coordinate system according to the equations (10) and (11) relating to the camera 2-1.

As described above, corresponding to the patch u _p, a region in the real space by projecting into the forward image, the position of the forward image corresponding to each pixel in the patch u _p, i.e., front patch u _p The area projected on the image is required.
Projection unit 22, corresponding to each pixel in the patch u _p of each line segment on the rearward image, k frames before the pixel in the forward image coordinates and the line segment orientation to the associated portion 23 of the vector representing the Output.

Associating unit 23, the particular corresponding to the patch u _p of each line segment on the rearward image, and the region on the k frames before the front image, the area to be similar to each other by performing block matching between the front image By doing so, each line segment on the rear image is associated with the line segment on the front image.

なお、wは、（５）式の右辺を表す。 In the present embodiment, the association unit 23 sets the search range for searching the corresponding line segment on the front image as an error ellipse based on the error model for each line segment on the rear image. At that time, the error ellipse is determined by the covariance matrix Σ _F calculated by the following equation.

Note that w represents the right side of equation (5).

ここで、χ²は、χ二乗分布であり、誤差楕円の信用区間を表す。また、前方画像上の水平方向に対して誤差楕円の長径がなす角θ_εは、次式で表される。

The association unit 23 calculates the eigenvalues λ ₁ and λ ₂ of the covariance matrix Σ _F and the eigenvectors v ₁ and v _2. However, the larger eigenvalue is λ ₁ . At this time, the major axis a and the minor axis b of the error ellipse are expressed by the following equations.

Here, χ ² is a χ-square distribution and represents the confidence interval of the error ellipse. _{The angle θ ε} formed by the major axis of the error ellipse with respect to the horizontal direction on the front image is expressed by the following equation.

ここで、(u'_c,v'_c)は、後方画像上の着目する線分に対応する前方画像上での線分の中点の座標を表す。そして(u',v')は、前方画像上で検出された線分の中点の座標である。 For each line segment on the rear image, the association unit 23 is within the error ellipse which is the search range on the front image before k frames, and the angle formed with the line segment when projected onto the front image is predetermined. A line segment on the front image that is less than or equal to an angle (for example, 3 ° to 5 °) is detected as a candidate to be associated with a line segment on the rear image. The association unit 23 determines that the line segment on the front image satisfying the following equation is included in the search range.

Here, the _{(u 'c, v' c} ), representing the coordinates of the line segment at the midpoint on the front image corresponding to the line segment of interest on the rearward image. And (u', v') is the coordinates of the midpoint of the line segment detected on the front image.

The association unit 23 performs block matching for each line segment on the rear image and for each line segment within the search range on the front image k frames before, and calculates a normalized cross-correlation value. At that time, the associating unit 23 sets the center of the region on the front image on which the patch is projected, that is, the position where the line segment on the rear image is projected on the front image, and the front image to be block-matched. By moving the position of each pixel in the patch projected on the front image in parallel so as to cancel the difference in position between the lines, the pixel on the front image corresponding to each pixel in the patch on the rear image. Should be specified. Then, the association unit 23 can calculate the normalized cross-correlation value based on the brightness value of each pixel in the patch on the rear image and the brightness value of the corresponding pixel on the front image. The association unit 23 associates the line segment on the front image with the maximum normalized cross-correlation value with the line segment of interest on the rear image. The associating unit 23 may associate the two line segments only when the maximum value of the normalized cross-correlation value is equal to or more than a predetermined threshold value (for example, 0.6 to 0.7).

In addition, in order to reduce the amount of calculation, the association unit 23 sets the search range to a predetermined range set in advance centered on the coordinates on the front image corresponding to the midpoint of the line segment on the rear image. It may be set. In this case, the predetermined range can be, for example, 2 to 3 times the horizontal size of the patch in the horizontal direction and 2 to 3 times the vertical size of the patch in the vertical direction.

According to the modified example, since the patch shape itself includes information on the direction of the line segment on the rear image, the association unit 23 is in the search range on the front image before the k frame. All the line segments on the front image may be candidates for associating with the line segments on the rear image.

Further, according to another modification, the association unit 23 moves the patch in parallel so that each line segment on the rear image corresponds to the center of the patch in the corresponding search range, and is normal. Calculate the conversion intercorrelation value, and set the pixel with the maximum normalized mutual correlation value as the midpoint of the line segment on the front image corresponding to the line segment of interest on the rear image, and set the direction of the line segment of interest. The orientation on the corresponding front image may be the orientation of the corresponding line segment. In this case, the line segment extraction unit 21 does not have to extract the line segment from the front image.

The association unit 23 includes a set of vectors representing the coordinates and directions of the midpoints of the line segments on the front image associated with the line segments on the rear image, and the acquisition time and the front image of the corresponding rear image. Is stored in the storage unit 11 in association with the set of acquisition times of.

The map creation unit 24 creates a map based on the line segment on the rear image and the line segment on the corresponding front image. For example, according to the bundle adjustment method, the map creation unit 24 maps each set of line segments on the rear image and the line segments on the front image associated with each other, and the line segments corresponding to the set. Create a map by writing in. The map creation unit 24 uses only the set of the line segment on the rear image and the line segment on the front image associated with each other based on the condition (3) in the above Manhattan-World hypothesis for map creation. Is preferable. This prevents lines of structures other than the road surface from being drawn on the map.

FIG. 6 shows an operation flowchart of the alignment process controlled by the control unit 13. The control unit 13 executes the alignment process for each rear image according to this operation flowchart.

The line segment extraction unit 21 extracts a line segment from each of the rear image and the front image (step S101). Then, the line segment extraction unit 21 stores in the storage unit 11 a vector representing the coordinates and directions of the midpoints of each line segment extracted from the front image. Further, the line segment extraction unit 21 passes a vector representing the coordinates and directions of the midpoints of each line segment extracted from the rear image to the projection unit 22.

The projection unit 22 sets a patch centered on the midpoint of the line segment for each line segment extracted from the rear image, and sets one or more regions in the real space corresponding to the patch according to the Manhattan-World hypothesis. (Step S102). Then, the projection unit 22 projects each area in the real space corresponding to the patch on the front image acquired k frames before the acquisition of the rear image for each line segment extracted from the rear image (step S103). ).

The association unit 23 sets a search range for each line segment extracted from the rear image, centering on the projected position on the front image at the center of the region in the real space corresponding to the patch (step S104). The matching unit 23 is the most block-matched between the area around the line segment on the front image within the search range and the projection range of the real space area corresponding to the patch on the front image. The matching line segment is associated with the line segment on the rear image (step S105).

The map creation unit 24 creates a map by writing on the map the line segments corresponding to the set of the line segments on the rear image and the line segments on the front image that are associated with each other (step S106). Then, the control unit 13 ends the alignment process.

FIG. 7A is a diagram showing an example of a road on which a vehicle travels, and FIG. 7B uses an alignment device according to the present embodiment corresponding to the road shown in FIG. 7A. It is a figure which shows an example of the map created by. FIG. 7A shows the road 700 on which the vehicle 10 traveled. On the other hand, in FIG. 7B, the map 710 is a map created by using the alignment device according to the present embodiment, and the dotted line 711 shows the trajectory of the vehicle 10 when the vehicle 10 passes through the road 700. Represent.

As shown in FIGS. 7 (a) and 7 (b), it can be seen that the shape of the road 700 and the road display shown on the road 700 are well reproduced on the map 710.

As described above, the alignment device uses Manhattan- when associating line segments corresponding to the lines of the same structure in real space between two images generated at different timings by different cameras. Based on the World hypothesis, by identifying the area in real space that corresponds to the patch around the line segment on one image, the accuracy of projecting that patch onto the other image is improved. As a result, this alignment device can improve the accuracy of associating line segments with each other. Further, since this alignment device can reduce the number of regions in the real space corresponding to the patch, the amount of calculation can be reduced.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments. For example, the control unit 13 selects a plurality of front images to be associated with line segments on one rear image, and processes the projection unit 22 and the association unit 23 on the plurality of front images. Line segments may be associated with each other. At that time, the projection unit 22 selects only the front image including the projection range of the patch set for the line segment of interest on the rear image among the plurality of front images generated at different times. It may be selected as a front image to be associated with. Alternatively, in the projection unit 22, among a plurality of front images generated at different times, the amount of movement of the vehicle 10 between the time of acquiring the rear image and the time of acquiring the front image is the shooting range of the two cameras. Only the front images included in the predetermined movement distance range that are expected to overlap may be selected as the front images for associating the line segments with each other. The amount of movement of the vehicle 10 is calculated based on the odometry information or the wheel speed at the time of acquiring the rear image and the acquisition time of the front image. Further, the control unit 13 only determines that the vehicle 10 is traveling straight between the time of acquiring the rear image and the time of acquiring the front image to be associated with each other based on the odometry information and the like. The line segment on the rear image and the line segment on the front image may be associated with each other.

According to another modification, the control unit 13 may further have a road surface detection unit that detects the road surface on the rear image. In this case, among the line segments extracted on the rear image, the projection unit 22 corresponds to the patch according to the equations (2) and (4) for the line segments within the range of the detected road surface. The area in the real space may be set. On the other hand, among the line segments extracted on the rear image, the projection unit 22 corresponds to the patch according to the equations (2) and (3) for the line segments outside the detected road surface range. The area on the space may be set.

In addition, in order to detect the road surface from the rear image, the road surface detection unit can use various methods for detecting the area where the road surface is reflected on the image. For example, the road surface detection unit uses a method of detecting a line representing a lane such as a white line on both sides of the own vehicle by, for example, template matching, and using a region sandwiched between the lines representing the lanes on both sides as the road surface. it can. Alternatively, the road surface detection unit is an image generated by two cameras having overlapping shooting ranges (in this case, in addition to the camera 2-2, another camera for shooting the rear of the vehicle 10 is required). Or, a method of detecting the road surface by three-dimensional measurement using two images taken at different times with one camera (for example, Okutomi et al., "Flat part for visual guidance using stereo moving image". Continuous estimation ”, IPSJ Journal, Vol.43, No.4, pp.1061-1069, 2002) may be used. In this case, the alignment device can further limit the area in the real space corresponding to the patch, so that the accuracy of line segment mapping can be further improved.

According to yet another modification, the mapping unit 23 does not calculate the normalized cross-correlation value for the block matching between the patch and the corresponding region on the front image, but instead calculates the absolute difference in the brightness value between the corresponding pixels. The sum of the values may be calculated as the evaluation value. In this case, the associating unit 23 may associate the line segment on the front image with the minimum evaluation value with the line segment on the rear image.

Furthermore, according to another modification, the projection unit 22 has a real-space region corresponding to a patch set for each line segment on the rear image, which is the position of the vehicle 10 and the front image when the rear image is generated. It may be on a plane parallel to the road and perpendicular to the road surface at any position of the road on which the vehicle 10 travels, between the position of the vehicle 10 and the position of the vehicle 10 at the time of generation.
FIG. 8 is a diagram showing an example of a region in the real space corresponding to the patch in this modified example. In this case, from the coordinates of the line segment on the rear image, the angle between the direction toward the line of the structure in the real space corresponding to the line segment and the optical axis of the camera 2-2 at the time of generating the rear image can be known. Therefore, the direction from the vehicle 10 to the line of the structure in the real space corresponding to the line segment at the time of generating the rear image can be known. Therefore, for example, the projection unit 22 refers to the locus information of the vehicle 10 obtained from the odometry information stored in the storage unit 11 from the position of the vehicle 10 at the time of generating the rear image on the real space corresponding to the line segment. The length of the perpendicular line 803 that descends from the straight line 801 toward the midpoint of the structure line to the locus 802 that the vehicle 10 has traveled is the structure in the real space corresponding to the vehicle 10 and the line segment. The position on the line 801 included in the search range of the distance d to the line (for example, it can be 1 m to 20 m as in the above embodiment) is specified. Then, in the projection unit 22, the line 804 of the structure in the real space corresponding to the line segment is parallel to the tangential direction of the locus 802 on the foot of the perpendicular line 803 drawn from the specified position to the locus 802 of the vehicle 10. And it is assumed that it is on a plane 805 perpendicular to the road surface. Since it can be assumed that the vehicle 10 is traveling in a direction parallel to the road, the tangential direction of the locus on the foot of the perpendicular line drawn from the specified position to the locus of the vehicle 10 is parallel to the road. Can be regarded as. In this case, the region in the real space corresponding to the patch set for the line segment is the angle between the extension direction of the road at the time of generating the rear image and the extension direction of the road at the specified position of the vehicle 10. By assuming that the vehicle is tilted from the straight direction and is in a plane perpendicular to the road surface, the coordinates of each point in the real space region corresponding to each pixel in the patch may be obtained.

Furthermore, according to another variant, the alignment device sets a real-space region that corresponds to a patch for each line segment extracted from the forward image, based on the Manhattan-World hypothesis. The same processing may be performed to associate with the line segment on the rear image. Alternatively, the two cameras that generate images that associate line segments with each other are not limited to those facing the front and the rear of the vehicle, but face different directions from each other and one camera as the vehicle travels. It suffices if the area included in the shooting range of is included in the shooting range of the other camera.

For example, assume that one camera is attached to the vehicle 10 so as to face the front of the vehicle 10 and the other camera is attached to the vehicle 10 so as to face the side of the vehicle 10. In this case, a plane orthogonal to the traveling direction of the vehicle 10 and perpendicular to the road surface is also included in the shooting range of the two cameras. Therefore, in such a case, the line segment corresponding to the line on the side wall of the building (for example, the line 503 shown in FIG. 5) orthogonal to the straight line direction of the vehicle 10 shown in (2) above is formed. , It becomes possible to detect in each image obtained by each camera.

Therefore, the projection 22 (2) and (3), or (2) and (4) according to the real corresponding to the patch u _p of each line segment in the image obtained by one camera _{In addition to finding the coordinates P p} of each point in the area in space on the camera coordinate system, in real space corresponding to _{the patch u p} when each line segment is a line corresponding to (2) above. Also find _{the coordinates P p} on the camera coordinate system for each point in the area.

When the line of the structure in the real space corresponding to the line segment is on the side wall of the building orthogonal to the straight direction of the vehicle 10, the camera is used for each point included in the area in the real space corresponding to the patch of the line segment. In the coordinate system, the distance d'from the camera in the traveling direction of the vehicle 10 is equal, so z = d'. Therefore, when the above condition (1) is assumed _{, the coordinates P p} in the camera coordinate system for each point in the real space region corresponding to the _{patch u p} are expressed by the following equations.

Therefore, the projection unit 22 is used in the case where the line of the structure in the real space corresponding to the line segment is on the side wall of the building orthogonal to the straight line direction of the vehicle 10 according to the equations (2) and (14). _{The coordinates P p} on the camera coordinate system of each point in the real-space region corresponding to _{the patch u p} for each line segment on the image obtained by the camera of. Note that d'may be changed in 1 m units in an arbitrary predetermined section, for example, a section of [1,20] (m).

The set of line segments associated between different images according to the above embodiment or modification can be used not only for creating a map as described above, but also for, for example, estimating the self-position of the vehicle 10. In this case, the distance between the positions of the cameras when the two images corresponding to the set of line segments, that is, the baseline length becomes long, so that the structure in the real space corresponding to the set of line segments Improves accuracy in identifying lines.
As described above, those skilled in the art can make various changes within the scope of the present invention according to the embodiment.

1 Alignment device 2-1, 2-2 Camera 3 IMU
4 Wheel speed sensor 5 ECU
6 CAN
11 Storage unit 12 Communication unit 13 Control unit 21 Line segment extraction unit 22 Projection unit 23 Correspondence unit 24 Map creation unit

Claims (10)

At least one first line segment from the first image generated at the first time by the first imaging unit (2-2) mounted so as to face the vehicle (10) in the first direction. A line segment extraction unit (21) that extracts minutes, and
The positional relationship between the vehicle (10) and the structures around the vehicle (10) is defined in the first region in the real space corresponding to the matching range including the first line segment on the first image. The second imaging unit (2) is attached so that the first region faces the vehicle (10) in a second direction different from the first direction. A projection unit that specifies a range on the second image corresponding to the matching range by projecting onto the second image generated at a second time different from the first time according to -1). (22) and
Block matching is performed between the matching range and the second image while changing the relative position of the range on the second image corresponding to the matching range within a predetermined search range on the second image. By doing so, the association unit (23) that identifies the region most similar to the matching range and sets the line segment in the most similar region as the second line segment corresponding to the first line segment.
Alignment device with. The predetermined condition is a condition that the first region corresponding to the matching range is on a plane parallel to the straight-ahead direction of the vehicle (10) and perpendicular to the road surface, or on the road surface. The alignment device according to 1. The predetermined condition is that the first region corresponding to the matching range is placed between the position of the vehicle (10) at the first time and the position of the vehicle (10) at the second time. The alignment device according to claim 1, which is a condition that the vehicle (10) is on a plane parallel to the road and perpendicular to the road surface at any position on the road on which the vehicle (10) travels. The projection unit (22) has the first time and the first time obtained based on sensor information acquired from a sensor that detects the movement amount or speed of the vehicle (10) mounted on the vehicle (10). Based on the amount of movement of the vehicle (10) during the second time, the coordinates of the first region in the coordinate system based on the first imaging unit are referred to the second imaging unit. By converting to the coordinates of the first region and projecting the converted coordinates of the first region onto the second image, the range on the second image corresponding to the matching range is specified. The alignment device according to any one of claims 1 to 3. Among a plurality of images generated by the second imaging unit (2-1) at different times, the projection unit (22) has a range corresponding to the matching range obtained for the image in the image. The alignment device according to any one of claims 1 to 4, wherein the image is the second image when included in the above. The projection unit (22) is the vehicle between the time when the image was generated and the first time among a plurality of images generated by the second imaging unit (2-1) at different times. The alignment device according to any one of claims 1 to 4, wherein the image in which the movement amount of (10) is within a predetermined range is defined as the second image. It further has a road surface detection unit that detects a region in which the road surface is reflected on the first image.
The projection unit (22) specifies that when the first line segment is included in the region in which the road surface is captured, the first region corresponding to the matching range is present on the road surface. 6. The alignment device according to any one of 6. The item according to any one of claims 1 to 7, further comprising a map creation unit (24) for creating a map by writing the first line segment and the line segment corresponding to the second line segment on the map. The alignment device described. At least one first line from the first image generated at the first time by the first imaging unit (2-2) mounted so as to face the vehicle (10) in the first direction. Steps to extract the minutes and
The positional relationship between the vehicle (10) and the structures around the vehicle (10) is defined in the first region in the real space corresponding to the matching range including the first line segment on the first image. The second imaging unit (2) is attached so that the first region faces the vehicle (10) in a second direction different from the first direction. -1) With the step of specifying the range on the second image corresponding to the matching range by projecting onto the second image generated at the second time different from the first time. ,
Block matching is performed between the matching range and the second image while changing the relative position of the range on the second image corresponding to the matching range within a predetermined search range on the second image. By performing this, a step of identifying a region most similar to the matching range and setting a line segment in the most similar region as a second line segment corresponding to the first line segment is performed.
Alignment method including. At least one first line from the first image generated at the first time by the first imaging unit (2-2) mounted so as to face the vehicle (10) in the first direction. Steps to extract the minutes and
The positional relationship between the vehicle (10) and the structures around the vehicle (10) is defined in the first region in the real space corresponding to the matching range including the first line segment on the first image. The second imaging unit (2) is attached so that the first region faces the vehicle (10) in a second direction different from the first direction. -1) With the step of specifying the range on the second image corresponding to the matching range by projecting onto the second image generated at the second time different from the first time. ,
Block matching is performed between the matching range and the second image while changing the relative position of the range on the second image corresponding to the matching range within a predetermined search range on the second image. By performing this, a step of identifying a region most similar to the matching range and setting a line segment in the most similar region as a second line segment corresponding to the first line segment is performed.
A computer program for alignment that allows a computer to run.

Images