JP2017182564A - Positioning device, positioning method, and positioning computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a positioning device capable of accurately associating a plurality of line segments, each of which corresponds to a specific line included in a target structure in a real space and is in an image created by a camera of a plurality of cameras mounted on a car, with each other.SOLUTION: A positioning device comprises: a line segment extraction unit (21) for extracting a first line segment from a first image created at a first time by a first imaging unit (2-2) mounted on a vehicle (10); a projection unit (22) for identifying a first area in a real space corresponding to a matching area including the first line segment in the first image satisfying predetermined conditions for specifying the relationship of the vehicle (10) with structures around the vehicle (10) and for identifying an area in a second image corresponding to the matching area by projecting the first area on the second image created at a second time by a second imaging unit (2-1) mounted on the vehicle (10); and an associating unit (23) for associating a line segment in an area that is most similar to the matching area on the second image with a second line segment corresponding to the first line segment.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an alignment apparatus, an alignment method, and an alignment computer program for associating line segments on images taken by a plurality of in-vehicle cameras directed in different directions.

  In recent years, one or more cameras may be attached to a vehicle for the purpose of driving assistance or the like. In such a case, in order to effectively use the image obtained by the camera, it can be used between a plurality of images generated by different cameras or between a plurality of images generated by the same camera at different positions. It has been proposed to associate feature points or line segments on an image corresponding to the same point or line in space (see, for example, Patent Document 1 and Non-Patent Documents 1 to 3).

  For example, Patent Document 1 discloses an image in which characteristic points such as the center of a manhole cover and the end of a white line on a road surface are photographed by a camera attached to the moving body before and after moving when the moving body goes straight. In the meantime, it has been proposed to associate using object tracking processing.

  Non-Patent Document 1 describes that SURF features are extracted and collated for two consecutive frames between a camera mounted on a vehicle that captures the front and a camera that captures the left or right side. Further, Non-Patent Document 2 proposes that a region around a feature point on one image is deformed by warping to detect a corresponding point on the other image to be associated. Furthermore, Non-Patent Document 3 proposes that line segments detected from different frames are associated with each other in order to create an environment map (Simultaneous Localization and Mapping, SLAM).

JP, 2014-101075, A

Gim Hee Lee et al., "Motion Estimation for Self-Driving Cars With a Generalized Camera", Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on. IEEE, 2013 Simon Baker et al., `` Lucas-Kanade 20 Years On: A Unifying Framework: Part 1 '', International journal of computer vision 56.3, pp. 221-255, 2004 Keisuke Hirose et al., "Fast Line Description for Line-based SLAM" BMVC 2012, 2012

  In the technique disclosed in Patent Literature 1, the association between two images is limited to points on the road surface. However, depending on applications, it may be required to associate feature points corresponding to points other than the road surface between images.

  Further, in the technologies disclosed in Non-Patent Documents 1 to 3, since the viewpoints for viewing the positions of the two images taken at the same position in the real space are different, how the position is seen on the images as well. Is different. For this reason, there is a possibility that sufficient accuracy cannot be obtained in association with the feature points or the line segments.

  Therefore, the present invention can improve the accuracy of the correspondence between the line segments on the image corresponding to the line of the same structure in the real space between the images generated by each of the plurality of cameras mounted on the vehicle. An object is to provide an alignment apparatus, an alignment method, and an alignment computer program.

According to the first aspect of the present invention, an alignment apparatus is provided as one aspect of the present invention. The alignment device according to the present invention includes at least a first image generated at a first time by a first imaging unit (2-2) attached to the vehicle (10) so as to face the first direction. A line segment extraction unit (21) for extracting one first line segment, and a first area in real space corresponding to a matching range including the first line segment on the first image, 10) and the vehicle (10) are specified in accordance with a predetermined condition that defines the positional relationship between the structures around the vehicle (10), and the first region is directed in a second direction different from the first direction with respect to the vehicle (10). By projecting onto the second image generated at the second time different from the first time by the second imaging unit (2-1) attached in this way, the second corresponding to the matching range A projection unit (22) for specifying a range on the image, and a predetermined search range on the second image Then, by performing block matching between the matching range and the second image while changing the relative position of the range on the second image corresponding to the matching range, an area most similar to the matching range is identified, And an associating unit (23) that sets the line segment in the most similar region as the second line segment corresponding to the first line segment.
The alignment apparatus according to the present invention has the above-described configuration, so that an image corresponding to a line of the same structure in real space is generated between images generated by each of a plurality of cameras mounted on the vehicle. The accuracy of correspondence between line segments can be improved.

According to the second aspect of the present invention, the predetermined condition is that the first region corresponding to the matching range is on a plane parallel to the straight traveling direction of the vehicle (10) and perpendicular to the road surface, or on the road surface. The conditions are preferable.
Thereby, the alignment apparatus can appropriately determine the range on the second image corresponding to the matching range.

Alternatively, according to the third aspect, the predetermined condition is that the first region corresponding to the matching range is the position of the vehicle (10) at the first time and the position of the vehicle (10) at the second time. It is preferable that the conditions are such that the vehicle (10) is on a plane parallel to and perpendicular to the road at any position of the road on which the vehicle (10) travels.
As a result, the alignment apparatus appropriately sets the range on the second image corresponding to the matching range even if the vehicle moves in a curved line along the road between the first time and the second time. Can be determined.

According to the fourth aspect of the present invention, the projection unit (22) is obtained based on sensor information acquired from a sensor that detects a movement amount or speed of the vehicle (10) mounted on the vehicle (10). Based on the amount of movement of the vehicle (10) between the first time and the second time, the coordinates of the first region in the coordinate system with the first imaging unit as a reference are used as the reference for the second imaging unit. It is preferable to specify a range on the second image corresponding to the matching range by converting the coordinates of the first region to be converted and projecting the converted coordinates of the first region onto the second image. .
Thereby, even if the vehicle is moving between the production | generation times of two images, the alignment apparatus can specify the range on the 2nd image corresponding to a matching range appropriately.

Furthermore, according to the description of claim 5, the projection unit (22) includes the matching range obtained for the image among the plurality of images generated at different times by the second imaging unit (2-1). When the range corresponding to is included in the image, the image is preferably the second image.
Thereby, the alignment apparatus can determine appropriately the 2nd image which matches line segments.

Furthermore, according to the description of claim 6, the projection unit (22) includes the time when the image is generated among the plurality of images generated at different times by the second imaging unit (2-1). It is preferable that an image in which the amount of movement of the vehicle (10) during the first time falls within a predetermined range is the second image.
Thereby, the alignment apparatus can determine appropriately the 2nd image which matches line segments with simple calculation.

According to a seventh aspect of the present invention, the alignment apparatus further includes a road surface detection unit that detects an area in which the road surface is reflected on the first image, and the projection unit (22) includes the first line. When the minute is included in the area where the road surface is shown, it is preferable to specify that the first area corresponding to the matching range is on the road surface.
Thereby, the alignment apparatus can set the area | region on real space corresponding to a matching range more appropriately.

Furthermore, according to the description of claim 8, the alignment device includes a map creation unit (24) for creating a map by writing the line segments corresponding to the first line segment and the second line segment on the map. Furthermore, it is preferable to have.
Thereby, the alignment apparatus can implement SLAM.

According to a ninth aspect of the present invention, a positioning method is provided as another aspect of the present invention. Such an alignment method includes at least a first image generated at a first time by the first imaging unit (2-2) attached to the vehicle (10) so as to face the first direction. A step of extracting one first line segment, and a first area on the real space corresponding to the matching range including the first line segment on the first image are represented by a vehicle (10) and a vehicle (10 ) Is specified according to a predetermined condition that defines the positional relationship of the surrounding structures, and the first region is attached to the vehicle (10) so as to face a second direction different from the first direction. The range on the second image corresponding to the matching range is specified by projecting onto the second image generated at the second time different from the first time by the second imaging unit (2-1) And matching within a predetermined search range on the second image By performing block matching between the matching range and the second image while changing the relative position of the range on the second image corresponding to the encircling, the region most similar to the matching range is identified, and the most similar And making the line segment in the region a second line segment corresponding to the first line segment.
The alignment method according to the present invention includes the above-described steps, so that an image corresponding to a line of the same structure in real space is generated between images generated by each of a plurality of cameras mounted on the vehicle. The accuracy of correspondence between line segments can be improved.

According to a tenth aspect of the present invention, an alignment computer program is provided as another embodiment of the present invention. The computer program for alignment is based on the first image generated at the first time by the first imaging unit (2-2) attached so as to face the vehicle (10) in the first direction. A step of extracting at least one first line segment, and a first area in the real space corresponding to the matching range including the first line segment on the first image, the vehicle (10) and the vehicle The first region is attached to the vehicle (10) so as to face a second direction different from the first direction, according to a predetermined condition that defines the positional relationship of the surrounding structures of (10). A range on the second image corresponding to the matching range is projected by the second imaging unit (2-1) onto the second image generated at a second time different from the first time. And a predetermined search on the second image By performing block matching between the matching range and the second image while changing the relative position of the range on the second image corresponding to the matching range, the region most similar to the matching range is identified, And a step of causing the computer to execute the step of setting the line segment in the most similar region as the second line segment corresponding to the first line segment.
The computer program for alignment according to the present invention has the above-described instructions, so that an image corresponding to a line of the same structure in real space between images generated by each of a plurality of cameras mounted on the vehicle. The accuracy of the correspondence between the upper line segments can be improved.

  The reference numerals in parentheses attached to the above-described parts are examples that show the correspondence with specific means described in the embodiments described later.

It is a schematic block diagram of the alignment apparatus which concerns on one Embodiment of this invention. It is a functional block diagram of the control part of the alignment apparatus which concerns on one Embodiment of this invention. It is a figure which shows the relationship between the camera coordinate system about each of two cameras, and a vehicle coordinate system. (A)-(c) is a figure which shows an example of the patch which is a matching range which performs block matching, respectively. It is a figure which shows the positional relationship between the point in real space, and a vehicle based on the Manhattan-World hypothesis. It is an operation | movement flowchart of a position alignment process. (A) is a figure showing an example of the road on which the vehicle traveled, and (b) is a map created by using the alignment device according to the present embodiment, corresponding to the road shown in (a). It is a figure showing an example. It is a figure which shows the example of the area | region on the real space corresponding to a patch in a modification.

Hereinafter, an alignment apparatus according to an embodiment will be described with reference to the drawings.
This alignment device is mounted on a vehicle, and the two lines of the same structure in real space (structures) between two images generated at different timings by two cameras directed in different directions. The line segments on the image corresponding to the shape (including not only the line based on the shape of the figure but also the line based on the figure or the character drawn on the structure) are associated with each other. At this time, this alignment apparatus sets the area in the real space corresponding to the patch, which is an area for block matching, set around the line segment extracted from the image generated by one camera. Matching line segments by setting based on the World hypothesis, projecting the area in real space onto the image generated by the other camera, and specifying the target area for block matching on the other image In addition to improving the accuracy, the amount of computation is reduced.

  FIG. 1 is a schematic configuration diagram of an alignment apparatus according to one embodiment. As shown in FIG. 1, the alignment device 1 is mounted on a vehicle 10 and includes cameras 2-1 and 2-2, an inertial measurement device (IMU) 3, a wheel speed sensor 4, and an electronic control unit (ECU) 5. Are connected to each other via a control area network (hereinafter referred to as CAN) 6. In FIG. 1, for convenience of explanation, each component mounted on the vehicle 10 such as the alignment device 1 and the shape, size, and arrangement of the vehicle 10 are different from the actual ones.

  Each of the cameras 2-1 and 2-2 is an example of an imaging unit. In the present embodiment, each of the cameras 2-1 and 2-2 includes a two-dimensional detector composed of an array of photoelectric conversion elements having sensitivity to visible light, such as CCD or C-MOS, and the two-dimensional detector. An imaging optical system that forms an image of a region to be photographed on is provided. The camera 2-1 is attached to, for example, the vehicle interior of the vehicle 10 so that the optical axis of the imaging optical system is substantially parallel to the ground and faces the front of the vehicle 10. Then, the camera 2-1 captures a front area of the vehicle 10 at a predetermined shooting period (for example, 1/30 second), and generates an image in which the front area is captured. On the other hand, the camera 2-2 is attached to the rear end of the vehicle 10 so that the optical axis of the imaging optical system is substantially parallel to the ground and faces the rear of the vehicle 10. And the camera 2-2 image | photographs the back area | region of the vehicle 10 for every predetermined | prescribed imaging | photography period, and produces | generates the image which the back area | region was reflected. The images obtained by the cameras 2-1 and 2-2 may be color images or gray images.

  Each time the cameras 2-1 and 2-2 generate an image, they output the generated image to the alignment apparatus 1.

The alignment apparatus 1 includes a storage unit 11, a communication unit 12, and a control unit 13.
The storage unit 11 includes, for example, a semiconductor memory such as an electrically rewritable nonvolatile memory and a volatile memory. The storage unit 11 includes various programs for controlling the alignment apparatus 1, various types of information used in the alignment process, images generated by the camera 2-1 or the camera 2-2, and the control unit 13. Stores temporary calculation results by.

  The communication unit 12 includes a communication interface that communicates with the cameras 2-1, 2-2, the IMU 3, the wheel speed sensor 4, the ECU 5, and the like through the CAN 6 and its control circuit. The communication unit 12 receives images from the cameras 2-1 and 2-2 and passes the images to the control unit 13. In addition, the communication unit 12 acquires odometry information representing the speed and movement amount of the vehicle 10 from the IMU 3 or acquires the wheel speed from the wheel speed sensor 4 for each period of executing the alignment process. It passes to the control unit 13.

The control unit 13 includes one or a plurality of processors (not shown) and their peripheral circuits. And the control part 13 controls the alignment apparatus 1 whole.
FIG. 2 shows a functional block diagram of the control unit 13. As illustrated in FIG. 2, the control unit 13 includes a line segment extraction unit 21, a projection unit 22, an association unit 23, and a map creation unit 24. Each of these units included in the control unit 13 is implemented as, for example, a functional module realized by a computer program executed on a processor included in the control unit 13.

In the present embodiment, the control unit 13 displays the rear of the vehicle 10 at a certain time n (here, the time is expressed in units of frame intervals, and the time in units of seconds, minutes, etc. is referred to as real time). The line segment on the image generated by the camera 2-2 that captures the front side of the vehicle 10 at the time (nk) k frames before the time. Correlate with. This is because, when the vehicle 10 is moving forward, the range photographed at the time n by the camera 2-2 for photographing the rear of the vehicle 10 is to photograph the front of the vehicle 10 at a time (nk) before that. This is because it is assumed that there is a portion that overlaps the range photographed by the camera 2-1.
Hereinafter, for convenience of explanation, an image generated by the camera 2-1 is referred to as a front image, and an image generated by the camera 2-2 is referred to as a rear image.

Each time the control unit 13 receives a rear image from the camera 2-2, the line segment extraction unit 21 uses a road surface, a road sign, a signboard, or a building in the shooting area of the camera 2-2 from the rear image. A line segment corresponding to the line of the structure is extracted. For this purpose, the line segment extraction unit 21 detects a line segment on the rear image using, for example, a line segment detector. The line segment extraction unit 21 is, for example, RG von Gioi et al., “A fast line segment detector with a false detection control”, IEEE Transactions on Pattern Analysis & Machine Intelligence, (4), 2008, pp. The algorithm disclosed in .722-732 can be used. Alternatively, the line segment extraction unit 21 may use another line segment detector that can detect a line segment from an image.
The line segment extraction unit 21 calculates, for each rear image, a vector representing the acquisition time of the rear image and the coordinates, direction, and length of the midpoint of the line segment on the rear image representing the line segment extracted from the rear image. It passes to the projection unit 22.

  Similarly, every time the control unit 13 receives a front image from the camera 2-1, the line segment extraction unit 21 extracts a line segment from the front image. Then, the line segment extraction unit 21 stores a vector representing the coordinates, direction, and length of the midpoint of the extracted line segment in the storage unit 11 together with the acquisition time of the front image.

For each line segment on the rear image, the projection unit 22 sets a block matching patch centered on the line segment, and specifies an area corresponding to the patch in real space. Then, the projection unit 22 projects the corresponding area of the patch in the real space on the front image acquired k frames before the rear image.
This projection processing requires conversion between coordinates on the image and coordinates in the real space. First, a coordinate system used for the conversion will be described.

  FIG. 3 is a diagram illustrating the relationship between the camera coordinate system and the vehicle coordinate system for each of the two cameras. A camera coordinate system (hereinafter referred to as a forward camera coordinate system for convenience) C1, which is a coordinate system in the real space with the camera 2-1 as a reference, uses the image-side focal point of the camera 2-1 as an origin and a straight traveling direction of the vehicle 10 Is a coordinate system with the z axis, the direction parallel to the road surface and perpendicular to the z axis as the x axis, and the direction perpendicular to the road surface as the y axis. With respect to the z axis, the forward direction of the vehicle 10 is positive, and the positive direction of the x axis is defined by the right-handed coordinate system. Further, a camera coordinate system (hereinafter referred to as a rear camera coordinate system for convenience) C2 which is a coordinate system in the real space with the camera 2-2 as a reference has an image side focal point of the camera 2-2 as an origin, and the vehicle 10 This is a coordinate system in which the straight direction is the z axis, the direction parallel to the road surface and perpendicular to the z axis is the x axis, and the direction perpendicular to the road surface is the y axis. For the z axis, the direction toward the rear of the vehicle 10 is positive, and the positive direction of the x axis is defined by the right-handed coordinate system.

  The vehicle coordinate system V, which is a coordinate system in the real space with respect to the vehicle 10, is an arbitrary point of the vehicle 10, for example, a midpoint between front wheels, a midpoint between rear wheels, or a front side The midpoint between the midpoint between the wheels and the midpoint between the rear wheels is the origin, and the straight direction of the vehicle 10 is the z axis, the direction parallel to the road surface and perpendicular to the z axis is the x axis, and the direction perpendicular to the road surface Is the coordinate system with y-axis. With respect to the z axis, the forward direction of the vehicle 10 is positive, and the positive direction of the x axis is defined by the right-handed coordinate system. In these coordinate systems, the angle formed with the z axis in the xz plane is represented by θ.

  FIG. 4A to FIG. 4C are diagrams illustrating examples of patches that are matching ranges in which block matching is performed. In the example shown in FIG. 4A, the patch 401 has the same length as the line segment 410 along the direction parallel to the line segment 410 of interest, and a predetermined width in the direction orthogonal to the line segment 410. It is set as a rectangular area having (for example, 10 to 20 pixels). In the example shown in FIG. 4B, the patch 402 has a predetermined width (for example, 10 to 20 pixels) in a direction orthogonal to the focused line segment 410 and is parallel to the line segment 410. A rectangular region longer than the line segment 410 by a predetermined length (for example, 10 to 20 pixels) is set. Furthermore, in the example shown in FIG. 4C, the patch 403 is set as a region where the distance to the focused line segment 410 is a predetermined value (for example, 10 to 20 pixels) or less.

ここで、(u_m,2,v_m,2)は、後方画像上での着目する線分の中点の座標を表し、α、βは、それぞれ、後方画像上でのパッチu_p内の画素の線分の中点からの水平方向及び垂直方向の距離を表す。なお、αの取り得る値の範囲[α₀, α₁]、及び、βの取り得る値の範囲[β₀, β₁]は、それぞれ、垂直方向の位置及び水平方向の位置についての関数として表される。そしてその関数は、線分の向き、及び、図４（ａ）〜図４（ｃ）に示される、パッチの形状に応じて決定される。 In this embodiment, the patch is u _p a matching range for performing block matching, for example, is defined as follows:.

_{Here, (u m, 2, v} m, 2) represents the coordinates of the midpoint of the target line segment on the rearward image, alpha, beta, respectively, in the patch u _p on the rear image It represents the distance in the horizontal and vertical directions from the midpoint of the line segment of the pixel. Note that the range of values [α ₀ , α ₁ ] that can be taken by α and the range [β ₀ , β ₁ ] that can be taken of β are respectively expressed as functions of the vertical position and the horizontal position. expressed. The function is determined according to the direction of the line segment and the shape of the patch shown in FIGS. 4 (a) to 4 (c).

The projection 22 from the patch u _p to be set on the basis of the line segment on the rearward image, corresponding in the real space in accordance with a predetermined condition which defines the positional relationship of the structure in the vicinity of the vehicle 10 and the vehicle 10 Identify the area. Then, the projection unit 22 for the corresponding region, the rear camera coordinate system → the vehicle coordinate system at time n → the world coordinate system that is the absolute coordinate in real space → the vehicle coordinate system at time (nk) → the front camera coordinate system → the front image Project in the order above.
Thus is described first coordinates are derived in the camera coordinate system for the area in the real space corresponding to the patch u _p.

ここで、(c_u,c_v)は後方画像の中心の座標、すなわち、カメラ２−２の光軸上の位置に相当する後方画像上の位置の座標を表す。またf_u、f_vは、それぞれ、後方画像上の画素の水平方向及び垂直方向のサイズを考慮した、水平方向及び垂直方向に相当するカメラ２−２の焦点距離を表す。そして(u_n、v_n)は、線分の中点(u_m,2,v_m,2)に対応する正規化座標であり、U_n=(u_n+α_n,v_n+β_n)は、正規化座標系でのパッチu_p内の各画素の座標である。そして(x,y,z)は、U_nに対応するカメラ座標系上の点の座標である。 In converting the coordinates on the rear image or the front image (two-dimensional coordinates) to the coordinates (three-dimensional coordinates) of the camera coordinate system, a pinhole camera model is applied in the present embodiment. In this case, the coordinates of each pixel in the patch u _p on the rearward image, is converted into the coordinate U _n on the normalized coordinate system in accordance with the following equation.

Here, (c _u , c _v ) represents the coordinates of the center of the rear image, that is, the coordinates of the position on the rear image corresponding to the position on the optical axis of the camera 2-2. Further, f _u and f _v represent the focal lengths of the camera 2-2 corresponding to the horizontal direction and the vertical direction, respectively, in consideration of the size of the pixel on the rear image in the horizontal direction and the vertical direction. And (u _n , v _n ) is a normalized coordinate corresponding to the midpoint of the line segment (u _{m, 2} , v _{m, 2} ), and U _n = (u _n + α _n , v _n + β _n ) are the coordinates of each pixel in the patch u _p in a normalized coordinate system. (X, y, z) are the coordinates of a point on the camera coordinate system corresponding to U _n .

In the present embodiment, the Manhattan-World hypothesis is introduced as a predetermined condition that defines the positional relationship between the vehicle 10 and the structures around the vehicle 10. Based on the Manhattan-World hypothesis, the real space region corresponding to the patch for each line segment is specified. In the Manhattan-World hypothesis, the line of the structure in the real space corresponding to the line segment on the image is one of the following three types.
(1) A line on the side wall of the building parallel to the straight direction of the vehicle 10 (2) A line on the side wall of the building orthogonal to the straight direction of the vehicle 10 (3) A line on the road surface or a line segment on the image It is assumed that the line around the structure in real space is a plane.

  FIG. 5 is a diagram showing the positional relationship between the line of the structure in the real space and the vehicle based on the Manhattan-World hypothesis according to the present embodiment. In the present embodiment, since the camera 2-1 faces the front of the vehicle 10, while the camera 2-2 faces the rear of the vehicle 10, both of the lines corresponding to the above (2) are used. The line segment on one image is not associated with the line segment on the other image. Therefore, a line segment corresponding to line 501 on the side wall parallel to the straight traveling direction of the vehicle 10 corresponding to (1) or a line segment corresponding to line 502 on the road surface corresponding to (3) is Corresponding between the image and the back image.

When the line of the structure in the real space corresponding to the line segment on the image is on the side wall of the building 500 parallel to the straight traveling direction of the vehicle 10, as in the line 501, on the real space corresponding to the patch of the line segment For each point included in the region, x = d in the camera coordinate system. In addition, d is the distance to the line in the real space corresponding to the vehicle 10 and its line segment. Therefore, if the above condition (1) is assumed, the coordinates P _p in the camera coordinate system for each point in the area in the real space corresponding to the patch u _p is expressed by the following equation.

On the other hand, when the line of the structure in the real space corresponding to the line segment on the image is on the road surface, as in the point 502, for each point included in the area on the real space corresponding to the patch of the line segment In the camera coordinate system, y = h. Note that h is the height from the road surface to the camera 2-2 and is stored in the storage unit 11 in advance. Therefore, if the above condition (3) is assumed, the coordinates P _p in the camera coordinate system for each point in the area in the real space corresponding to the patch u _p is expressed by the following equation.

Projection 22 (2) and (3), or (2) and (4) according to wherein each of the regions in the real space corresponding to the patch u _p of each line segment on the rearward image A coordinate P _p on the camera coordinate system of the point is obtained. Note that d is changed in units of 1 m in an arbitrary predetermined section, for example, a section of [1,20] (m). That is, for each line segment, for each value of d, 20 areas of coordinates P _p obtained based on the expressions (2) and (3) and 1 obtained based on the expressions (2) and (4) The coordinates P _p of the individual areas are obtained.

  Note that the projection unit 22 may set the section d to be narrower than the predetermined section by bundle adjustment. In this case, the projection unit 22, for example, between the two rear images obtained by photographing the line of the structure in the real space corresponding to the line segment on the image at different times with the camera 2-2. Specify by associating line segments with. Then, the projection unit 22 performs a point in the real space corresponding to a point (for example, a midpoint) on the line segment from the vehicle 10 by triangulation using the positions of the two vehicles corresponding to the generation of the two rear images. And the interval d may be set so as to include the measured distance.

ここで、u_n'は、座標P_pに対応する、前方画像上の領域の座標である。また関数π_vc1()は、車両座標系からカメラ２-１のカメラ座標系への変換関数であり、関数π_vc2()は、車両座標系からカメラ２-２のカメラ座標系への変換関数である。なお、これらの逆関数は、それぞれ、対応するカメラ座標系から車両座標系への変換関数となる。なお、車両座標系とカメラ座標系間の変換は、それぞれの座標系の原点の差に相当する並進行列T_vc1,T_vc2と、車両１０の直進方向とカメラの光軸間の傾きに相当する回転行列R_vc1,R_vc2により表される。 For the patch of each line segment on the rear image, the projection unit 22 projects a warping function for projecting each of the areas on the real space corresponding to the patch onto the front image.

Here, u _n ′ is a coordinate of a region on the front image corresponding to the coordinate P _p . The function π _vc1 () is a conversion function from the vehicle coordinate system to the camera coordinate system of the camera 2-1, and the function π _vc2 () is a conversion function from the vehicle coordinate system to the camera coordinate system of the camera 2-2. It is. Each of these inverse functions is a conversion function from the corresponding camera coordinate system to the vehicle coordinate system. The conversion between the vehicle coordinate system and the camera coordinate system corresponds to the _{parallel progression trains} T _vc1 , T _vc2 corresponding to the difference between the origins of the respective coordinate systems, and the inclination between the straight traveling direction of the vehicle 10 and the optical axis of the camera. _Represented by rotation matrices R _vc1 and R _vc2 .

The function π _ci () is a conversion function from the camera coordinate system to the coordinates on the image, and the coordinates (u _n , v _n ) on the image from the camera coordinates (x, y, z) in equation (2). Equivalent to conversion to

The function π ⁿ _wv () is a conversion function from the world coordinate system to the vehicle coordinate system at time n. On the other hand, the inverse function π ⁿ _wv ⁻¹ () is a conversion function from the vehicle coordinate system to the world coordinate system at time n.

ここで、u_nは、オドメトリ情報であり、v_n,right、v_n,leftは、それぞれ、時刻nにおける車両１０の右側後輪の車輪速、及び左側後輪の車輪速を表す。そしてω_n、v_nは、それぞれ、時刻ｎにおける車両１０の角速度及び速度を表す。またd_aは、車両１０の左右の後輪間の間隔である。さらに、Δtは、時刻(n-k)と時刻n間の実時刻の差であり、Δt=k×fで表される。なお、fは、カメラ２−１、２−２のフレーム周期（撮影間隔）である。またε_xは、移動量の誤差を確率分布で表す誤差モデルである。本実施形態では、誤差モデルε_xは、ガウス分布で表されるものとし、その共分散Σ_xは、x軸方向、z軸方向及びθに関して、以下のように定義される。

ここで、σ_x、σ_z及びσ_θは、例えば、オドメトリ情報あるいは車輪速の誤差に基づいて設定され、例えば、σ_x=σ_z=0.1[m]であり、σ_θ=2π/360[rad]である。 Details of the function π ⁿ _wv () will be described below. Let X _n = ^t (x _n , z _n , θ _n ) be a position vector representing the position and orientation of the vehicle 10 at time n in the world coordinate system. In this case, X _n is the time (nk) of the vehicle 10 at the position ^{_{X nk = t (x nk,}} z nk, θ nk) between the relationship is established as represented by the following formula.

Here, u _n is odometry information, and v _{n, right} and v _{n, left} represent the wheel speed of the right rear wheel and the left rear wheel of the vehicle 10 at time n, respectively. Ω _n and v _n represent the angular velocity and speed of the vehicle 10 at time n, respectively. D _a is the distance between the left and right rear wheels of the vehicle 10. Furthermore, Δt is the difference in real time between time (nk) and time n, and is represented by Δt = k × f. Note that f is the frame period (photographing interval) of the cameras 2-1 and 2-2. Further, ε _x is an error model that expresses an error in the movement amount by a probability distribution. In the present embodiment, the error model ε _x is represented by a Gaussian distribution, and the covariance Σ _x is defined as follows with respect to the x-axis direction, the z-axis direction, and θ.

Here, σ _x , σ _z, and σ _θ are set based on, for example, odometry information or wheel speed error. For example, σ _x = σ _z = 0.1 [m], and σ _θ = 2π / 360 [ rad].

したがって、関数πⁿ _wv()は、次式で表される。

ここで、P_wは、世界座標系での着目点の座標を表し、P_vは、P_wに対応する点の車両座標系での座標を表す。なお、（５）式における、π^n-k _wv()では、x_n=x_n-k、z_n=z_n-k、θ_n=θ_n-kとすればよい。 The rotation matrix R ^t and the parallel progression T ^t representing the transformation from the world coordinate system to the vehicle coordinate system at time n are expressed by the following equations using the position vector X _n of the vehicle 10 at time n.

Therefore, the function π ⁿ _wv () is expressed by the following equation.

Here, P _w represents the coordinates of the point of interest in the world coordinate system, and P _v represents the coordinates of the point corresponding to P _w in the vehicle coordinate system. In π ^nk _wv () in equation (5), x _n = x _nk , z _n = z _nk , θ _n = θ _nk may be set.

ここでRⁱ _vwは、時刻iにおける、変換式πⁱ _wvに含まれる回転行列Rの逆行列であり、（８）式及び（９）式に従って算出される。 Further, in order to more accurately identify the corresponding line segment on the front image with reference to the line segment on the rear image, the line segment on the front image corresponding to the direction of the line segment on the rear image. It is preferable that the orientation of is required. In addition, a line segment in the real space includes a three-dimensional vector r (rx, px) representing the three-dimensional coordinates p (px, py, pz) of the point on the line segment (for example, the midpoint) and the direction of the line segment. ry, rz). Therefore, the conversion formula for converting the three-dimensional vector r representing the direction of the line segment in the world coordinate system into the vector A ₁ of the camera coordinate system based on the camera 2-1, and the camera 2-2 as a reference. A conversion formula for converting to the vector A _{2 in the} camera coordinate system is expressed by the following formula.

Here, R ⁱ _vw is an inverse matrix of the rotation matrix R included in the conversion equation π ⁱ _wv at time i, and is calculated according to the equations (8) and (9).

ここで、q₁(x₁,y₁,z₁)は、カメラ２−１を基準とするカメラ座標系での線分上の点pの座標を表し、q₂(x₂,y₂,z₂)は、カメラ２−２を基準とするカメラ座標系での線分上の点pの座標を表し、それぞれ、（５）〜（９）式に従って算出される。したがって、投影部２２は、後方画像上の各線分について、その線分の向きを表すベクトルを、カメラ２−２に関する（１０）式及び（１１）式の逆変換に従って世界座標系でのベクトルに変換し、その後、カメラ２−１に関する（１０）式及び（１１）式に従って、世界座標系でのベクトルから、前方画像上での対応ベクトルを求めることができる。 In addition, an expression for projecting the camera coordinate system vector A ₁ to the front image and calculating the corresponding vector a ₁ on the front image, and a camera coordinate system vector A ₂ to the rear image and corresponding on the rear image An expression for calculating the vector a ₂ is expressed by the following expression.

Here, q ₁ (x ₁ , y ₁ , z ₁ ) represents the coordinates of the point p on the line segment in the camera coordinate system with respect to the camera 2-1, and q ₂ (x ₂ , y ₂ , z ₂ ) represents the coordinates of the point p on the line segment in the camera coordinate system with respect to the camera 2-2, and is calculated according to equations (5) to (9), respectively. Therefore, the projection unit 22 converts the vector representing the direction of each line segment on the rear image into a vector in the world coordinate system according to the inverse transformation of the expressions (10) and (11) regarding the camera 2-2. Then, the corresponding vector on the front image can be obtained from the vector in the world coordinate system according to the equations (10) and (11) regarding the camera 2-1.

As described above, corresponding to the patch u _p, a region in the real space by projecting into the forward image, the position of the forward image corresponding to each pixel in the patch u _p, i.e., front patch u _p An area projected on the image is obtained.
Projection unit 22, corresponding to each pixel in the patch u _p of each line segment on the rearward image, k frames before the pixel in the forward image coordinates and the line segment orientation to the associated portion 23 of the vector representing the Output.

Associating unit 23, the particular corresponding to the patch u _p of each line segment on the rearward image, and the region on the k frames before the front image, the area to be similar to each other by performing block matching between the front image By doing so, the line segment on the front image is associated with each line segment on the rear image.

なお、wは、（５）式の右辺を表す。 In the present embodiment, the associating unit 23 sets, for each line segment on the rear image, a search range for searching for a line segment on the corresponding front image as an error ellipse based on the error model. At that time, the error ellipse is determined by a covariance matrix Σ _F calculated by the following equation.

Note that w represents the right side of equation (5).

ここで、χ²は、χ二乗分布であり、誤差楕円の信用区間を表す。また、前方画像上の水平方向に対して誤差楕円の長径がなす角θ_εは、次式で表される。

Associating unit 23 calculates eigenvalues lambda ₁ of the covariance matrix sigma _F, and lambda _2, the eigenvector v _1, v _2. However, the larger eigenvalue is λ ₁ . At this time, the major axis a and minor axis b of the error ellipse are expressed by the following equations.

Here, χ ² is a χ square distribution and represents the confidence interval of the error ellipse. Further, the angle θ _ε formed by the major axis of the error ellipse with respect to the horizontal direction on the front image is expressed by the following equation.

ここで、(u'_c,v'_c)は、後方画像上の着目する線分に対応する前方画像上での線分の中点の座標を表す。そして(u',v')は、前方画像上で検出された線分の中点の座標である。 For each line segment on the rear image, the associating unit 23 is within an error ellipse that is a search range on the front image k frames before, and the angle formed with the line segment when projected onto the front image is predetermined. A line segment on the front image having an angle (for example, 3 ° to 5 °) or less is detected as a candidate associated with the line segment on the rear image. Note that the associating unit 23 determines that the line segment on the forward image that satisfies the following expression is included in the search range.

Here, (u ′ _c , v ′ _c ) represents the coordinates of the midpoint of the line segment on the front image corresponding to the line segment of interest on the rear image. (U ′, v ′) is the coordinates of the midpoint of the line segment detected on the front image.

  For each line segment on the rear image, the associating unit 23 calculates a normalized cross-correlation value by performing block matching for each line segment within the search range on the front image before k frames. At this time, the associating unit 23 determines the center of the area on the front image on which the patch is projected, that is, the position at which the line segment on the rear image is projected on the front image and the front image on which block matching is performed. Pixels on the front image corresponding to each pixel in the patch on the rear image by translating the position of each pixel in the patch projected on the front image so as to cancel the position difference between the line segments of Should be specified. The association unit 23 can calculate a normalized cross-correlation value based on the luminance value of each pixel in the patch on the rear image and the luminance value of the pixel on the corresponding front image. The associating unit 23 associates the line segment on the front image having the maximum normalized cross-correlation value with the line segment of interest on the rear image. Note that the association unit 23 may associate the two line segments only when the maximum value of the normalized cross-correlation values is equal to or greater than a predetermined threshold (for example, 0.6 to 0.7).

  In order to reduce the amount of calculation, the association unit 23 sets the search range to a predetermined range that is centered on the coordinates on the front image corresponding to the midpoint of the line segment on the rear image. It may be set. In this case, the predetermined range may be, for example, 2 to 3 times the horizontal size of the patch in the horizontal direction and 2 to 3 times the vertical size of the patch in the vertical direction.

  Note that, according to the modification, since the patch shape itself includes information on the direction of the line segment on the rear image, the associating unit 23 is in the search range on the front image before k frames. All the line segments on the front image may be candidates to be associated with the line segments on the rear image.

  Further, according to another modification, the associating unit 23 translates the patch so that each pixel in the corresponding search range corresponds to the center of the patch for each line segment on the rear image, respectively. The cross-correlation value is calculated, and the pixel having the maximum normalized cross-correlation value is set as the midpoint of the line segment on the front image corresponding to the line segment of interest on the rear image, and the direction of the line segment of interest is calculated. The direction on the corresponding front image may be the direction of the corresponding line segment. In this case, the line segment extraction unit 21 may not extract a line segment from the front image.

  The associating unit 23 includes a set of vectors representing the coordinates and orientations of the midpoints of the respective line segments on the front image associated with the line segments on the rear image, and the corresponding rear image acquisition time and front image. Are stored in the storage unit 11 in association with each other.

  The map creation unit 24 creates a map based on the line segment on the rear image and the corresponding line segment on the front image. For example, according to the bundle adjustment method, the map creating unit 24 maps the line segments corresponding to the set for each line segment on the rear image and the line segment on the front image that are associated with each other. Create a map by writing to. Note that the map creation unit 24 uses only the pair of the line segment on the rear image and the line segment on the front image associated with each other based on the condition (3) in the Manhattan-World hypothesis for map creation. It is preferable. Thereby, it is suppressed that the line of structures other than a road surface is written in a map.

  FIG. 6 shows an operation flowchart of the alignment process controlled by the control unit 13. In addition, the control part 13 performs the alignment process according to this operation | movement flowchart for every back image.

  The line segment extraction unit 21 extracts line segments from each of the rear image and the front image (step S101). The line segment extraction unit 21 stores a vector representing the coordinates and orientation of the midpoint of each line segment extracted from the front image in the storage unit 11. Further, the line segment extraction unit 21 passes a vector representing the coordinates and direction of the midpoint of each line segment extracted from the rear image to the projection unit 22.

  For each line segment extracted from the rear image, the projection unit 22 sets a patch centered on the midpoint of the line segment, and sets one or more regions on the real space corresponding to the patch according to the Manhattan-World hypothesis. (Step S102). Then, the projection unit 22 projects each region in the real space corresponding to the patch on the front image acquired k frames before the rear image is acquired for each line segment extracted from the rear image (step S103). ).

  For each line segment extracted from the rear image, the associating unit 23 sets a search range centered on the projection position on the front image at the center of the area on the real space corresponding to the patch (step S104). Then, the associating unit 23 performs the block matching between the area around the line segment on the front image within the search range and the projection range of the area on the real space corresponding to the patch on the front image. The matching line segment is associated with the line segment on the rear image (step S105).

  The map creation unit 24 creates a map by writing a line segment corresponding to a set of line segments on the rear image and line segments on the front image, which are associated with each other, on the map (step S106). Then, the control unit 13 ends the alignment process.

  FIG. 7A is a diagram illustrating an example of a road on which a vehicle has traveled, and FIG. 7B uses the alignment apparatus according to the present embodiment corresponding to the road illustrated in FIG. It is a figure showing an example of the map created in this way. FIG. 7A shows a road 700 on which the vehicle 10 has traveled. On the other hand, in FIG. 7B, a map 710 is a map created by using the alignment apparatus according to the present embodiment, and a dotted line 711 indicates the trajectory of the vehicle 10 when the vehicle 10 passes the road 700. Represent.

  As shown in FIGS. 7A and 7B, it can be seen that the shape of the road 700 and the road display shown on the road 700 are reproduced well on the map 710.

  As described above, when the alignment device associates line segments corresponding to lines of the same structure in the real space between two images generated at different timings by different cameras, the Manhattan- Based on the World hypothesis, an area in real space corresponding to a patch around a line segment on one image is specified, thereby improving the accuracy in projecting the patch onto the other image. Thereby, this alignment apparatus can improve the precision of matching of line segments. In addition, this alignment apparatus can reduce the number of areas in the real space corresponding to the patches, and thus the amount of calculation can be reduced.

  The preferred embodiments of the present invention have been described above, but the present invention is not limited to these embodiments. For example, the control unit 13 selects a plurality of front images to be associated with a line segment on one rear image, performs the processing of the projection unit 22 and the association unit 23 on the plurality of front images, The line segments may be associated with each other. At that time, the projection unit 22 selects only the front image including the projection range of the patch set for the target line segment on the rear image from among the plurality of front images generated at different times. You may select as a front image which matches about. Alternatively, the projection unit 22 has a movement range of the vehicle 10 between the time when the rear image is acquired and the time when the front image is acquired, among the plurality of front images generated at different times. Only a front image included in a predetermined moving distance range that is assumed to overlap may be selected as a front image for associating line segments. Note that the movement amount of the vehicle 10 is calculated based on odometry information or wheel speed of the acquisition time of the rear image and the acquisition time of the front image. Furthermore, only when the control unit 13 determines that the vehicle 10 is traveling straight between the acquisition of the rear image and the acquisition of the front image to be associated based on the odometry information. You may perform the matching of the line segment on a back image, and the line segment on a front image.

  According to another modification, the control unit 13 may further include a road surface detection unit that detects a road surface on the rear image. In this case, the projection unit 22 corresponds to the patch according to the equations (2) and (4) for the line segments within the detected road surface range among the line segments extracted on the rear image. An area in real space may be set. On the other hand, among the line segments extracted on the rear image, the projection unit 22 performs the actual processing corresponding to the patch according to the expressions (2) and (3) for the line segments outside the detected road surface range. What is necessary is just to set the area | region on space.

  The road surface detection unit can use various methods for detecting a region where the road surface is shown on the image in order to detect the road surface from the rear image. For example, the road surface detection unit detects a line representing a lane such as a white line on both sides of the host vehicle by, for example, template matching, and uses a method in which an area sandwiched between lines representing the lanes on both sides is used as a road surface. it can. Alternatively, the road surface detection unit generates images respectively generated by two cameras whose shooting ranges overlap (in this case, another camera for shooting the rear of the vehicle 10 is required in addition to the camera 2-2). Alternatively, a method for detecting a road surface by three-dimensional measurement using two images taken at different times by a single camera (for example, Okutomi et al., “Flat part for visual guidance using stereo video images”). Continuous estimation ”, IPSJ Journal, Vol.43, No.4, pp.1061-1069, 2002) may be used. In this case, since the alignment apparatus can further limit the area in the real space corresponding to the patch, it is possible to further improve the accuracy of association between line segments.

  According to yet another modification, the associating unit 23 calculates the absolute value of the difference in luminance value between corresponding pixels instead of calculating a normalized cross-correlation value for block matching between corresponding regions on the patch and the front image. The sum of the values may be calculated as the evaluation value. In this case, the associating unit 23 may associate the line segment on the front image with the smallest evaluation value with the line segment on the rear image.

Furthermore, according to another modification, the projecting unit 22 determines that the area in the real space corresponding to the patch set for each line segment on the rear image is the position of the vehicle 10 and the front image when the rear image is generated. It may be on a plane parallel to the road at any position of the road on which the vehicle 10 travels and perpendicular to the road surface between the position of the vehicle 10 at the time of generation.
FIG. 8 is a diagram showing an example of an area on the real space corresponding to the patch in this modification. In this case, the angle between the direction of the line of the structure in the real space corresponding to the line segment and the optical axis of the camera 2-2 when generating the rear image can be determined from the coordinates of the line segment on the rear image. Therefore, the direction from the vehicle 10 toward the line of the structure in the real space corresponding to the line segment when the rear image is generated can be known. Therefore, for example, the projection unit 22 refers to the trajectory information of the vehicle 10 obtained from the odometry information stored in the storage unit 11 and refers to the real space corresponding to the line segment from the position of the vehicle 10 when the rear image is generated. The length of a perpendicular line 803 that extends from a straight line 801 in the direction toward the middle point of the structure line to a locus 802 on which the vehicle 10 has traveled is the length of the structure in the real space corresponding to the vehicle 10 and the line segment. A position on the line 801 that is included in the search range of the distance d to the line (for example, can be set to 1 to 20 m as in the above embodiment) is specified. In the projection unit 22, the line 804 of the structure in the real space corresponding to the line segment is parallel to the tangential direction of the trajectory 802 at the foot of the perpendicular line 803 that descends from the specified position to the trajectory 802 of the vehicle 10. Assume that the surface 805 is perpendicular to the road surface. Since it can be assumed that the vehicle 10 is traveling in a direction parallel to the road, the tangential direction of the trajectory of the perpendicular foot drawn from the specified position to the trajectory of the vehicle 10 is parallel to the road. Can be considered. In this case, the region in the real space corresponding to the patch set for the line segment is the angle of the vehicle 10 by the angle between the road extension direction when the rear image is generated and the road extension direction at the specified position. The coordinates of each point in the area in the real space corresponding to each pixel in the patch may be obtained by assuming that it is inclined from the straight direction and is in a plane perpendicular to the road surface.

  Furthermore, according to another modification, the alignment device sets an area in the real space corresponding to a patch for each line segment extracted from the forward image based on the Manhattan-World hypothesis as described above. Similar processing may be performed to associate the line segment on the rear image. Alternatively, the two cameras that generate the images for associating the line segments are not limited to those facing the front and the rear of the vehicle, but are directed in different directions, and one of the cameras as the vehicle travels It is only necessary that the area included in the imaging range is attached so as to be included in the imaging range of the other camera.

  For example, it is assumed that one camera is attached to the vehicle 10 so as to face the front of the vehicle 10, and the other camera is attached so as to face the side of the vehicle 10. In this case, a plane perpendicular to the traveling direction of the vehicle 10 and perpendicular to the road surface is also included in the shooting range of the two cameras. Therefore, in such a case, the line segment corresponding to the line (for example, line 503 shown in FIG. 5) on the side wall of the building orthogonal to the straight traveling direction of the vehicle 10 shown in (2) above is obtained. It becomes possible to detect in each image obtained by each camera.

Therefore, the projection 22 (2) and (3), or (2) and (4) according to the real corresponding to the patch u _p of each line segment in the image obtained by one camera not only determine the coordinates P _p on the camera coordinate system of each point in the region of the space, each line segment, in the real space corresponding to the patch u _p in the case where a line corresponding to the above (2) A coordinate P _p on the camera coordinate system of each point in the region is also obtained.

When the line of the structure in the real space corresponding to the line segment is on the side wall of the building orthogonal to the straight traveling direction of the vehicle 10, the camera for each point included in the area in the real space corresponding to the patch of the line segment In the coordinate system, since the distance d ′ from the camera in the traveling direction of the vehicle 10 is equal, z = d ′. Therefore, if the above condition (1) is assumed, the coordinates P _p in the camera coordinate system for each point in the area in the real space corresponding to the patch u _p is expressed by the following equation.

Therefore, the projection unit 22 is one in the case where the line of the structure in the real space corresponding to the line segment is on the side wall of the building orthogonal to the straight traveling direction of the vehicle 10 according to the expressions (2) and (14). may be obtained coordinates P _p on the camera coordinate system of each point in the area in the real space corresponding to the patch u _p of each line segment in the image obtained by the camera. Note that d ′ may be changed in units of 1 m in an arbitrary predetermined section, for example, a section of [1,20] (m).

A set of line segments associated with different images according to the embodiment or the modification described above can be used not only for the creation of a map as described above but also for the self-position estimation of the vehicle 10, for example. In this case, since the distance between the positions of the cameras when two images corresponding to the set of line segments are generated, that is, the base line length becomes long, the structure in the real space corresponding to the set of line segments. The accuracy in identifying the line is improved.
As described above, those skilled in the art can make various modifications in accordance with the embodiment to be implemented within the scope of the present invention.

1 Positioning device 2-1, 2-2 Camera 3 IMU
4 Wheel speed sensor 5 ECU
6 CAN
DESCRIPTION OF SYMBOLS 11 Memory | storage part 12 Communication part 13 Control part 21 Line segment extraction part 22 Projection part 23 Correlation part 24 Map preparation part

Claims (10)

At least one first line from the first image generated at the first time by the first imaging unit (2-2) attached to the vehicle (10) so as to face the first direction. A line segment extraction unit (21) for extracting a minute;
The first region in the real space corresponding to the matching range including the first line segment on the first image is the positional relationship between the vehicle (10) and structures around the vehicle (10). And a second imaging unit (2) attached to the vehicle (10) so as to face a second direction different from the first direction. -1), the projection unit that specifies the range on the second image corresponding to the matching range by projecting onto the second image generated at the second time different from the first time (22)
Block matching is performed between the matching range and the second image while changing a relative position of the range on the second image corresponding to the matching range within a predetermined search range on the second image. An association unit (23) that identifies a region most similar to the matching range and sets a line segment in the most similar region as a second line segment corresponding to the first line segment;
An alignment device having   The predetermined condition is a condition that the first region corresponding to the matching range is on a plane parallel to a straight traveling direction of the vehicle (10) and perpendicular to a road surface, or on a road surface. 2. The alignment apparatus according to 1.   The predetermined condition is that the first region corresponding to the matching range is between the position of the vehicle (10) at the first time and the position of the vehicle (10) at the second time. The alignment apparatus according to claim 1, wherein the alignment device is on a plane parallel to the road and perpendicular to the road surface at any position of the road on which the vehicle (10) travels.   The projection unit (22) is configured to obtain the first time determined based on sensor information acquired from a sensor that detects a moving amount or speed of the vehicle (10) mounted on the vehicle (10) and the Based on the amount of movement of the vehicle (10) during a second time, the coordinates of the first region in the coordinate system with the first imaging unit as a reference are used with the second imaging unit as a reference. A range on the second image corresponding to the matching range is specified by converting the coordinates of the first region onto the second image by converting the converted coordinates of the first region onto the second image. The alignment apparatus according to any one of claims 1 to 3.   The projection unit (22) includes a plurality of images generated at different times by the second imaging unit (2-1) so that a range corresponding to the matching range obtained for the image is within the image. The alignment apparatus according to any one of claims 1 to 4, wherein the image is the second image if included in the image.   The projection unit (22) includes the vehicle between the time when the image is generated and the first time among the plurality of images generated at different times by the second imaging unit (2-1). The alignment apparatus according to any one of claims 1 to 4, wherein an image in which the movement amount of (10) is within a predetermined range is the second image. A road surface detection unit for detecting a region in which the road surface is reflected on the first image;
The said projection part (22) specifies that the said 1st area | region corresponding to the said matching range exists on a road surface, when the said 1st line segment is contained in the area | region where the said road surface is reflected. The alignment apparatus as described in any one of -6.   8. The apparatus according to claim 1, further comprising a map creation unit (24) that creates a map by writing a line segment corresponding to the first line segment and the second line segment on the map. 9. The alignment apparatus as described. At least one first line from the first image generated at the first time by the first imaging unit (2-2) attached to the vehicle (10) so as to face the first direction. Extracting the minutes;
The first region in the real space corresponding to the matching range including the first line segment on the first image is the positional relationship between the vehicle (10) and structures around the vehicle (10). And a second imaging unit (2) attached to the vehicle (10) so as to face a second direction different from the first direction. -1), specifying a range on the second image corresponding to the matching range by projecting onto a second image generated at a second time different from the first time; ,
Block matching is performed between the matching range and the second image while changing a relative position of the range on the second image corresponding to the matching range within a predetermined search range on the second image. Identifying a region most similar to the matching range, and setting a line segment in the most similar region as a second line segment corresponding to the first line segment;
Including an alignment method. At least one first line from the first image generated at the first time by the first imaging unit (2-2) attached to the vehicle (10) so as to face the first direction. Extracting the minutes;
The first region in the real space corresponding to the matching range including the first line segment on the first image is the positional relationship between the vehicle (10) and structures around the vehicle (10). And a second imaging unit (2) attached to the vehicle (10) so as to face a second direction different from the first direction. -1), specifying a range on the second image corresponding to the matching range by projecting onto a second image generated at a second time different from the first time; ,
Block matching is performed between the matching range and the second image while changing a relative position of the range on the second image corresponding to the matching range within a predetermined search range on the second image. Identifying a region most similar to the matching range, and setting a line segment in the most similar region as a second line segment corresponding to the first line segment;
A computer program for alignment for causing a computer to execute.

Images