JP2018116147A - Map creation device, map creation method and map creation computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a map creation device which can improve the accuracy of information indicated on a map formed of an image which is created by an on-vehicle camera.SOLUTION: A map creation device extracts a plurality of line components from a plurality of images around a vehicle (10), makes the extracted line components corresponding to line components in the same actual space between the plurality of images associated with one another, estimates a position of the vehicle (10) at the time of acquisition of the images on the basis of an evaluation value in which an angle which is formed of the line components in the actual space corresponding to orientations of the line components in the actual space which are calculated on the basis of a candidate position of the vehicle (10) at the time of acquisition of the images and the line components which are associated with one another between the images, and a progressing direction of the vehicle (10) in the candidate position of the vehicle (10) at the time of acquisition of any images is calculated from a likelihood in which the line components in the angle which is formed of the progressing direction of the vehicle (10) and the line components around the vehicle (10) are line components having prescribed orientations, and creates a map by acquiring positions of the line components in the actual space corresponding to the line components which are associated with one another between the images on the basis of the estimated position of the vehicle (10).SELECTED DRAWING: Figure 2

Description

  The present invention relates to a map creation device, a map creation method, and a map creation computer program for creating a map from an image around a vehicle obtained by an in-vehicle camera.

  In recent years, one or more cameras may be attached to a vehicle for the purpose of driving assistance or the like. In such a case, in order to effectively use the image obtained by the camera, it can be used between a plurality of images generated by different cameras or between a plurality of images generated by the same camera at different positions. It has been proposed to associate feature points or line segments on an image corresponding to the same point or line in space (see, for example, Patent Document 1 and Non-Patent Documents 1 to 3).

  For example, Patent Document 1 discloses an image in which characteristic points such as the center of a manhole cover and the end of a white line on a road surface are photographed by a camera attached to the moving body before and after moving when the moving body goes straight. In the meantime, it has been proposed to associate using object tracking processing.

  Non-Patent Document 1 describes that SURF features are extracted and collated for two consecutive frames between a camera mounted on a vehicle that captures the front and a camera that captures the left or right side. Further, Non-Patent Document 2 proposes that a region around a feature point on one image is deformed by warping to detect a corresponding point on the other image to be associated. Furthermore, Non-Patent Document 3 proposes that line segments detected from different frames are associated with each other in order to create an environment map (Simultaneous Localization and Mapping, SLAM).

JP, 2014-101075, A

Gim Hee Lee et al., "Motion Estimation for Self-Driving Cars With a Generalized Camera", Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on. IEEE, 2013 Simon Baker et al., `` Lucas-Kanade 20 Years On: A Unifying Framework: Part 1 '', International journal of computer vision 56.3, pp. 221-255, 2004 Keisuke Hirose et al., "Fast Line Description for Line-based SLAM" BMVC 2012, 2012

  In an automatic driving system or an advanced driving support system, information on a road (for example, a road marking such as a lane line or a pedestrian crossing) shown on a map is used to perform automatic driving or driving support. Therefore, in this map, it is preferable that information such as road markings drawn on the road surface is reproduced as accurately as possible. Therefore, there is a need for a technique for creating a map on which information such as road markings is reproduced with high accuracy based on a line segment detected from an image obtained by an in-vehicle camera.

  Therefore, the present invention provides a map creation device, a registration method, and a registration computer program capable of improving the accuracy of information represented on a map created from an image generated by a camera mounted on a vehicle. Objective.

According to the first aspect of the present invention, a map creation device is provided as one aspect of the present invention. The map creating apparatus stores a probability distribution representing a probability that the line segment is a line segment in a predetermined direction at an angle formed by the traveling direction of the vehicle (10) and the line segment around the vehicle (10). From each of a plurality of images around the vehicle (10) generated at different times by the unit (11) and at least one imaging unit (2-1, 2-2) attached to the vehicle (10), A line segment extraction unit (21) that extracts a plurality of line segments, and a plurality of line segments that are extracted from each of the plurality of images and that correspond to the line segments in the same real space. A line segment in the real space calculated based on the associating unit (23) for associating the vehicle, the candidate position of the vehicle (10) at the time of acquisition of each of the plurality of images, and the line segment associated between the plurality of images. Depending on the orientation of the real space line segment and multiple The line segment of the real space obtained by inputting the angle formed by the traveling direction of the vehicle (10) at the candidate position of any vehicle (10) at the time of acquisition of each image into the probability distribution is a predetermined direction A position estimation unit (24) for estimating the position of the vehicle (10) at the time of acquisition of each of the plurality of images based on an evaluation value calculated from the likelihood of the line segment, and acquisition of each of the plurality of images Based on the estimated position of the vehicle (10) at the time, a map creation unit (25) that creates a map by obtaining the position of the line segment in the real space corresponding to the line segment associated between the plurality of images And have.
The map creation device according to the present invention can improve the accuracy of information represented on a map created from an image generated by a camera mounted on a vehicle by having the above-described configuration.

According to a second aspect of the present invention, the predetermined direction includes a first direction parallel to the road extending direction and a second direction orthogonal to the road extending direction, and the probability distribution includes the vehicle The first probability distribution representing the probability of being a line segment in the first direction at the angle formed by the traveling direction of (10) and the line segment around the vehicle (10), and the second direction at the formed angle And a second probability distribution representing the probability of the line segment.
Thereby, the map creation device can accurately evaluate whether or not the line segment in the real space corresponding to the line segment associated between the images is any line segment marked on the road. As a result, the estimation accuracy of the position of the vehicle at the time of acquiring each image can be improved.

Alternatively, according to the description of claim 3, the position estimation unit (24) determines the candidate position of the vehicle (10) at the time of acquisition of each of the plurality of images when the evaluation value is minimized, for each of the plurality of images. It is preferable to estimate the position of the vehicle (10) at the time of acquisition.
Thereby, the map creation apparatus can estimate the position of the vehicle at the time of acquiring each image more accurately.

Furthermore, according to the description of claim 4, the position estimation unit (24) includes a likelihood that the line segment in the real space is a line segment in a predetermined direction, and a line segment in the real space for each of the plurality of images. Is evaluated based on the position when the image is projected on the image based on the candidate position of the vehicle (10) at the time of acquiring the image, and the value corresponding to the positional deviation between the line segments associated on the image It is preferable to calculate the value.
Thereby, since the map creation apparatus can evaluate the correctness of the candidate position at the time of each image acquisition more correctly, it can estimate the position of the vehicle at the time of each image acquisition more accurately.

Furthermore, according to the description of claim 5, the line segment extraction unit (21) includes a plurality of surroundings of the vehicle (10) generated at different times by the at least one imaging unit (2-1, 2-2). The plurality of line segments are extracted from each of the second images, and the associating unit (23) extracts the plurality of line segments extracted from each of the plurality of second images between the plurality of second images. Among these, it is preferable to associate line segments corresponding to line segments in the same real space. The map creation device calculates the plurality of second images based on the candidate positions of the vehicle (10) at the time of acquisition and the line segments associated between the plurality of second images. According to the direction of the line segment of the real space corresponding to the line segment associated between the second images, and any vehicle at the time of acquiring each of the line segment of the real space and the plurality of second images ( It is preferable to further include a learning unit (26) that learns the probability distribution based on an angle formed by the traveling direction of the vehicle (10) at the candidate position of 10).
Thereby, the map creation apparatus can reflect the distribution of the line segment for each angle formed by the line segment in the real space when the vehicle (10) actually travels and the traveling direction of the vehicle in the probability distribution. As a result, this map creation device can improve the estimation accuracy of the position of the vehicle when each image is acquired.

Furthermore, according to the description of claim 6, a map creation method is provided as another embodiment of the present invention. The map creation method is based on each of a plurality of images around the vehicle (10) generated at different times by at least one imaging unit (2-1, 2-2) attached to the vehicle (10). Extracting a plurality of line segments; associating line segments corresponding to line segments in the same real space among a plurality of line segments extracted from each of the plurality of images; The line segment of the real space according to the direction of the line segment of the real space calculated based on the candidate position of the vehicle (10) at the time of acquisition of each of the images and the line segment associated between the plurality of images And the angle formed by the traveling direction of the vehicle (10) at the candidate position of any vehicle (10) at the time of acquisition of each of the plurality of images, the traveling direction of the vehicle (10) and the surroundings of the vehicle (10) At the angle formed by the line segment, Based on the evaluation value calculated from the likelihood that the line segment in the real space is a line segment in the predetermined direction, which is obtained by inputting into a probability distribution representing the probability that the line segment in the predetermined direction is a line segment in the predetermined direction, Associating between the plurality of images based on the step of estimating the position of the vehicle (10) at the time of acquiring each of the plurality of images and the estimated position of the vehicle (10) at the time of acquiring each of the plurality of images. Creating a map by determining the position of a line segment in real space corresponding to the line segment.
The map creation method according to the present invention can improve the accuracy of information represented on a map created from an image generated by a camera mounted on a vehicle by having the above steps.

According to a seventh aspect of the present invention, a computer program for creating a map is provided as still another aspect of the present invention. Such a computer program for creating a map includes a plurality of images around the vehicle (10) generated at different times by at least one imaging unit (2-1, 2-2) attached to the vehicle (10). A step of extracting a plurality of line segments, and a step of associating line segments corresponding to a line segment in the same real space among a plurality of line segments extracted from each of the plurality of images between the plurality of images. The real space according to the direction of the line segment of the real space calculated based on the candidate position of the vehicle (10) at the time of acquisition of each of the plurality of images and the line segment associated with the plurality of images. The angle formed by the line segment and the traveling direction of the vehicle (10) at the candidate position of any vehicle (10) when each of the plurality of images is acquired is defined as the traveling direction of the vehicle (10) and the traveling direction of the vehicle (10). Line segment around An evaluation calculated from the likelihood that a line segment in the real space is a line segment in a predetermined direction obtained by inputting into a probability distribution representing the probability that the line segment is a line segment in a predetermined direction at an angle formed by Based on the value, the step of estimating the position of the vehicle (10) at the time of acquiring each of the plurality of images, and the position of the estimated vehicle (10) at the time of acquiring each of the plurality of images A step of creating a map by obtaining a position of a line segment in real space corresponding to a line segment associated between images, and a command for causing a computer to execute the map.
The computer program for map creation according to the present invention can improve the accuracy of information represented on a map created from an image generated by a camera mounted on a vehicle by having the above-described command.

  The reference numerals in parentheses attached to the above-described parts are examples that show the correspondence with specific means described in the embodiments described later.

It is a schematic block diagram of the map creation apparatus which concerns on one Embodiment of this invention. It is a functional block diagram of the control part of the map creation apparatus which concerns on one Embodiment of this invention. It is a figure which shows the relationship between the camera coordinate system about each of two cameras, and a vehicle coordinate system. (A)-(c) is a figure which shows an example of the patch which is a matching range which performs block matching, respectively. It is a figure which shows the positional relationship between the point in real space, and a vehicle based on the Manhattan-World hypothesis. It is an operation | movement flowchart of a positioning process. It is a figure which shows the example of the line segment detected on a road. It is a figure which shows an example of probability distribution for every direction of a line segment. (A) is a figure showing an example of the road where the vehicle traveled, (b) is a map created using the map creation device according to the present embodiment, corresponding to the road shown in (a). It is a figure showing an example. It is an operation | movement flowchart of a map production | generation process. It is a schematic block diagram of the control part by a modification.

Hereinafter, a map creation device according to one embodiment will be described with reference to the drawings.
This cartography device is mounted on a vehicle and has two lines of the same structure in real space (structures) between two images generated at different timings by two cameras directed in different directions. The line segments on the image corresponding to the shape (including not only the line based on the shape of the figure but also the line based on the figure or the character drawn on the structure) are associated with each other. And this map creation apparatus creates a map based on the line segment matched between images. At that time, the map creation device obtains the estimated position of the vehicle so that the evaluation value of the estimated position of the vehicle at the time of each image acquisition is minimized, and corresponds to the line segment associated between the images according to the estimated position. Find the position of the line segment in real space. At that time, the map creation device uses a real space line segment and a vehicle that are obtained in advance based on the assumption that many of the line segments drawn on the road surface are parallel or orthogonal to the extending direction of the road. The evaluation value is calculated based on the likelihood that the line segment in the real space corresponding to the line segment associated between images is a line segment in a predetermined direction using the probability distribution of the angle formed by the traveling direction of To do. As a result, the map creation device can improve the estimation accuracy of the position of the vehicle and can create a map in which information such as road display is reproduced with high accuracy.

  FIG. 1 is a schematic configuration diagram of a map creation device according to one embodiment. As shown in FIG. 1, a map creation device 1 is mounted on a vehicle 10 and includes cameras 2-1 and 2-2, an inertial measurement device (IMU) 3, a wheel speed sensor 4, and an electronic control unit (ECU) 5. Are connected to each other via a control area network (hereinafter referred to as CAN) 6. In FIG. 1, for convenience of explanation, each component mounted on the vehicle 10 such as the map creation device 1 and the shape, size, and arrangement of the vehicle 10 are different from actual ones.

  Each of the cameras 2-1 and 2-2 is an example of an imaging unit. In the present embodiment, each of the cameras 2-1 and 2-2 includes a two-dimensional detector composed of an array of photoelectric conversion elements having sensitivity to visible light, such as CCD or C-MOS, and the two-dimensional detector. An imaging optical system that forms an image of a region to be photographed on is provided. The camera 2-1 is attached to, for example, the vehicle interior of the vehicle 10 so that the optical axis of the imaging optical system is substantially parallel to the ground and faces the front of the vehicle 10. Then, the camera 2-1 captures a front area of the vehicle 10 at a predetermined shooting period (for example, 1/30 second), and generates an image in which the front area is captured. On the other hand, the camera 2-2 is attached to the rear end of the vehicle 10 so that the optical axis of the imaging optical system is substantially parallel to the ground and faces the rear of the vehicle 10. And the camera 2-2 image | photographs the back area | region of the vehicle 10 for every predetermined | prescribed imaging | photography period, and produces | generates the image which the back area | region was reflected. The images obtained by the cameras 2-1 and 2-2 may be color images or gray images.

  Each time the cameras 2-1 and 2-2 generate an image, they output the generated image to the map creation device 1.

The map creation device 1 includes a storage unit 11, a communication unit 12, and a control unit 13.
The storage unit 11 includes, for example, a semiconductor memory such as an electrically rewritable nonvolatile memory and a volatile memory. The storage unit 11 includes various programs for controlling the map creation device 1, various types of information used in the map creation process, images generated by the camera 2-1 or the camera 2-2, and the control unit 13. Stores temporary calculation results by.

  The communication unit 12 includes a communication interface that communicates with the cameras 2-1, 2-2, the IMU 3, the wheel speed sensor 4, the ECU 5, and the like through the CAN 6 and its control circuit. The communication unit 12 receives images from the cameras 2-1 and 2-2 and passes the images to the control unit 13. In addition, the communication unit 12 acquires odometry information representing the speed and movement amount of the vehicle 10 from the IMU 3 or acquires the wheel speed from the wheel speed sensor 4 for each cycle of executing the map creation process. It passes to the control unit 13.

The control unit 13 includes one or a plurality of processors (not shown) and their peripheral circuits. And the control part 13 controls the map creation apparatus 1 whole.
FIG. 2 shows a functional block diagram of the control unit 13. As illustrated in FIG. 2, the control unit 13 includes a line segment extraction unit 21, a projection unit 22, an association unit 23, a position estimation unit 24, and a map creation unit 25. Each of these units included in the control unit 13 is implemented as, for example, a functional module realized by a computer program executed on a processor included in the control unit 13.

In the present embodiment, the control unit 13 displays the rear of the vehicle 10 at a certain time n (here, the time is expressed in units of frame intervals, and the time in units of seconds, minutes, etc. is referred to as real time). The line segment on the image generated by the camera 2-2 that captures the front side of the vehicle 10 at the time (nk) k frames before the time. Correlate with. This is because, when the vehicle 10 is moving forward, the range photographed at the time n by the camera 2-2 for photographing the rear of the vehicle 10 is to photograph the front of the vehicle 10 at a time (nk) before that. This is because it is assumed that there is a portion that overlaps the range photographed by the camera 2-1. And the control part 13 specifies the line segment matched over three or more images by performing the above associations sequentially. And the control part 13 estimates the position of the vehicle at the time of acquisition of each image using the line segment matched in each of the three or more images, and matches between images according to the estimated position of the vehicle. The position of the line segment in the real space corresponding to the given line segment is obtained and written on the map.
Hereinafter, for convenience of explanation, an image generated by the camera 2-1 is referred to as a front image, and an image generated by the camera 2-2 is referred to as a rear image. In the map creation process, the process related to the line segment association between the images, which is executed by the line segment extraction unit 21, the projection unit 22, and the association unit 23 may be referred to as an alignment process.

Each time the control unit 13 receives a rear image from the camera 2-2, the line segment extraction unit 21 uses a road surface, a road sign, a signboard, or a building in the shooting area of the camera 2-2 from the rear image. A line segment corresponding to the line of the structure is extracted. For this purpose, the line segment extraction unit 21 detects a line segment on the rear image using, for example, a line segment detector. The line segment extraction unit 21 is, for example, RG von Gioi et al., “A fast line segment detector with a false detection control”, IEEE Transactions on Pattern Analysis & Machine Intelligence, (4), 2008, pp. The algorithm disclosed in .722-732 can be used. Alternatively, the line segment extraction unit 21 may use another line segment detector that can detect a line segment from an image.
The line segment extraction unit 21 calculates, for each rear image, a vector representing the acquisition time of the rear image and the coordinates, direction, and length of the midpoint of the line segment on the rear image representing the line segment extracted from the rear image. It passes to the projection unit 22.

  Similarly, every time the control unit 13 receives a front image from the camera 2-1, the line segment extraction unit 21 extracts a line segment from the front image. Then, the line segment extraction unit 21 stores a vector representing the coordinates, direction, and length of the midpoint of the extracted line segment in the storage unit 11 together with the acquisition time of the front image.

For each line segment on the rear image, the projection unit 22 sets a block matching patch centered on the line segment, and specifies an area corresponding to the patch in real space. Then, the projection unit 22 projects the corresponding area of the patch in the real space on the front image acquired k frames before the rear image.
This projection processing requires conversion between coordinates on the image and coordinates in the real space. First, a coordinate system used for the conversion will be described.

  FIG. 3 is a diagram illustrating the relationship between the camera coordinate system and the vehicle coordinate system for each of the two cameras. A camera coordinate system (hereinafter referred to as a forward camera coordinate system for convenience) C1, which is a coordinate system in the real space with the camera 2-1 as a reference, uses the image-side focal point of the camera 2-1 as an origin, and a straight traveling direction of the vehicle 10 Is a coordinate system with the z axis, the direction parallel to the road surface and perpendicular to the z axis as the x axis, and the direction perpendicular to the road surface as the y axis. With respect to the z axis, the forward direction of the vehicle 10 is positive, and the positive direction of the x axis is defined by the right-handed coordinate system. Further, a camera coordinate system (hereinafter referred to as a rear camera coordinate system for convenience) C2 which is a coordinate system in the real space with the camera 2-2 as a reference has an image side focal point of the camera 2-2 as an origin, and the vehicle 10 This is a coordinate system in which the straight direction is the z axis, the direction parallel to the road surface and perpendicular to the z axis is the x axis, and the direction perpendicular to the road surface is the y axis. For the z axis, the direction toward the rear of the vehicle 10 is positive, and the positive direction of the x axis is defined by the right-handed coordinate system.

  The vehicle coordinate system V, which is a coordinate system in the real space with respect to the vehicle 10, is an arbitrary point of the vehicle 10, for example, a midpoint between front wheels, a midpoint between rear wheels, or a front side The midpoint between the midpoint between the wheels and the midpoint between the rear wheels is the origin, and the straight direction of the vehicle 10 is the z axis, the direction parallel to the road surface and perpendicular to the z axis is the x axis, and the direction perpendicular to the road surface Is the coordinate system with y-axis. With respect to the z axis, the forward direction of the vehicle 10 is positive, and the positive direction of the x axis is defined by the right-handed coordinate system. In these coordinate systems, the angle formed with the z axis in the xz plane is represented by θ.

  FIG. 4A to FIG. 4C are diagrams illustrating examples of patches that are matching ranges in which block matching is performed. In the example shown in FIG. 4A, the patch 401 has the same length as the line segment 410 along the direction parallel to the line segment 410 of interest, and a predetermined width in the direction orthogonal to the line segment 410. It is set as a rectangular area having (for example, 10 to 20 pixels). In the example shown in FIG. 4B, the patch 402 has a predetermined width (for example, 10 to 20 pixels) in a direction orthogonal to the focused line segment 410 and is parallel to the line segment 410. A rectangular region longer than the line segment 410 by a predetermined length (for example, 10 to 20 pixels) is set. Furthermore, in the example shown in FIG. 4C, the patch 403 is set as a region where the distance to the focused line segment 410 is a predetermined value (for example, 10 to 20 pixels) or less.

ここで、(u_m,2,v_m,2)は、後方画像上での着目する線分の中点の座標を表し、α、βは、それぞれ、後方画像上でのパッチu_p内の画素の線分の中点からの水平方向及び垂直方向の距離を表す。なお、αの取り得る値の範囲[α₀, α₁]、及び、βの取り得る値の範囲[β₀, β₁]は、それぞれ、垂直方向の位置及び水平方向の位置についての関数として表される。そしてその関数は、線分の向き、及び、図４（ａ）〜図４（ｃ）に示される、パッチの形状に応じて決定される。 In this embodiment, the patch is u _p a matching range for performing block matching, for example, is defined as follows:.

_{Here, (u m, 2, v} m, 2) represents the coordinates of the midpoint of the target line segment on the rearward image, alpha, beta, respectively, in the patch u _p on the rear image It represents the distance in the horizontal and vertical directions from the midpoint of the line segment of the pixel. Note that the range of values [α ₀ , α ₁ ] that can be taken by α and the range [β ₀ , β ₁ ] that can be taken of β are respectively expressed as functions of the vertical position and the horizontal position. expressed. The function is determined according to the direction of the line segment and the shape of the patch shown in FIGS. 4 (a) to 4 (c).

The projection 22 from the patch u _p to be set on the basis of the line segment on the rearward image, corresponding in the real space in accordance with a predetermined condition which defines the positional relationship of the structure in the vicinity of the vehicle 10 and the vehicle 10 Identify the area. Then, the projection unit 22 for the corresponding region, the rear camera coordinate system → the vehicle coordinate system at time n → the world coordinate system that is the absolute coordinate in real space → the vehicle coordinate system at time (nk) → the front camera coordinate system → the front image Project in the order above.
Thus is described first coordinates are derived in the camera coordinate system for the area in the real space corresponding to the patch u _p.

ここで、(c_u,c_v)は後方画像の中心の座標、すなわち、カメラ２−２の光軸上の位置に相当する後方画像上の位置の座標を表す。またf_u、f_vは、それぞれ、後方画像上の画素の水平方向及び垂直方向のサイズを考慮した、水平方向及び垂直方向に相当するカメラ２−２の焦点距離を表す。そして(u_n、v_n)は、線分の中点(u_m,2,v_m,2)に対応する正規化座標であり、U_n=(u_n+α_n,v_n+β_n)は、正規化座標系でのパッチu_p内の各画素の座標である。そして(x,y,z)は、U_nに対応するカメラ座標系上の点の座標である。 In converting the coordinates on the rear image or the front image (two-dimensional coordinates) to the coordinates (three-dimensional coordinates) of the camera coordinate system, a pinhole camera model is applied in the present embodiment. In this case, the coordinates of each pixel in the patch u _p on the rearward image, is converted into the coordinate U _n on the normalized coordinate system in accordance with the following equation.

Here, (c _u , c _v ) represents the coordinates of the center of the rear image, that is, the coordinates of the position on the rear image corresponding to the position on the optical axis of the camera 2-2. Further, f _u and f _v represent the focal lengths of the camera 2-2 corresponding to the horizontal direction and the vertical direction, respectively, in consideration of the size of the pixel on the rear image in the horizontal direction and the vertical direction. And (u _n , v _n ) is a normalized coordinate corresponding to the midpoint of the line segment (u _{m, 2} , v _{m, 2} ), and U _n = (u _n + α _n , v _n + β _n ) are the coordinates of each pixel in the patch u _p in a normalized coordinate system. (X, y, z) are the coordinates of a point on the camera coordinate system corresponding to U _n .

In the present embodiment, the Manhattan-World hypothesis is introduced as a predetermined condition that defines the positional relationship between the vehicle 10 and the structures around the vehicle 10. Based on the Manhattan-World hypothesis, the real space region corresponding to the patch for each line segment is specified. In the Manhattan-World hypothesis, the line of the structure in the real space corresponding to the line segment on the image is one of the following three types.
(1) A line on the side wall of the building parallel to the straight direction of the vehicle 10 (2) A line on the side wall of the building orthogonal to the straight direction of the vehicle 10 (3) A line on the road surface or a line segment on the image It is assumed that the line around the structure in real space is a plane.

  FIG. 5 is a diagram showing the positional relationship between the line of the structure in the real space and the vehicle based on the Manhattan-World hypothesis according to the present embodiment. In the present embodiment, since the camera 2-1 faces the front of the vehicle 10, while the camera 2-2 faces the rear of the vehicle 10, both of the lines corresponding to the above (2) are used. The line segment on one image is not associated with the line segment on the other image. Therefore, a line segment corresponding to line 501 on the side wall parallel to the straight traveling direction of the vehicle 10 corresponding to (1) or a line segment corresponding to line 502 on the road surface corresponding to (3) is Corresponding between the image and the back image.

When the line of the structure in the real space corresponding to the line segment on the image is on the side wall of the building 500 parallel to the straight traveling direction of the vehicle 10, as in the line 501, on the real space corresponding to the patch of the line segment For each point included in the region, x = d in the camera coordinate system. In addition, d is the distance to the line in the real space corresponding to the vehicle 10 and its line segment. Therefore, if the above condition (1) is assumed, the coordinates P _p in the camera coordinate system for each point in the area in the real space corresponding to the patch u _p is expressed by the following equation.

On the other hand, when the line of the structure in the real space corresponding to the line segment on the image is on the road surface like the line 502, each point included in the area on the real space corresponding to the patch of the line segment In the camera coordinate system, y = h. Note that h is the height from the road surface to the camera 2-2 and is stored in the storage unit 11 in advance. Therefore, if the above condition (3) is assumed, the coordinates P _p in the camera coordinate system for each point in the area in the real space corresponding to the patch u _p is expressed by the following equation.

Projection 22 (2) and (3), or (2) and (4) according to wherein each of the regions in the real space corresponding to the patch u _p of each line segment on the rearward image A coordinate P _p on the camera coordinate system of the point is obtained. Note that d is changed in units of 1 m in an arbitrary predetermined section, for example, a section of [1,20] (m). That is, for each line segment, for each value of d, 20 areas of coordinates P _p obtained based on the expressions (2) and (3) and 1 obtained based on the expressions (2) and (4) The coordinates P _p of the individual areas are obtained.

  Note that the projection unit 22 may set the section d to be narrower than the predetermined section by bundle adjustment. In this case, the projection unit 22, for example, between the two rear images obtained by photographing the line of the structure in the real space corresponding to the line segment on the image at different times with the camera 2-2. Specify by associating line segments with. Then, the projection unit 22 performs a point in the real space corresponding to a point (for example, a midpoint) on the line segment from the vehicle 10 by triangulation using the positions of the two vehicles corresponding to the generation of the two rear images. And the interval d may be set so as to include the measured distance.

ここで、u_n'は、座標P_pに対応する、前方画像上の領域の座標である。また関数π_vc1()は、車両座標系からカメラ２-１のカメラ座標系への変換関数であり、関数π_vc2()は、車両座標系からカメラ２-２のカメラ座標系への変換関数である。なお、これらの逆関数は、それぞれ、対応するカメラ座標系から車両座標系への変換関数となる。なお、車両座標系とカメラ座標系間の変換は、それぞれの座標系の原点の差に相当する並進行列T_vc1,T_vc2と、車両１０の直進方向とカメラの光軸間の傾きに相当する回転行列R_vc1,R_vc2により表される。 For the patch of each line segment on the rear image, the projection unit 22 projects a warping function for projecting each of the areas on the real space corresponding to the patch onto the front image.

Here, u _n ′ is a coordinate of a region on the front image corresponding to the coordinate P _p . The function π _vc1 () is a conversion function from the vehicle coordinate system to the camera coordinate system of the camera 2-1, and the function π _vc2 () is a conversion function from the vehicle coordinate system to the camera coordinate system of the camera 2-2. It is. Each of these inverse functions is a conversion function from the corresponding camera coordinate system to the vehicle coordinate system. The conversion between the vehicle coordinate system and the camera coordinate system corresponds to the _{parallel progression trains} T _vc1 , T _vc2 corresponding to the difference between the origins of the respective coordinate systems, and the inclination between the straight traveling direction of the vehicle 10 and the optical axis of the camera. _Represented by rotation matrices R _vc1 and R _vc2 .

The function π _ci () is a conversion function from the camera coordinate system to the coordinates on the image, and the coordinates (u _n , v _n ) on the image from the camera coordinates (x, y, z) in equation (2). Equivalent to conversion to

The function π ⁿ _wv () is a conversion function from the world coordinate system to the vehicle coordinate system at time n. On the other hand, the inverse function π ⁿ _wv ⁻¹ () is a conversion function from the vehicle coordinate system to the world coordinate system at time n.

ここで、u_nは、オドメトリ情報であり、v_n,right、v_n,leftは、それぞれ、時刻nにおける車両１０の右側後輪の車輪速、及び左側後輪の車輪速を表す。そしてω_n、v_nは、それぞれ、時刻ｎにおける車両１０の角速度及び速度を表す。またd_aは、車両１０の左右の後輪間の間隔である。さらに、Δtは、時刻(n-k)と時刻n間の実時刻の差であり、Δt=k×fで表される。なお、fは、カメラ２−１、２−２のフレーム周期（撮影間隔）である。またε_xは、移動量の誤差を確率分布で表す誤差モデルである。本実施形態では、誤差モデルε_xは、ガウス分布で表されるものとし、その共分散Σ_xは、x軸方向、z軸方向及びθに関して、以下のように定義される。

ここで、σ_x、σ_z及びσ_θは、例えば、オドメトリ情報あるいは車輪速の誤差に基づいて設定され、例えば、σ_x=σ_z=0.1[m]であり、σ_θ=2π/360[rad]である。 Details of the function π ⁿ _wv () will be described below. Let X _n = ^t (x _n , z _n , θ _n ) be a position vector representing the position and orientation of the vehicle 10 at time n in the world coordinate system. In this case, X _n is the time (nk) of the vehicle 10 at the position ^{_{X nk = t (x nk,}} z nk, θ nk) between the relationship is established as represented by the following formula.

Here, u _n is odometry information, and v _{n, right} and v _{n, left} represent the wheel speed of the right rear wheel and the left rear wheel of the vehicle 10 at time n, respectively. Ω _n and v _n represent the angular velocity and speed of the vehicle 10 at time n, respectively. D _a is the distance between the left and right rear wheels of the vehicle 10. Furthermore, Δt is the difference in real time between time (nk) and time n, and is represented by Δt = k × f. Note that f is the frame period (photographing interval) of the cameras 2-1 and 2-2. Further, ε _x is an error model that expresses an error in the movement amount by a probability distribution. In the present embodiment, the error model ε _x is represented by a Gaussian distribution, and the covariance Σ _x is defined as follows with respect to the x-axis direction, the z-axis direction, and θ.

Here, σ _x , σ _z, and σ _θ are set based on, for example, odometry information or wheel speed error. For example, σ _x = σ _z = 0.1 [m], and σ _θ = 2π / 360 [ rad].

したがって、関数πⁿ _wv()は、次式で表される。

ここで、P_wは、世界座標系での着目点の座標を表し、P_vは、P_wに対応する点の車両座標系での座標を表す。なお、（５）式における、π^n-k _wv()では、x_n=x_n-k、z_n=z_n-k、θ_n=θ_n-kとすればよい。 The rotation matrix R ^t and the parallel progression T ^t representing the transformation from the world coordinate system to the vehicle coordinate system at time n are expressed by the following equations using the position vector X _n of the vehicle 10 at time n.

Therefore, the function π ⁿ _wv () is expressed by the following equation.

Here, P _w represents the coordinates of the point of interest in the world coordinate system, and P _v represents the coordinates of the point corresponding to P _w in the vehicle coordinate system. In π ^nk _wv () in equation (5), x _n = x _nk , z _n = z _nk , θ _n = θ _nk may be set.

ここでRⁱ _vwは、時刻iにおける、変換式πⁱ _wvに含まれる回転行列Rの逆行列であり、（８）式及び（９）式に従って算出される。 Further, in order to more accurately identify the corresponding line segment on the front image with reference to the line segment on the rear image, the line segment on the front image corresponding to the direction of the line segment on the rear image. It is preferable that the orientation of is required. In addition, a line segment in the real space includes a three-dimensional vector r (rx, px) representing the three-dimensional coordinates p (px, py, pz) of the point on the line segment (for example, the midpoint) and the direction of the line segment. ry, rz). Therefore, the conversion formula for converting the three-dimensional vector r representing the direction of the line segment in the world coordinate system into the vector A ₁ of the camera coordinate system based on the camera 2-1, and the camera 2-2 as a reference. A conversion formula for converting to the vector A _{2 in the} camera coordinate system is expressed by the following formula.

Here, R ⁱ _vw is an inverse matrix of the rotation matrix R included in the conversion equation π ⁱ _wv at time i, and is calculated according to the equations (8) and (9).

ここで、q₁(x₁,y₁,z₁)は、カメラ２−１を基準とするカメラ座標系での線分上の点pの座標を表し、q₂(x₂,y₂,z₂)は、カメラ２−２を基準とするカメラ座標系での線分上の点pの座標を表し、それぞれ、（５）〜（９）式に従って算出される。したがって、投影部２２は、後方画像上の各線分について、その線分の向きを表すベクトルを、カメラ２−２に関する（１０）式及び（１１）式の逆変換に従って世界座標系でのベクトルに変換し、その後、カメラ２−１に関する（１０）式及び（１１）式に従って、世界座標系でのベクトルから、前方画像上での対応ベクトルを求めることができる。 In addition, an expression for projecting the camera coordinate system vector A ₁ to the front image and calculating the corresponding vector a ₁ on the front image, and a camera coordinate system vector A ₂ to the rear image and corresponding on the rear image An expression for calculating the vector a ₂ is expressed by the following expression.

Here, q ₁ (x ₁ , y ₁ , z ₁ ) represents the coordinates of the point p on the line segment in the camera coordinate system with respect to the camera 2-1, and q ₂ (x ₂ , y ₂ , z ₂ ) represents the coordinates of the point p on the line segment in the camera coordinate system with respect to the camera 2-2, and is calculated according to equations (5) to (9), respectively. Therefore, the projection unit 22 converts the vector representing the direction of each line segment on the rear image into a vector in the world coordinate system according to the inverse transformation of the expressions (10) and (11) regarding the camera 2-2. Then, the corresponding vector on the front image can be obtained from the vector in the world coordinate system according to the equations (10) and (11) regarding the camera 2-1.

As described above, corresponding to the patch u _p, a region in the real space by projecting into the forward image, the position of the forward image corresponding to each pixel in the patch u _p, i.e., front patch u _p An area projected on the image is obtained.
Projection unit 22, corresponding to each pixel in the patch u _p of each line segment on the rearward image, k frames before the pixel in the forward image coordinates and the line segment orientation to the associated portion 23 of the vector representing the Output.

Associating unit 23, the particular corresponding to the patch u _p of each line segment on the rearward image, and the region on the k frames before the front image, the area to be similar to each other by performing block matching between the front image By doing so, the line segment on the front image is associated with each line segment on the rear image.

なお、wは、（５）式の右辺を表す。 In the present embodiment, the associating unit 23 sets, for each line segment on the rear image, a search range for searching for a line segment on the corresponding front image as an error ellipse based on the error model. At that time, the error ellipse is determined by a covariance matrix Σ _F calculated by the following equation.

Note that w represents the right side of equation (5).

ここで、χ²は、χ二乗分布であり、誤差楕円の信用区間を表す。また、前方画像上の水平方向に対して誤差楕円の長径がなす角θ_εは、次式で表される。

Associating unit 23 calculates eigenvalues lambda ₁ of the covariance matrix sigma _F, and lambda _2, the eigenvector v _1, v _2. However, the larger eigenvalue is λ ₁ . At this time, the major axis a and minor axis b of the error ellipse are expressed by the following equations.

Here, χ ² is a χ square distribution and represents the confidence interval of the error ellipse. Further, the angle θ _ε formed by the major axis of the error ellipse with respect to the horizontal direction on the front image is expressed by the following equation.

ここで、(u'_c,v'_c)は、後方画像上の着目する線分に対応する前方画像上での線分の中点の座標を表す。そして(u',v')は、前方画像上で検出された線分の中点の座標である。 For each line segment on the rear image, the associating unit 23 is within an error ellipse that is a search range on the front image k frames before, and the angle formed with the line segment when projected onto the front image is predetermined. A line segment on the front image having an angle (for example, 3 ° to 5 °) or less is detected as a candidate associated with the line segment on the rear image. Note that the associating unit 23 determines that the line segment on the forward image that satisfies the following expression is included in the search range.

Here, (u ′ _c , v ′ _c ) represents the coordinates of the midpoint of the line segment on the front image corresponding to the line segment of interest on the rear image. (U ′, v ′) is the coordinates of the midpoint of the line segment detected on the front image.

  For each line segment on the rear image, the associating unit 23 calculates a normalized cross-correlation value by performing block matching for each line segment within the search range on the front image before k frames. At this time, the associating unit 23 determines the center of the area on the front image on which the patch is projected, that is, the position at which the line segment on the rear image is projected on the front image and the front image on which block matching is performed. Pixels on the front image corresponding to each pixel in the patch on the rear image by translating the position of each pixel in the patch projected on the front image so as to cancel the position difference between the line segments of Should be specified. The association unit 23 can calculate a normalized cross-correlation value based on the luminance value of each pixel in the patch on the rear image and the luminance value of the pixel on the corresponding front image. The associating unit 23 associates the line segment on the front image having the maximum normalized cross-correlation value with the line segment of interest on the rear image. Note that the association unit 23 may associate the two line segments only when the maximum value of the normalized cross-correlation values is equal to or greater than a predetermined threshold (for example, 0.6 to 0.7).

  In order to reduce the amount of calculation, the association unit 23 sets the search range to a predetermined range that is centered on the coordinates on the front image corresponding to the midpoint of the line segment on the rear image. It may be set. In this case, the predetermined range may be, for example, 2 to 3 times the horizontal size of the patch in the horizontal direction and 2 to 3 times the vertical size of the patch in the vertical direction.

  Note that, according to the modification, since the patch shape itself includes information on the direction of the line segment on the rear image, the associating unit 23 is in the search range on the front image before k frames. All the line segments on the front image may be candidates to be associated with the line segments on the rear image.

  Further, according to another modification, the associating unit 23 translates the patch so that each pixel in the corresponding search range corresponds to the center of the patch for each line segment on the rear image, respectively. The cross-correlation value is calculated, and the pixel having the maximum normalized cross-correlation value is set as the midpoint of the line segment on the front image corresponding to the line segment of interest on the rear image, and the direction of the line segment of interest is calculated. The direction on the corresponding front image may be the direction of the corresponding line segment. In this case, the line segment extraction unit 21 may not extract a line segment from the front image.

  The associating unit 23 includes a set of vectors representing the coordinates and orientations of the midpoints of the respective line segments on the front image associated with the line segments on the rear image, and the corresponding rear image acquisition time and front image. Are stored in the storage unit 11 in association with each other.

  FIG. 6 shows an operation flowchart of the alignment process controlled by the control unit 13. In addition, the control part 13 performs the alignment process according to this operation | movement flowchart for every back image.

  The line segment extraction unit 21 extracts line segments from each of the rear image and the front image (step S101). The line segment extraction unit 21 stores a vector representing the coordinates and orientation of the midpoint of each line segment extracted from the front image in the storage unit 11. Further, the line segment extraction unit 21 passes a vector representing the coordinates and direction of the midpoint of each line segment extracted from the rear image to the projection unit 22.

  For each line segment extracted from the rear image, the projection unit 22 sets a patch centered on the midpoint of the line segment, and sets one or more regions on the real space corresponding to the patch according to the Manhattan-World hypothesis. (Step S102). Then, the projection unit 22 projects each region in the real space corresponding to the patch on the front image acquired k frames before the rear image is acquired for each line segment extracted from the rear image (step S103). ).

  For each line segment extracted from the rear image, the associating unit 23 sets a search range centered on the projection position on the front image at the center of the area on the real space corresponding to the patch (step S104). Then, the associating unit 23 performs the block matching between the area around the line segment on the front image within the search range and the projection range of the area on the real space corresponding to the patch on the front image. The matching line segment is associated with the line segment on the rear image (step S105). Then, the control unit 13 ends the alignment process.

  The position estimation unit 24 includes, for each of the three or more sets of the front image and the rear image, each including a plurality of line segments (hereinafter, simply referred to as line segments) associated with each other. Based on this, an evaluation value representing the certainty of the estimated position of the vehicle is calculated. And the position estimation part 24 estimates the position of a vehicle based on the evaluation value. Since the position estimation unit 24 only needs to execute the same processing for each set of images, processing for one set of images will be described below.

In the present embodiment, the position estimation unit 24 firstly, for each line segment associated between images, two vehicle position candidates at the time of acquisition (hereinafter simply referred to as candidate positions) of two sets of images. based on the initial value of the called), according to the principle of triangulation, the position of the line segment L ⁱ in the real space corresponding to the line segment containing (orientation, obtaining the same below). Note that the initial value of the candidate position of the vehicle can be obtained by, for example, eliminating the second term ε _x in the equation (6). In addition, the line segment L ^{i in the} real space includes, for example, the coordinates p ^{i of} the midpoint and a vector r ⁱ = (r ⁱ _x , r ⁱ _y , r ⁱ _z ) representing the direction and length of the line segment L ^i. And (p ⁱ , r ⁱ ).

Position estimator 24, each line segment L ⁱ in the real space represented in the world coordinate system is projected on each image. At that time, the position estimation unit 24, based on the initial value of the candidate position of the vehicle, (9), according to (5) and (2), corresponding to the two end points of each line segment L ⁱ in the real space Then, the line segment l ⁱ projected on the image may be obtained by obtaining the coordinates on each image.

ここで、(uⁱ,vⁱ)は、投影線分lⁱの中点の画像上での座標であり、(dxⁱ, dyⁱ)は、投影線分lⁱの向きを表す単位ベクトルである。また、g_n ⁱ=(g_xn ⁱ,g_yn ⁱ)(n=1,2)は、投影線分lⁱに対応する、画像上で検出され、かつ、他の画像上の線分と対応付けられた線分の両端点の座標を表す。この評価値Eは、（９）式における回転行列R及び並進行列T（すなわち、車両１０の候補位置）と実空間の各線分Lⁱの位置及び向きとの関数となる。 The position estimating unit 24, each line segment L ⁱ in the real space, the line segment which is projected on the image (hereinafter, the corresponding, detected on the image, and distinguished from the segment associated between images The evaluation value is based on the distance between l ⁱ and the end point of the corresponding line segment on the image, and the angle formed by each line segment L ^{i in} real space and the traveling direction of the vehicle 10. Is calculated. For example, the position estimation unit 24 calculates the evaluation value E according to the following equation.

Here, (u ⁱ , v ⁱ ) are the coordinates of the midpoint of the projection line segment l ⁱ on the image, and (dx ⁱ , dy ⁱ ) is a unit vector that represents the direction of the projection line segment l ^i. is there. G _n ⁱ = (g _xn ⁱ , g _yn ⁱ ) (n = 1,2) is detected on the image corresponding to the projected line segment l ⁱ and corresponds to a line segment on another image Represents the coordinates of the endpoints of the attached line segment. The evaluation value E is a function of the rotation matrix R and translation matrix T (i.e., a candidate position of the vehicle 10) the location and orientation of each line segment L ⁱ between the real space in equation (9).

ここでF=(F_x,F_y,F_z)は、車両１０の進行方向を表す。そしてP_j(d_θ ^m)(j=1,2,3,4)は、実空間上のm番目の線分が、j番目の所定の向きの線分である尤度を表す。本実施形態では、所定の向きは、道路の延伸方向と平行な方向と、道路の延伸方向に対して時計回りに90°回転した方向と、道路の延伸方向に対して反時計回りに90°回転した方向と、その他の方向とを含む。 Further, d _θ ^m is the m-th line segment L ^m = (p ^m , r ^m ) in the real space and at the candidate position of the vehicle 10 when any image is acquired, for example, when the latest image is acquired. An angle formed by the traveling direction of the vehicle 10 is represented. For example, d _θ ^m is calculated according to the following equation.

Here, F = (F _x , F _y , F _z ) represents the traveling direction of the vehicle 10. P _j (d _θ ^m ) (j = 1, 2, 3, 4) represents the likelihood that the m-th line segment in the real space is the j-th predetermined line segment. In the present embodiment, the predetermined direction includes a direction parallel to the road extending direction, a direction rotated 90 ° clockwise relative to the road extending direction, and 90 ° counterclockwise relative to the road extending direction. Includes rotated direction and other directions.

ここで、σ₁ ²及びμ₁(=0)は、道路の延伸方向と略平行な線分と車両１０の進行方向とがなす角の分散及び平均値を表す。同様に、σ₂ ²及μ₂(=π/2)は、道路の延伸方向に対して時計回りに略90°回転した線分と車両１０の進行方向とがなす角の分散及び平均値を表す。また、σ₃ ²及びμ₃(=-π/2)は、道路の延伸方向に対して反時計回りに略90°回転した線分と車両１０の進行方向とがなす角の分散及び平均値を表す。そしてσ₄ ²及びμ₄(=0)は、その他の向きの線分と車両の進行方向がなす角の分散及び平均値を表す。なお、道路の延伸方向と略直交する線分についての確率分布が二つ存在するのは、画像間で対応付けられた線分の向きが、その線分の一端から他端へ向かうベクトルで規定されるため、対応する実空間の線分にも同様の向きが規定されることによる。
なお、これらの４種類の線分の向きの確率分布全体が、一つの混合ガウス分布で表されてもよい。この場合には、P_j(d_θ ^m)ごとに重み係数が規定される。
P_j(d_θ ^m)を表すパラメータ（すなわち、分散σ_j ²及び平均値μ_j）は、予め記憶部１１に保存される。 FIG. 7 is a diagram illustrating an example of a line segment detected on a road. A lane marking or a boundary between a roadway and a sidewalk is along the extending direction of the road 700. Therefore, the line segment 701 detected at the boundary between the lane marking or the roadway and the sidewalk is substantially parallel to the extending direction of the road. On the other hand, a line such as a pedestrian crossing or a stop line is drawn so as to be substantially orthogonal to the extending direction of the road. Therefore, the line segment 702 detected along these lines is substantially orthogonal to the extending direction of the road. Further, since a part of the contour such as a character or a symbol drawn on the road is neither parallel nor perpendicular to the extending direction of the road, a line segment 703 detected along the part of the contour is detected. Is neither parallel nor perpendicular to the direction of road stretch. In addition, it is assumed that there are relatively many line segments along the road extending direction or in a direction perpendicular to the road extending direction, while other line segments are relatively small. . Therefore, the line segment detected on the road is classified into a line segment substantially parallel to the road extending direction, a line segment substantially orthogonal to the road extending direction, or another line segment. be able to. In general, the vehicle 10 travels along the extending direction of the road 700. Accordingly, the traveling direction of the vehicle 10 and the extending direction of the road substantially coincide. Therefore, the probability that the line segment in the real space corresponding to the line segment associated between the images is a line segment along the extending direction of the road depends on the traveling direction of the vehicle and the line segment in the real space. It is represented by a probability distribution according to the angle formed by. Similarly, the probability that the line segment in the real space is a line segment orthogonal to the extending direction of the road, and the probability that the line segment in the other space is other than the traveling direction of the vehicle and the line segment in the real space. It is represented by a probability distribution according to the angle formed. Therefore, P _j (d _θ ^m ) is expressed by a Gaussian distribution as follows, for example.

Here, σ ₁ ² and μ ₁ (= 0) represent a variance and an average value of angles formed by a line segment substantially parallel to the road extending direction and the traveling direction of the vehicle 10. Similarly, σ ₂ ² and μ ₂ (= π / 2) are the dispersion and average value of the angle formed by the line segment rotated approximately 90 ° clockwise relative to the road extending direction and the traveling direction of the vehicle 10. Represent. Further, σ ₃ ² and μ ₃ (= −π / 2) are the dispersion and average value of the angle formed by the line segment rotated approximately 90 ° counterclockwise with respect to the road extending direction and the traveling direction of the vehicle 10. Represents. Σ ₄ ² and μ ₄ (= 0) represent the dispersion and average value of the angles formed by the line segments in other directions and the traveling direction of the vehicle. Note that there are two probability distributions for a line segment that is substantially orthogonal to the direction of road extension because the direction of the line segment associated between images is defined by a vector from one end of the line segment to the other. Therefore, the same direction is defined for the corresponding line segment in the real space.
Note that the entire probability distribution in the direction of these four types of line segments may be represented by one mixed Gaussian distribution. In this case, a weighting factor is defined for each P _j (d _θ ^m ).
Parameters representing P _j (d _θ ^m ) (that is, variance σ _j ² and average value μ _j ) are stored in the storage unit 11 in advance.

FIG. 8 is a diagram illustrating an example of a probability distribution for each direction of a line segment in the real space. 8, the horizontal axis represents the angle d _theta is the traveling direction and the line of the vehicle, the vertical axis represents the probability. A graph 801 represents a probability distribution for a line segment substantially parallel to the road extending direction. A graph 802 and a graph 803 respectively represent probability distributions for line segments that are substantially orthogonal to the road extending direction. A graph 804 represents the probability distribution for the other line segments. As shown in the graph 801 to 803, the stretching direction substantially parallel to the line of the road, and, for the line segments substantially perpendicular to the stretching direction of the road, although the relatively high probability in the average value of d _theta , D _θ decreases rapidly as it departs from the average value. On the other hand, as shown in the graph 804, since the other traveling direction and the angle between the line segment and the vehicle 10 d _theta distributed over a wide range, a relatively low probability across corners d _theta.

Position estimator 24, as evaluation value decreases, the position and update at least one of the orientation of the line segments L ⁱ of the candidate positions and the real space of the vehicle 10 during each image acquisition, at each image acquisition updating re calculates an evaluation value based on the position and orientation of the candidate position of the vehicle 10 and the line segment L ^i. The position estimating unit 24, when the end condition evaluation value is assumed to have the minimum value is satisfied, and ends the position and orientation of the update of each line segment L ⁱ of each candidate location and the real space. Then, the position estimating unit 24 estimates the candidate position of the vehicle 10 at the time of each image acquisition corresponding to the minimum evaluation value when the end condition is met as the actual position of the vehicle 10 at the time of image acquisition. , the position and orientation of each line segment L ⁱ in the real space corresponding to the minimum value of the evaluation value, the position and orientation of the real line segment in the real space. The end condition is, for example, when the change amount between the evaluation value based on the previous candidate position of the vehicle 10 and the evaluation value based on the latest candidate position of the vehicle 10 is equal to or less than a predetermined change amount threshold value. When the number of updates of the candidate position reaches a predetermined number, or when the evaluation value becomes a predetermined threshold value or less.

  The position estimation unit 24 updates the candidate position of the vehicle 10 at the time of acquiring each image, for example, according to the Levenberg-Marquardt (LM) method in order to obtain the minimum value of the evaluation value. Or the position estimation part 24 may update the candidate position of the vehicle 10 at the time of each image acquisition according to another optimization method, for example, the steepest descent method or the simulated annealing method.

  The position estimation unit 24 passes the estimated position of the vehicle at the time of each image acquisition (or a rotation matrix and a parallel progression representing the conversion between the vehicle coordinate system and the world coordinate system corresponding to the estimated position) to the map creation unit 25. .

For each set of images for which the estimated position of the vehicle has been obtained by the position estimation unit 24, the map creation unit 25 uses the world coordinates of the line segment in the real space corresponding to each line segment associated between the images included in the set. A map is created by writing the position represented by the system on the map. That is, the mapping unit 25, each line segment L ⁱ in the real space corresponding to the minimum value of the evaluation value may be written to the map.

  Note that the map creation unit 25 uses only the pair of the line segment on the rear image and the line segment on the front image associated with each other based on the condition (3) in the Manhattan-World hypothesis for map creation. It is preferable. Thereby, it is suppressed that the line of structures other than a road surface is written in a map.

  FIG. 9A is a diagram illustrating an example of a road on which a vehicle has traveled, and FIG. 9B uses the map creation device according to the present embodiment corresponding to the road shown in FIG. 9A. It is a figure showing an example of the map created in this way. FIG. 9A shows a road 900 on which the vehicle 10 has traveled. On the other hand, in FIG. 9B, a map 910 is a map created using the map creation device according to the present embodiment, and a dotted line 911 shows the trajectory of the vehicle 10 when the vehicle 10 passes the road 900. Represent.

  As shown in FIG. 9A and FIG. 9B, it can be seen that the shape of the road 900 and the road display shown on the road 900 are reproduced well on the map 910 as well.

  FIG. 10 is an operation flowchart of the map generation process. The control unit 13 executes map generation processing according to the following operation flowchart for each set of three or more images.

The position estimation unit 24 sets an initial value of the candidate position of the vehicle at the time of image acquisition (step S201). Then, the position estimation unit 24 calculates the position of the corresponding line segment in the real space by triangulation according to the candidate position of each image for each line segment associated with each image (step S202). Then, the position estimating unit 24 projects each line segment of the real space on each image according to the candidate position of the vehicle at the time of acquiring the image (step S203). In addition, the position estimation unit 24 calculates, for each line segment in the real space, an angle d _θ ^m formed by the traveling direction of the vehicle at the candidate position of the vehicle at the time of image acquisition and the line segment (step S204).

For each projected line segment in each image, the position estimation unit 24 calculates the sum of the distance between the projected line segment and the corresponding line segment on the image, and each line segment represented in real space. An evaluation value corresponding to the calculated angle d _θ ^m is calculated that is a sum of values corresponding to the likelihood that the line segment is a line segment in a predetermined direction (step S205). Then, the position estimation unit 24 determines whether or not the evaluation value satisfies the end condition (Step S206).

  When the evaluation value does not satisfy the termination condition (No in step S206), the position estimation unit 24 follows the predetermined optimization method so that the evaluation value decreases according to a predetermined optimization method. At least one of each line segment is updated (step S207). And the position estimation part 24 repeats the process after step S202 based on the updated candidate position.

  On the other hand, when the evaluation value satisfies the termination condition (step S206—Yes), the position estimation unit 24 sets the vehicle candidate position at the time of each image acquisition when the evaluation value is minimized as the estimated position of the vehicle. (Step S208).

  The map creation unit 25 creates a map by writing a line segment in the real space corresponding to each line segment associated between the images in accordance with the estimated position of the vehicle at the time of each image acquisition (step S209). Then, the control unit 13 ends the map creation process.

  As described above, the map creation device associates line segments corresponding to lines of the same structure in real space between two images generated at different timings by different cameras. And this map creation device, when the line segment on the real space corresponding to each line segment associated between the images is projected on each image, the positional deviation from the corresponding line segment on the image, and on the real space The evaluation value of the candidate position of the vehicle at the time of each image acquisition is calculated based on the likelihood that the line segment is a line segment in a predetermined direction according to the angle between the line segment and the traveling direction of the vehicle. And this map creation device updates the candidate position of the vehicle at the time of each image acquisition so that the evaluation value becomes the minimum, and the candidate position of the vehicle at the time of each image acquisition when the evaluation value becomes the minimum, respectively The estimated position of the vehicle at the time of each image acquisition. And this map creation apparatus creates a map by projecting the line segment matched between each image on a world coordinate system based on the estimated position of the vehicle at the time of each image acquisition. As described above, this map creation device not only detects the positional deviation from the line segment on the image when the line segment on the real space is projected on each image, but also the line segment in the real space and the traveling direction of the vehicle. Since the evaluation value corresponding to the angle formed is used, the position of the vehicle at the time of acquiring each image can be accurately estimated. As a result, this map creation device can create a highly accurate map.

  The preferred embodiments of the present invention have been described above, but the present invention is not limited to these embodiments. For example, the control unit 13 selects a plurality of front images to be associated with a line segment on one rear image, performs the processing of the projection unit 22 and the association unit 23 on the plurality of front images, The line segments may be associated with each other. Thereby, it becomes possible to associate line segments between three or more front images via one rear image. Therefore, the control unit 13 may perform the processing of the position estimation unit 24 and the map creation unit 25 based on a set of three or more front images. Similarly, the control unit 13 associates line segments between three or more rear images via one front image, and based on a set of three or more rear images, the position estimation unit 24 and You may perform the process of the map preparation part 25. FIG. Further, the control unit 13 may associate the line segments between the images according to another method for associating the line segments between the images, for example, the method described in Non-Patent Document 3.

あるいは、位置推定部２４は、（１６）式における右辺の第２項を次式に示されるもの変更して、評価値を算出してもよい。

あるいはまた、位置推定部２４は、（１６）式における右辺の第２項を次式に示されるもの変更して、評価値を算出してもよい。

According to another modification, the position estimation unit 24 may calculate the evaluation value by changing the second term on the right side in the equation (16) as shown in the following equation.

Alternatively, the position estimation unit 24 may calculate the evaluation value by changing the second term on the right side in the equation (16) shown in the following equation.

Alternatively, the position estimation unit 24 may calculate the evaluation value by changing the second term on the right side in the equation (16) as shown in the following equation.

  According to yet another modification, the control unit 13 uses a probability distribution for a line segment in a predetermined direction between a plurality of images obtained in a period before the map creation process (hereinafter referred to as a learning period). You may learn based on the some line segment matched by.

  FIG. 11 is a schematic configuration diagram of the control unit 13 according to this modification. The control unit 13 according to this modification includes a line segment extraction unit 21, a projection unit 22, an association unit 23, a position estimation unit 24, a map creation unit 25, and a learning unit 26. Compared with the control unit 13 according to the above-described embodiment, the control unit 13 according to this modification is different in that the learning unit 26 is provided. Therefore, the learning unit 26 and related parts will be described below.

The learning unit 26, for each of the line segments associated between the plurality of images obtained during the learning period, similarly to the position estimation unit 24, based on the candidate position of the vehicle at the time of each image acquisition. The position of the line segment in the real space corresponding to the attached line segment is calculated. Note that the learning unit 26 can eliminate the second term ε _x in the equation (6), for example, as a candidate vehicle position at the time of acquiring each image. Then, for each line segment in the real space, the learning unit 26 calculates an angle between the line segment and the vehicle traveling direction determined based on the vehicle candidate position. Then, the learning unit 26 learns the probability distribution for each direction of the line segment based on the angle formed with the traveling direction of the vehicle calculated for each line segment in the real space. In that case, the learning part 26 should just learn probability distribution for every direction of a line segment, for example according to EM method. In addition, when the probability distribution for each direction is represented by one mixed Gaussian distribution, the learning unit 26 may learn the weighting coefficient corresponding to the Gaussian distribution for each direction according to the EM method.
Then, the learning unit 26 stores, in the storage unit 11, a parameter that represents the probability distribution for each line segment direction obtained by the learning. In this case, the position estimation unit 24 may calculate an evaluation value using a parameter that is learned by the learning unit 26 and represents a probability distribution for each direction of the line segment.

  According to this modification, the map creation device determines the probability distribution for each direction of the line segment in the real space based on the line segment detected from the image of the road on which the vehicle actually traveled and the surrounding area. Therefore, the accuracy of the probability distribution for each direction of the line segment can be improved.

  According to another modification, the probability distribution for each direction of the line segment may be represented by a neural network. In this case, the learning unit 26 may learn a probability distribution for each direction of the line segment according to an unsupervised learning method such as a Boltzmann machine. The learning unit 26 stores the neural network parameters (coefficients representing connections between the nodes, biases of the nodes, etc.) obtained by the learning and representing the probability distribution for each direction of the line segment in the storage unit 11. . In this case, the position estimation unit 24 uses the angle formed by the line segment corresponding to the candidate position of the vehicle and the traveling direction of the vehicle in the neural network representing the probability distribution for each direction of the line segment learned by the learning unit 26. The evaluation value may be calculated using the output value obtained by inputting as the value of the second term in equation (16).

  According to this modification, the map creation device can accurately represent the probability distribution even when the probability distribution for each direction of the line segment has a complicated shape. Therefore, since the map creation device can improve the estimation accuracy of the position of the vehicle, as a result, the accuracy of the created map can be improved.

  According to still another modification, the processing of the position estimation unit 24 and the map creation unit 25 may be executed in a device different from the control unit 13, for example, a server provided separately from the vehicle 10. In this case, for each set of images, the acquisition time of each image, the odometry information of the vehicle 10 at each acquisition time, the position and orientation on each image of each line segment associated between the images, Information such as the installation position and focal length is stored in the storage unit 11. These pieces of information may be sent to the server via a communication interface (not shown), and the server may execute the processing of the position estimation unit 24 and the map creation unit 25.

  As described above, those skilled in the art can make various modifications in accordance with the embodiment to be implemented within the scope of the present invention.

1 Map making device 2-1, 2-2 Camera 3 IMU
4 Wheel speed sensor 5 ECU
6 CAN
DESCRIPTION OF SYMBOLS 11 Memory | storage part 12 Communication part 13 Control part 21 Line segment extraction part 22 Projection part 23 Association part 24 Position estimation part 25 Map preparation part 26 Learning part

Claims (7)

A storage unit (11) for storing a probability distribution representing a probability that the line segment is a line segment in a predetermined direction at an angle formed by a traveling direction of the vehicle (10) and a line segment around the vehicle (10); ,
A plurality of line segments from each of a plurality of images around the vehicle (10) generated at different times by at least one imaging unit (2-1, 2-2) attached to the vehicle (10). A line segment extraction unit (21) for extracting
An association unit (23) for associating line segments corresponding to line segments in the same real space among the plurality of line segments extracted from each of the plurality of images between the plurality of images;
A line in the real space according to the direction of the line segment in the real space calculated based on the candidate position of the vehicle (10) at the time of acquisition of each of the plurality of images and the associated line segment The actual angle obtained by inputting the angle and the angle formed by the traveling direction of the vehicle (10) at the candidate position of the vehicle (10) at the time of acquisition of each of the plurality of images into the probability distribution. A position estimation unit that estimates the position of the vehicle (10) at the time of acquisition of each of the plurality of images based on an evaluation value calculated from the likelihood that a line segment in space is a line segment in the predetermined direction. 24)
Based on the estimated position of the vehicle (10) at the time of acquisition of each of the plurality of images, a map is created by obtaining the position of the line segment in the real space corresponding to the associated line segment. A map creation unit (25);
A cartography device having   The predetermined direction includes a first direction parallel to a road extending direction, and a second direction orthogonal to the road extending direction, and the probability distribution includes the first direction at the angle formed by the first direction. 2. The map according to claim 1, comprising: a first probability distribution representing a probability of being a line segment in a direction; and a second probability distribution representing a probability of being a line segment in the second direction at the formed angle. Creation device.   The position estimation unit (24) determines the candidate position of the vehicle (10) when each of the plurality of images is acquired when the evaluation value is the minimum, and the vehicle when each of the plurality of images is acquired. The map creation device according to claim 1, wherein the map creation device estimates the position of (10).   The position estimation unit (24) converts the likelihood and a line segment of the real space for each of the plurality of images into the image based on the candidate position of the vehicle (10) at the time of acquiring the image. The evaluation value is calculated according to any one of claims 1 to 3, wherein the evaluation value is calculated based on a position when projected and a value corresponding to a positional deviation between the associated line segments on the image. Map making device. The line segment extraction unit (21) includes a plurality of second images around the vehicle (10) generated at different times by the at least one imaging unit (2-1, 2-2). , Extract multiple line segments,
The associating unit (23) is a line corresponding to a line segment in the same real space among the plurality of line segments extracted from each of the plurality of second images between the plurality of second images. Match minutes,
Corresponding between the plurality of second images calculated based on the candidate position of the vehicle (10) at the time of acquisition of each of the plurality of second images and the associated line segment The line segment in the real space according to the direction of the line segment in the real space corresponding to the line segment and the candidate position of the vehicle (10) at the time of acquisition of each of the plurality of second images. The map creation device according to any one of claims 1 to 4, further comprising a learning unit (26) that learns the probability distribution based on an angle formed by a traveling direction of the vehicle (10). A plurality of line segments are obtained from each of a plurality of images around the vehicle (10) generated at different times by at least one imaging unit (2-1, 2-2) attached to the vehicle (10). Extracting, and
Correlating line segments corresponding to line segments in the same real space among the plurality of line segments extracted from each of the plurality of images between the plurality of images;
A line in the real space according to the direction of the line segment in the real space calculated based on the candidate position of the vehicle (10) at the time of acquisition of each of the plurality of images and the associated line segment And the angle formed by the traveling direction of the vehicle (10) at the candidate position of the vehicle (10) at the time of acquisition of each of the plurality of images, and the traveling direction of the vehicle (10) and the vehicle The line segment in the real space obtained by inputting into a probability distribution representing the probability that the line segment is a line segment in a predetermined direction at an angle formed by the line segment in the periphery of (10) Estimating a position of the vehicle (10) at the time of acquisition of each of the plurality of images based on an evaluation value calculated from a likelihood that is a line segment;
Based on the estimated position of the vehicle (10) at the time of acquisition of each of the plurality of images, a map is created by obtaining the position of the line segment in the real space corresponding to the associated line segment. Steps,
Including methods. A plurality of line segments are obtained from each of a plurality of images around the vehicle (10) generated at different times by at least one imaging unit (2-1, 2-2) attached to the vehicle (10). Extracting, and
Correlating line segments corresponding to line segments in the same real space among the plurality of line segments extracted from each of the plurality of images between the plurality of images;
A line in the real space according to the direction of the line segment in the real space calculated based on the candidate position of the vehicle (10) at the time of acquisition of each of the plurality of images and the associated line segment And the angle formed by the traveling direction of the vehicle (10) at the candidate position of the vehicle (10) at the time of acquisition of each of the plurality of images, and the traveling direction of the vehicle (10) and the vehicle The line segment in the real space obtained by inputting into a probability distribution representing the probability that the line segment is a line segment in a predetermined direction at an angle formed by the line segment in the periphery of (10) Estimating a position of the vehicle (10) at the time of acquisition of each of the plurality of images based on an evaluation value calculated from a likelihood that is a line segment;
Based on the estimated position of the vehicle (10) at the time of acquisition of each of the plurality of images, a map is created by obtaining the position of the line segment in the real space corresponding to the associated line segment. Steps,
A computer program for alignment for causing a computer to execute.

Images