JP2007249103A - Road image creation system, road image creation method, and road image compositing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a road image compositing technology for compositing an accurate road image even when there is a change in inclination, direction, or speed of a traveling vehicle, and in road gradient. <P>SOLUTION: In a road image creation system, the road image of the road ahead imaged from the traveling vehicle is stored in an image storage means 6. Location coordinate data of the vehicle measured in synchronization with the imaging of each road image is stored in a location history storage means 10. An image conversion means 12 creates an orthographic image (orthographic road image) of the road surface by applying orthographic projection conversion to each road image. An image compositing means 13 determines, according to the location coordinate data of each orthographic road image measured by the vehicle location measuring means, the world coordinate and direction of the orthographic road image, and arranges each orthographic road image on a horizontal plane of projection in the determined world coordinate system respectively to composite a linked road image. At this time, corrections according to the location, attitude, and height of the vehicle are applied to the orthographic projection. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a road image creation technique used for reference when creating a map, and more particularly to a road image creation technique including detailed features such as road markings.

  In recent years, in electronic maps and residential maps used in car navigation systems, etc., detailed information about roads such as road markings such as center lines and outer lines of roads, median strips, road features such as sidewalks, etc. Is required. In order to create such a map on which detailed road information is described, a road image obtained by photographing a road marking or a feature on the road from directly above is required.

  Conventionally, aerial photographs are generally used to obtain road images used for map creation. In this case, since the photographed aerial photograph is a central projection image, ortho (orthogonal projection) processing is performed as seen from directly above to create an orthographic projection aerial photograph. A map is created with reference to the map. In general, since the ground is not a flat surface, it is necessary to perform orthographic projection in consideration of the unevenness of the ground in the ortho process. Therefore, a method has been devised in which the distance from the aircraft to the ground is measured using a laser profiler to measure the unevenness of the ground, and an image is projected onto the uneven surface (see Patent Document 1).

  However, in the method of creating a road image using an orthographic projection aerial photograph obtained from an aerial photograph, when a vehicle is present on the road, a road image hidden under the vehicle cannot be obtained. In addition, images of roads hidden by bridges and roads in tunnels cannot be obtained from aerial photographs. In addition, when there are high features (such as buildings), some roads are hidden behind the features in the central projection image and cannot be photographed. Therefore, it is impossible to obtain a road image of a portion hidden when orthographic projection is performed.

  Therefore, a line camera is mounted on the vehicle traveling on the road, the road surface is photographed from directly above by the line camera, and a linear road surface image elongated in the road width direction is continuously photographed, and these linear images are obtained. Techniques have been devised for acquiring road images by combining them along the traveling direction of the vehicle (see Patent Documents 2 to 4). FIG. 9 shows a configuration of the road image photographing device described in Patent Documents 2 to 4. A line camera 102 is provided in front of the vehicle 101 for photographing the road surface from directly above. Lights 103 and 104 are provided at the lower front of the vehicle 101. When shooting a road image, the vehicle 101 illuminates the road surface to be shot with the lights 103 and 104 while traveling on the road, and the line camera 102 continuously takes a line image of the road surface for one lane width. In this way, a detailed image of the road surface is taken and synthesized on the same principle as the scanner.

On the other hand, in the field of recognition technology such as road marking, a technique for acquiring a road image by synthesizing a two-dimensional image obtained by a camera mounted on a vehicle has been devised (see Patent Document 5). In the recognition apparatus for road display and the like described in Patent Document 5, a landscape image in front of the vehicle is taken at an interval of about 40 frames per second by a CCD camera mounted on the vehicle. An image area of the road portion is cut out from the captured frame images by excluding the image of the preceding vehicle. In this case, when the distance between the preceding vehicle and the own vehicle is narrow, the image area cut out as shown in FIG. 10 is also a narrow belt-like area below the entire image. By cutting out and synthesizing the image area of the road portion in each successive frame, the entire road image is synthesized as shown in FIG. Then, the road marking is extracted from the combined overall road image and is compared with the road marking image stored in the database, thereby recognizing the road marking.
JP 2002-74323 A JP 2002-54911 A JP 2004-258924 A JP 2005-55318 A JP 2003-123197 A Zhengyou Zhang, "A Flexible New Technique for Camera Calibration", Microsoft Research Technical Report MSR-TR-98-71, December 2, 1998, http://research.microsoft.com/~zhang/Papers/TR98-71.pdf

  When obtaining a road image using an aerial photograph, as described above, because the influence of various shielding objects is large, as the image information as the basis for creating electronic maps and residential maps used in car navigation devices etc. Not enough. In addition, in order to obtain necessary information such as detailed road display, it is necessary to shoot at a large scale, and thus there is a problem that costs are increased.

  On the other hand, according to the method of synthesizing the road image using the image taken from the traveling vehicle, the detailed display such as the road display is not affected by the influence of the bridge, the tunnel, the roadside, or the feature or vehicle on the road. Image information can be obtained.

  However, the above-described road image capturing devices of Patent Documents 2 to 4 are used for the purpose of inspecting scratches on the road surface, and the road image captured in one road run is only for one lane. . Therefore, in order to obtain image information related to road markings and the like of all lanes, it is necessary to travel all lanes once, which increases labor and cost. Therefore, it is not suitable for use for the purpose of acquiring a road image for creating a map.

  On the other hand, if the road image composition method used in the recognition apparatus for road display and the like in Patent Document 5 is applied, a composite image of the entire road can be obtained by capturing and composing a relatively wide image while traveling. Can be easily obtained.

  However, the road image synthesizing method of Patent Document 5 is intended for recognition of road display and the like, and the synthesized image does not hold geographical coordinates. Therefore, although it is possible to create a composite image of a local road about several meters ahead of the vehicle for use in road marking recognition, it is not possible to create a wide-area road image for map creation or reference for map creation .

  Further, in the road image synthesizing method disclosed in Patent Document 5, as shown in FIG. 11, a synthesized image is created by simply joining the cut-out partial images. However, since the original road image was taken obliquely from above, the scale of the image becomes smaller as it goes in the depth direction, unlike the orthographic projection. Accordingly, when the partial images are simply joined together, the subject images are not continuously connected at the connection portion, and become a jagged image (see FIG. 11E). In addition, since an image viewed from an oblique direction is obtained instead of an image viewed from directly above, the obtained synthesized image itself is a distorted image. Therefore, it is difficult to use it as a reference for road markings for creating maps, and it cannot be used as it is.

  Accordingly, an object of the present invention is to provide a road image synthesis technique for synthesizing an accurate road image that can be used for map creation or the like using an image taken from a traveling vehicle.

The first configuration of the road image creation system according to the present invention is an imaging unit that images a road from a vehicle in front, rear, or side of the vehicle and outputs it as a two-dimensional frame image (hereinafter referred to as “road image”). ,
Vehicle position measuring means for measuring position coordinates of the vehicle and outputting as position coordinate data;
Image conversion means for generating an orthogonal image on a horizontal projection plane (hereinafter referred to as an “orthogonal road image”) by performing an orthogonal projection conversion on the road image captured by the imaging means;
Based on the position coordinate data measured by the vehicle position measuring means for each orthographic road image, the world coordinates and orientation of the orthographic road image are determined, and each orthographic road image is determined by the determined world coordinates. And image synthesizing means for synthesizing the connected road images by arranging them on the horizontal projection plane of the system.

  According to this configuration, the road image is orthographically projected by the image conversion means, and is converted into an orthogonal road image viewed from directly above. Each obtained orthographic road image is arranged on the projection plane of the world coordinate system with the position (the background coordinates) and the image orientation in the world coordinate system determined according to the position coordinate data. Thereby, each orthogonal road image can be pasted together and a connection road image can be combined.

  In this way, by combining the orthographic road images using the position coordinate data measured by the vehicle position measuring means, it is possible to accurately connect the roads even when the direction and speed of the vehicle change during the shooting of the road images. Images can be combined.

  In addition, since each road image to be photographed is converted into an orthographic road image by the image conversion means, when these orthographic road images are combined, there is little shift in the subject image at the joint between the images. Therefore, a composite image in which subject images are smoothly connected is obtained. Furthermore, since the subject image is converted into an image viewed from infinity just above by orthogonal transformation, it is possible to synthesize an accurate road image that can be used for map creation without distortion of the subject image. .

  Further, by using a two-dimensional frame image obtained by imaging a road ahead, behind or on the side of the vehicle, images of a plurality of lanes can be taken simultaneously. Therefore, the labor and cost required for taking a road image are reduced.

  Here, as the “imaging means”, a digital camera, a video camera, or the like can be used. As the “vehicle position measuring means”, a hybrid positioning device that combines a radio navigation unit such as a global positioning system (GPS) and an inertial navigation unit such as a gyroscope or an accelerometer can be used. .

  As the “vehicle position coordinates”, two-dimensional coordinates (the latitude coordinates) or three-dimensional coordinates (the latitude coordinates plus the height) can be used.

  As the method of “orthogonal projection transformation”, a normal ortho transformation method can be used. For example, as a typical example, a Zhang calibration method described later can be used (see Non-Patent Document 1).

  As a method of determining the “direction of the orthographic road image” by the image synthesizing means, the traveling direction of the vehicle is obtained from the position coordinate of the target orthographic road image and the position coordinate of the orthophoto road image immediately before that, Measured by measuring the rotation angle of the vehicle in the pitch direction, roll direction, and / or heading direction by using a method of setting the traveling direction as the direction of the orthographic road image or using a gyroscope or the like. A method of determining the direction of the orthogonal road image from these rotation angles can be adopted.

  In the present invention, there is provided synchronization means for outputting a synchronization signal for synchronizing the timing at which the imaging means captures the road image and the timing at which the vehicle position measurement means measures the position coordinates, and the imaging means and The vehicle position measuring means can synchronously capture a road image and measure the position coordinates of the vehicle.

  Thereby, since the position coordinate data is obtained on a one-to-one basis for each road image, it becomes easy to specify the position coordinates of the road imaged on the road image.

  As the “synchronization means”, a timer that outputs a trigger pulse as a synchronization signal at a predetermined time interval or travel distance interval can be used.

In the present invention, the time measuring means for measuring the time when the imaging means images each road image and the time when the vehicle position measuring means measures the position coordinates;
Position coordinate calculation means for calculating the position coordinate data of the vehicle when each road image is captured by temporally interpolating each position coordinate data measured by the vehicle position measurement means;
With
The image composition means determines the world coordinates and orientation of the orthographic road image based on the position coordinate data calculated by the position coordinate calculating means for each orthographic road image, The connected road images may be synthesized by placing them on the horizontal projection plane of the determined world coordinate system.

  Even with such a configuration, since the position coordinate data is obtained on a one-to-one basis for each road image, it is easy to specify the position coordinates of the road imaged on the road image.

A second configuration of the road image creation system according to the present invention includes, in the first configuration, posture angle measurement means that measures the posture angle of the vehicle and outputs the posture angle data.
The image conversion means adds the correction based on the attitude angle data, orthorectifies the road image captured by the imaging means, and generates the orthographic road image.

  With this configuration, it is possible to prevent deformation that occurs in the orthographic road image due to a change in the posture angle of the vehicle, and to obtain an accurate orthographic road image.

  That is, when a road image is shot while the vehicle is actually running, the posture angle of the imaging means with respect to the road surface changes due to the vehicle shaking or the suspension sinking. Therefore, if the captured road image is orthographically projected uniformly, the orthographically projected road image is distorted.

For example, as shown in FIG. 12, when the posture angle of the imaging unit 202 is changed by the vibration of the traveling time of the vehicle 201, the visual field f ₂ of the imaging unit 202 that is fixed to the vehicle, if there is no change in the attitude angle deviate from the field of view f ₁ is rotated in the pitch direction. Thereby, distortion and a shift | offset | difference arise in the image | photographed road image like Fig.12 (a). If the orthogonal projection conversion is performed without considering the change in the posture angle, the image shifts in the vehicle traveling direction and is enlarged / reduced in the vehicle traveling direction as shown in FIG. Accordingly, a deviation occurs with respect to the image position and the scale in each axial direction, and accurate image composition cannot be performed.

  Therefore, in the image conversion means, correction based on the posture angle of the vehicle measured by the posture angle measurement means is performed, and each road image picked up by the image pickup means is orthographically projected, as shown in FIG. It is possible to generate an accurate orthographic projection road image.

  Here, the “vehicle attitude angle” refers to a vehicle Euler angle, for example, a rotation angle in a heading direction, a pitch direction, and a roll direction. As the “posture angle measuring means”, a gyro sensor, an accelerometer sensor, a laser vehicle height meter, or the like can be used. Further, as a method of “correction based on attitude angle data”, for example, a Zhang calibration method described later can be used (see Non-Patent Document 1).

A third configuration of the road image creation system according to the present invention includes vehicle height measuring means for measuring the vehicle height of the vehicle and outputting vehicle height data in the first or second configuration,
The image conversion means adds the correction based on the vehicle height data, orthorectifies the road image captured by the imaging means, and generates the orthographic road image.

  With this configuration, it is possible to prevent the orthographic road image from being deformed due to a change in the vehicle height of the vehicle, and to obtain an accurate orthographic road image.

  That is, when a road image is taken while the vehicle is actually running, the height of the imaging means from the road surface changes due to the shaking of the vehicle or the sinking of the suspension. Therefore, if the captured road image is orthographically projected uniformly, the orthographically projected road image is distorted.

For example, as shown in FIG. 13, when the vehicle 201 sinks from the normal position due to the vertical vibration of the vehicle 201, the field of view f ₂ of the imaging unit 202 fixed to the vehicle 201 is the case where the vehicle 201 does not sink. displaced in parallel to the up-and-down direction of the field of view f _1. Therefore, the road surface imaged by the imaging unit 202 is a road surface in front of the road image imaged from a normal height (see FIG. 13A). For this reason, when the height of the vehicle 201 is constant and this is orthogonally projected, the image shifts in the vehicle traveling direction and slightly expands or contracts in the vehicle traveling direction as shown in FIG. Therefore, accurate image composition cannot be performed.

  Therefore, in the image conversion means, the correction based on the vehicle height measured by the vehicle height measurement means is added, and each road image captured by the imaging means is orthographically projected, as shown in FIG. It is possible to generate an accurate orthographic projection road image. In addition, even when the vehicle 201 is lifted from the normal position, it is possible to generate an accurate orthographic road image similarly.

  Here, as the “vehicle height measuring means”, a laser vehicle height meter or the like can be used. As a method of “correction based on vehicle height data”, for example, a Zhang calibration method described later can be used (see Non-Patent Document 1).

According to a fourth configuration of the road image creation system according to the present invention, in any one of the first to third configurations, the vehicle position measuring means calculates a three-dimensional position coordinate including a background coordinate and an altitude of the vehicle. Measure and output as position coordinate data,
Image storage means for storing a road image output by the imaging means;
Position history storage means for storing position coordinate data output by the vehicle position measurement means;
With
For each road image stored in the image storage means, the image conversion means reads out the position coordinate data of the road surface that is the subject of the road image or a plurality of points around the road image from the position history storage means, Obtaining the inclination of the road surface that is the vehicle and the subject from the coordinate data, and applying the orthographic projection conversion with correction by the inclination to the road image picked up by the image pickup means, thereby generating the orthogonal road image. Features.

  According to this configuration, the image conversion means obtains the inclination of the road that is the subject from the altitude difference between the imaging position of the road image and the position of the road that is the subject. Then, correction by inclination is applied, and the road image is orthographically converted. Thereby, when the inclination of the road ahead of the vehicle is changing, it is possible to prevent deformation that occurs in the orthographic road image due to the change in the inclination, and to obtain a high-precision orthographic road image.

  That is, since a general road has a gradient and a gradient change, the resultant composite image has a large error. In particular, there is a problem that a large error occurs in the composite image when the vehicle to be photographed is inclined due to road surface undulations or unevenness.

  For example, as shown in FIG. 14, when the road surface ahead of the vehicle 201 is raised, the road image captured by the imaging unit 202 is reduced in the traveling direction as illustrated in FIG. When this is converted into an image on a horizontal plane by a simple orthographic projection (orthogonal projection assuming that the road surface is a horizontal plane), the actual road (FIG. 14 (c)) is more than shown in FIG. 14 (b). The image is stretched in the traveling direction. Therefore, an accurate road image on the orthographic projection plane cannot be obtained.

  In view of this, in the image conversion means, the correction by the inclination of the subject road surface calculated from the position coordinate data of the vehicle is added, and each road image captured by the imaging means is orthographically projected, so that FIG. Thus, it is possible to generate an accurate orthographic projection road image. Note that it is possible to generate an orthographic projection road image in the same manner when the road surface is raised or raised, as well as when the road surface is raised.

Here, as a method of “determining the inclination of the vehicle and the road surface as the subject from the position coordinate data of the road surface as the subject or a plurality of points around it”, for example, as shown in FIG. the position P ₁ of the vehicle when the imaging surface S _o, determining the slope line L _a in the object road surface S _o to approximately by linear approximation from the position coordinates of the position P ₃ of the vehicle in the object road surface S _o the method, as in FIG. 15 (b), the position _P 2 of the vehicle in the object road surface _{S o, P 3, P 4} , gradient line in the object road surface _{S o} by linear approximation from the position coordinates of the _{P 5 L b} For example, a method of approximately calculating the value can be used.

A first configuration of a road image composition device according to the present invention is a plurality of roads composed of two-dimensional frame images obtained by imaging a road ahead or behind or side of the vehicle from a traveling vehicle at a predetermined time interval. From the image and the position coordinate data indicating the position coordinate of the vehicle measured in synchronization with the shooting timing of each road image, a connected road image obtained by orthographically projecting the road image onto the plane of the world coordinate system is pasted. A road image composition device for composition,
Image storage means for storing each road image;
Position history storage means for storing each position coordinate data;
Image conversion means for generating an orthogonal image on the road surface (hereinafter referred to as an “orthogonal road image”) by performing an orthogonal projection conversion on each of the road images;
Based on the position coordinate data measured by the vehicle position measuring means for each orthographic road image, the world coordinates and orientation of the orthographic road image are determined, and each orthographic road image is determined in the determined world coordinate system. Image composing means for composing a connected road image by disposing on a horizontal projection surface.

In a second configuration of the road image composition device according to the present invention, in the first configuration, the position history storage means stores the posture angle of the vehicle measured in synchronization with the shooting timing of each road image. The attitude angle data shown is stored,
The image conversion means adds the correction based on the attitude angle data, orthorectifies the road image captured by the imaging means, and generates the orthographic road image.

According to a third configuration of the road image composition device according to the present invention, in the first or second configuration, the position history storage unit measures the vehicle of the vehicle measured in synchronization with the shooting timing of each road image. Vehicle height data indicating the height is stored,
The image conversion means adds the correction based on the vehicle height data, orthorectifies the road image captured by the imaging means, and generates the orthographic road image.

According to a fourth configuration of the road image synthesizing apparatus according to the present invention, in any one of the first to third configurations, the position history storage means includes a three-dimensional position coordinate composed of a latitude coordinate and an altitude of the vehicle. Is stored as the position coordinate data,
For each road image stored in the image storage means, the image conversion means reads out the position coordinate data of a road surface or a plurality of points around the road image as a subject of the road image from the position history storage means, The orthographic road image is generated by obtaining the inclination of the vehicle and the road surface that is the subject from the position coordinate data, and subjecting the road image captured by the imaging means to the orthographic projection conversion with correction by the inclination. It is characterized by that.

  A road image composition program according to the present invention is read and executed by a computer, thereby causing the computer to function as a road image composition apparatus having any one of the first to fourth configurations.

The first configuration of the road image creation method according to the present invention is to capture a two-dimensional frame image (hereinafter referred to as “road image”) by imaging an image of a road ahead, rear or side of the vehicle from the vehicle at a predetermined time interval. Imaging step for generating
Vehicle position measuring means for measuring position coordinates of the vehicle and generating position coordinate data in synchronism with timing when the image pickup means images;
An image conversion step of generating an orthographic image on the road surface (hereinafter referred to as an “orthogonal road image”) by performing orthographic projection conversion on the road image captured by the imaging means;
Based on the position coordinate data measured in the vehicle position measurement step with respect to each orthographic road image, the world coordinates and direction of the orthographic road image are determined, and each orthographic road image is determined in each world coordinate And an image synthesis step of synthesizing the connected road images by arranging them on the horizontal projection plane of the system.

In a second configuration of the road image creation method according to the present invention, in the first configuration, the posture angle that measures the posture angle of the vehicle and generates the posture angle data in synchronization with the timing of the imaging by the imaging unit. Measuring step,
In the image conversion step, correction based on the posture angle data is performed, and the road image captured by the imaging unit is orthographically projected to generate the orthographic road image.

According to a third configuration of the road image creation method of the present invention, in the first or second configuration, the vehicle height of the vehicle is measured and the vehicle height data is generated in synchronization with the timing of the imaging by the imaging means. A vehicle height measuring step,
In the image conversion step, correction based on the vehicle height data is performed, and the road image captured by the imaging unit is orthographically projected to generate the orthographic road image.

According to a fourth configuration of the road image creation method of the present invention, in any one of the first to third configurations, in the imaging step, a road image obtained by imaging a road from a vehicle is stored as image storage means. Remember once,
In the vehicle position measurement step, the position coordinate data obtained by measuring the three-dimensional position coordinates consisting of the background coordinates and altitude of the vehicle is temporarily stored in the position history storage means,
In the image conversion step, for each road image stored in the image storage means, the position coordinate data of a plurality of points around or on the road surface that is the subject of the road image is read from the position history storage means, The orthographic road image is generated by obtaining the inclination of the vehicle and the road surface that is the subject from the position coordinate data, and applying the orthographic projection conversion corrected by the inclination to the road image captured by the imaging means. It is characterized by doing.

  As described above, according to the present invention, the direction and speed of the vehicle change during shooting of the road image by synthesizing each orthogonal road image using the position coordinate data measured in synchronization with the road image. Even in such a case, it is possible to synthesize an accurate linked road image.

  In addition, it is possible to prevent a deformation that occurs in the orthographic road image due to a change in the attitude angle of the vehicle, a change in the vehicle height, or a change in the inclination of the road ahead of the vehicle, thereby obtaining an accurate orthographic road image.

  The best mode for carrying out the present invention will be described below with reference to the drawings.

  FIG. 1 is a diagram illustrating a configuration of a road image creation system according to the first embodiment of the present invention. The road image creation system 1 includes two parts, a road image collection device 3 mounted on a vehicle 2 and a road image composition device 4. The road image composition device 4 is specifically configured by a computer, and is functionally realized by reading a road image composition program into the computer and executing it. The road image composition device 4 may be mounted on the vehicle 2 or may be installed separately from the road image composition device 4.

  The road image collection device 3 includes an imaging unit 5, an image storage unit 6, a vehicle position measurement unit 7, an attitude angle measurement unit 8, a vehicle height measurement unit 9, a position history storage unit 10, and a synchronization unit 11.

  The imaging means 5 is composed of a CCD camera and images a road ahead of the vehicle 2. The image storage unit 6 stores the road image captured by the imaging unit 5. The vehicle position measuring means 7 is composed of a hybrid positioning device including a global positioning system (GPS), a gyro sensor, an acceleration sensor, and the like. Measure.

  The posture angle measuring means 8 is a device that measures the posture angle of the vehicle 2. The posture angle is expressed by a rotation angle in a heading direction, a pitch direction, and a roll direction. As the attitude angle measuring means 8, a gyro sensor, an accelerometer sensor, a laser vehicle height meter, or the like can be used. The vehicle height measuring means 9 measures the vehicle height of the vehicle 2. As the vehicle height measuring means 9, a laser vehicle height meter is used. The position history storage unit 10 stores the position coordinate data output from the vehicle position measurement unit 7 and the attitude angle data output from the attitude angle measurement unit 8.

  The synchronization means 11 is a predetermined time interval (for example, 1/30 sec interval) that determines the image pickup timing of the image pickup means 5 and the measurement timing of the vehicle position measurement means 7, the attitude angle measurement means 8, and the vehicle height measurement means 9. To output. The imaging means 5, the vehicle position measuring means 7, the attitude angle measuring means 8, and the vehicle height measuring means 9 perform imaging or measurement in synchronization with this synchronization signal.

  On the other hand, the road image composition device 4 includes an image conversion means 12, an image composition means 13, a composite image storage means 14, and an output device 15.

  The image conversion means 12 reads the road image stored in the image storage means 6, performs camera calibration and orthographic projection conversion, and creates an orthographic road image. The orthographic road image is a road image when the viewpoint is placed at an infinitely far point above the subject road. In the camera calibration and the orthographic projection conversion, the inclination angle of the subject road obtained from the attitude angle data and the elevation data of the vehicle 2 at the time of capturing the road image is referred to, and correction using these parameters is performed.

  The image synthesizing unit 13 synthesizes the connected road images by arranging the orthographic road images obtained by orthographic projection of the road images on the horizontal projection plane of the world coordinate system based on the position coordinates. The combined linked road image is stored in the combined image storage unit 14. The output device 15 is configured by a display or the like, and displays and outputs a connected road image.

  Regarding the road image creation system 1 according to the present embodiment configured as described above, a road image creation method based on the system will be described below.

  First, a road image is picked up by the image pickup means 5 while traveling on the road by the vehicle 2. The captured road image is stored in the image storage means 6.

  FIG. 2 is an example of a road image taken from the vehicle 2. As shown in FIG. 2, an image of a road obliquely ahead of the vehicle 2 is captured as the road image. Note that the shooting direction is not limited to this, and may be a road on the rear or side of the vehicle 2. In the road image of FIG. 2, the image area used for creating the connected road image is a portion of the area S surrounded by a rectangular frame. In this example, an image of an area 5 to 7 m ahead of the imaging means 5 is used.

  The imaging unit 5 captures a road image in synchronization with the synchronization signal output from the synchronization unit 11. On the other hand, at the same time, the vehicle position measuring means 7, the posture angle measuring means 8, and the vehicle height measuring means 9 measure and output the position coordinates, posture angle, and vehicle height of the vehicle 2 in synchronization with the synchronization signal. Each measured data is stored in the position history storage means 10.

  As described above, after collecting information such as road images and their imaging positions, the combined road images are synthesized by the road image synthesizing device 4.

  First, the image conversion means 12 orthorectifies each road image to create an orthographic road image, and then the image composition means 13 arranges these orthographic road images on the horizontal projection plane of the world coordinate system. As a result, the connected road image is synthesized.

  The orthographic projection of the road image by the image conversion means 12 is performed based on the following Zhang calibration. First, it is assumed that the vehicle 2 is traveling horizontally and the subject road is on the same horizontal plane.

Let m = [u v] ^T be the two-dimensional coordinate on the road image screen. Let the three-dimensional coordinates of the world coordinate system be M = [XYZ] ^T. A vector obtained by adding one element to each coordinate as a direct product is defined as follows.

The relationship between the three-dimensional coordinate M and the two-dimensional coordinate m of the projected image is modeled by the following relational expression.

  Here, s is a scale factor, [R t] is an extrinsic parameter matrix, R is a rotation matrix, t is a translation matrix, and A is an intrinsic parameter matrix.

The internal parameter matrix A is an internal parameter of the imaging unit 5 and represents a mapping parameter from the real image coordinate system (xy coordinate system) to the frame coordinate system (uv coordinate system). α and β are the scale factors in the u-axis and v-axis directions, γ is a parameter represented by the skew of the two image axes, and [u ₀ , v ₀ ] ^T is the coordinate of the principal point (principal point) of the image ( Principal point coordinates). If the pixel size is (k _u , k _v ), the angle between the u axis and the v axis is θ, and the focal length is f, α, β, and γ are expressed by the following equations.

  The external parameter matrix [R t] is an external parameter of the imaging unit 5 and represents a mapping parameter from the world coordinate system (XYZ coordinate system) to the real image coordinate system (xy coordinate system). In the world coordinate system, the road surface immediately below the imaging means 5 is the origin, the horizontal axis perpendicular to the traveling direction of the vehicle 2 is the X axis, the vertical axis is the Y axis, and the horizontal axis in the traveling direction of the vehicle 2 is the Z axis. . The parallel movement vector t is a movement vector of the image principal point of the real image with respect to the origin in the world coordinate system. If the height of the imaging means 5 (the height of the image principal point of the real image) is h, the translation vector t is expressed by the following equation.

  In the world coordinate system, if the rotation angle (yaw angle) in the heading direction of the real image is φ, the pitch angle is ω, and the roll angle is κ, the rotation matrix R is expressed by the following equation.

The internal parameter matrix A is known from prior measurements. The yaw angle φ, pitch angle ω, and roll angle κ are the posture angles if the yaw angle φ ₀ , pitch angle ω ₀ , and roll angle κ _{0 in} the initial state (the vehicle 2 is on the horizontal ground) are known. It can be calculated from the attitude angle data measured by the measuring means 8. Further, the height h of the image principal point can be easily calculated from the height h ₀ of the image principal point in the stationary state on the horizontal plane obtained by the prior measurement and the vehicle height measured by the vehicle height measuring means 9. it can.

  Therefore, the image conversion means 12 converts the road image p in the frame coordinate system (uv coordinate system) into the projected road image P in the world coordinate system (XYZ coordinate system) using Equation (2) using these parameters. To do. A specific conversion method can be performed as follows.

  First, it is assumed that the road surface as a subject is a horizontal plane (Y = 0). From equation (2)

であるので、ピクセル（ｕ，ｖ）に対する世界座標（Ｘ，Ｚ）及びスケールパラメータｓは次式により求めることができる。

Therefore, the world coordinates (X, Z) and the scale parameter s for the pixel (u, v) can be obtained by the following equation.

Next, the image conversion means 12 performs correction in consideration of the inclination of the road surface that is the subject as follows. First, with reference to the position coordinate data measured by the vehicle position measuring means 7, the position coordinates (X ₀ , Y ₀ , Z ₀ ) of the point on the road surface immediately below the imaging means 5 at the time of shooting the road image p, The gradient of the road surface that is the subject is calculated from the position coordinates (X _i , Y _i , Z _i ) of a plurality of points in the vicinity of the road surface that is the subject. In this embodiment, in order to simplify the model, it is assumed that the gradient is uniform as shown in FIG. Specifically, the world coordinate point on the horizontal plane determined by equation (8) (X, Y, Z) from the position coordinate data in the vicinity, seeking Z, calculated altitude difference Delta] h by Δh = Z-Z _0, which Is an index representing the gradient.

  From FIG. 3, the depth Z ′ of the world coordinate point (X ′, Y ′, Z ′) on the road surface corrected with a uniform road gradient can be obtained by the following equation.

  When the corrected depth Z ′ on the road surface is determined, the relationship between the frame coordinate point (u, v) and the world coordinate point (X ′, Y ′, Z ′) is expressed by the following equation from Equation (2). Become.

  Thus, X ′ and Y ′ of the world coordinate point can be calculated by the following formula.

  In this way, even when the inclination angle of the road surface in front of the vehicle 2 is changing, distortion generated in the orthographic road image by the correction can be minimized. In addition, image distortion due to the sinking of the vehicle 2 or a change in posture angle can be minimized.

  For example, it is assumed that the road image captured by the imaging unit 5 is an image as shown in FIG. An image region used for combining connected road images is defined as region S. The image conversion means 12 generates an orthographic road image as shown in FIG. 4B by projecting the image of the region S onto the horizontal projection plane of the world coordinate system using orthographic projection conversion as described above. Since the road images are intermittently photographed at predetermined intervals along the traveling direction of the vehicle 2, a plurality of orthographic road images can be obtained as shown in FIG. 5 by performing similar orthographic projection conversion on each road image. Generated. The position coordinate data measured by the vehicle position measuring unit 7 and the posture angle data measured by the posture angle measuring unit 8 correspond to each orthogonal road image.

  The image synthesizing unit 13 synthesizes each orthogonal road image according to the corresponding position coordinate data and posture angle data. The arrangement position of the orthogonal road image on the horizontal projection plane can be specified by referring to the position coordinate data, and the arrangement direction of the orthogonal road image can be specified by referring to the posture angle data. .

  As a result, a coupled road image as shown in FIG. 6 is synthesized. The generated coupled road image is stored in the composite image storage means 14 and displayed on the output device 15.

  FIG. 7 shows an example of a connected road image actually synthesized by the road image creation system 1 according to the present embodiment. In this way, by synthesizing road images taken from the traveling vehicle 2, a road image like an aerial photograph taken with an aircraft can be obtained.

  In addition, since the imaging distance from the imaging means 5 to the subject is short, a fine image can be obtained at low cost.

  Furthermore, since the image is taken from the traveling vehicle 2, it is possible to eliminate the influence of high-rise features, shielding objects such as bridges and tunnels.

  For example, the road image of the loop portion of the interchange is synthesized as shown in FIG. FIG. 8A is a view of the composite road image viewed from infinity in the vertical direction. FIG. 8B is a view of the composite road image of FIG. Each trapezoid represents an orthogonal road image obtained by orthogonal projection conversion of the road image captured by the imaging means 5.

  In an image obtained by orthographic projection of an aerial photograph, an image of the road below the intersection of the loops cannot be obtained. However, in the road image creation system 1 of the present embodiment, the number of composite road images can be created as a three-dimensional image as shown in FIG. Can be reproduced.

It is a figure showing the structure of the road image creation system which concerns on Example 1 of this invention. 3 is an example of a road image taken from a vehicle 2; It is explanatory drawing which shows a gradient correction model. It is a figure explaining the example of orthographic projection conversion. It is a figure which shows the example of the orthographic road image obtained by orthographic projection. It is a figure which shows the example of a connection road image. It is a figure which shows the example of the connection road image actually synthesize | combined by the road image creation system which concerns on a present Example. It is a figure which shows the example which synthesize | combined the road image of the loop part of an interchange. It is a lineblock diagram of a road image photographing device given in patent documents 2-4. It is a figure explaining the synthetic | combination method of the road image by the recognition apparatus of the road display etc. of patent document 5. FIG. It is a figure explaining the synthetic | combination method of the road image by the recognition apparatus of the road display etc. of patent document 5. FIG. It is a figure explaining distortion of an orthographic projection image which arises when the road surface ahead of vehicles is rising. It is a figure explaining distortion of an orthographic projection image which arises by the sinking of a vehicle. It is a figure explaining the distortion of the orthographic projection image which arises by the change of the attitude angle of a vehicle. It is explanatory drawing of the method of calculating | requiring the inclination of the road surface which is a vehicle and a to-be-photographed object from the position coordinate data of the road surface which is a to-be-photographed object, or the several surrounding points.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Road image creation system 2 Vehicle 3 Road image collection device 4 Road image composition apparatus 5 Imaging means 6 Image storage means 7 Vehicle position measurement means 8 Attitude angle measurement means 9 Vehicle height measurement means 10 Position history storage means 11 Synchronization means 12 Image conversion Means 13 Image composition means 14 Composite image storage means 15 Output device

Claims (13)

Imaging means for imaging a road ahead or behind or on the side of the vehicle and outputting it as a two-dimensional frame image (hereinafter referred to as “road image”);
Vehicle position measuring means for measuring position coordinates of the vehicle and outputting as position coordinate data;
Image conversion means for generating an orthographic image on a horizontal projection plane (hereinafter referred to as an "orthogonal road image") by orthographic projection conversion of the road image captured by the imaging means;
Based on the position coordinate data measured by the vehicle position measuring means for each orthographic road image, the world coordinates and orientation of the orthographic road image are determined, and each orthographic road image is determined by the determined world coordinates. Image combining means for combining connected road images by placing them on a horizontal projection surface of the system;
A road image creation system characterized by comprising: Posture angle measuring means for measuring the posture angle of the vehicle and outputting it as posture angle data;
2. The road according to claim 1, wherein the image conversion unit performs correction based on the posture angle data, orthorectifies the road image captured by the imaging unit, and generates the orthogonal road image. Image creation system. Vehicle height measuring means for measuring the vehicle height of the vehicle and outputting vehicle height data;
The said image conversion means adds the correction | amendment by the said vehicle height data, carries out orthographic projection conversion of the road image which the said imaging means images, and produces | generates the said orthophoto road image. Road image creation system. The vehicle position measuring means measures three-dimensional position coordinates composed of the background coordinates and altitude of the vehicle and outputs it as position coordinate data.
Image storage means for storing a road image output by the imaging means;
Position history storage means for storing position coordinate data output by the vehicle position measurement means;
With
For each road image stored in the image storage means, the image conversion means reads out the position coordinate data of the road surface that is the subject of the road image or a plurality of points around the road image from the position history storage means, Obtaining the inclination of the road surface that is the vehicle and the subject from the coordinate data, and applying the orthographic projection conversion with correction by the inclination to the road image picked up by the image pickup means, thereby generating the orthogonal road image. The road image creation system according to any one of claims 1 to 3. From a plurality of road images composed of two-dimensional frame images obtained by imaging a road ahead or behind or sideways of the vehicle from a traveling vehicle at predetermined intervals, and position coordinate data indicating the position coordinates of the vehicle, A road image synthesizing device that synthesizes a connected road image obtained by orthogonally projecting road images onto a plane of the world coordinate system,
Image storage means for storing each road image;
Position history storage means for storing each position coordinate data;
Image conversion means for generating an orthogonal image on the road surface (hereinafter referred to as an “orthogonal road image”) by performing an orthogonal projection conversion on each of the road images;
Based on the position coordinate data measured by the vehicle position measuring means for each orthographic road image, the world coordinates and orientation of the orthographic road image are determined, and each orthographic road image is determined in the determined world coordinate system. An image synthesis means for synthesizing a connected road image by arranging on a horizontal projection surface;
A road image composition apparatus comprising: The position history storage means stores attitude angle data indicating the attitude angle of the vehicle,
The road according to claim 5, wherein the image conversion unit performs correction based on the posture angle data, orthorectifies the road image captured by the imaging unit, and generates the orthographic road image. Image composition device. The position history storage means stores vehicle height data indicating the vehicle height of the vehicle,
The said image conversion means adds correction | amendment by the said vehicle height data, carries out orthographic projection conversion of the road image which the said imaging means images, and produces | generates the said orthographic road image. Road image synthesizer. In the position history storage means, three-dimensional position coordinates consisting of the background coordinates and altitude of the vehicle are stored as the position coordinate data,
For each road image stored in the image storage means, the image conversion means reads out the position coordinate data of a road surface or a plurality of points around the road image as a subject of the road image from the position history storage means, The orthographic road image is generated by obtaining the inclination of the vehicle and the road surface that is the subject from the position coordinate data, and subjecting the road image captured by the imaging means to the orthographic projection conversion with correction by the inclination. The road image composition device according to any one of claims 5 to 7, wherein   A road image composition program which causes a computer to function as the road image composition apparatus according to any one of claims 5 to 8 by being read and executed by a computer. An imaging step of capturing a two-dimensional frame image (hereinafter referred to as “road image”) by capturing an image of a road ahead, behind or side of the vehicle from the vehicle at a predetermined interval;
Vehicle position measuring means for measuring position coordinates of the vehicle and generating position coordinate data;
An image conversion step of generating an orthographic image on the road surface (hereinafter referred to as an “orthogonal road image”) by performing orthographic projection conversion on the road image captured by the imaging means;
Based on the position coordinate data measured in the vehicle position measurement step with respect to each orthographic road image, the world coordinates and direction of the orthographic road image are determined, and each orthographic road image is determined in each world coordinate An image synthesizing step for synthesizing a connected road image by placing the system on a horizontal projection surface;
A road image creation method comprising: A posture angle measuring step of measuring a posture angle of the vehicle and generating posture angle data;
11. The orthographic road image is generated according to claim 10, wherein in the image converting step, the road image picked up by the image pickup means is subjected to orthographic projection conversion by adding correction based on the attitude angle data, and the orthographic road image is generated. Road image creation method. A vehicle height measurement step of measuring vehicle height of the vehicle and generating vehicle height data;
12. In the image conversion step, a correction based on the vehicle height data is added, and a road image captured by the imaging unit is orthographically projected to generate the orthogonal road image. The road image creation method described. In the imaging step, the road image obtained by imaging the road from the vehicle is temporarily stored in the image storage means,
In the vehicle position measurement step, the position coordinate data obtained by measuring the three-dimensional position coordinates consisting of the background coordinates and altitude of the vehicle is temporarily stored in the position history storage means,
In the image conversion step, for each road image stored in the image storage means, the position coordinate data of a plurality of points around or on the road surface that is the subject of the road image is read from the position history storage means, The orthographic road image is generated by obtaining the inclination of the vehicle and the road surface that is the subject from the position coordinate data, and applying the orthographic projection conversion corrected by the inclination to the road image captured by the imaging means. The road image creation method according to any one of claims 10 to 12, characterized by:

Images