JP5281424B2 - Road marking map generation method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To obtain the position coordinate of a road marking by generating an image including the road marking from images of road surface photographed while being traveled on a road. <P>SOLUTION: In the road marking map generation method, the road surface is photographed with a video camera while being traveled on the road and the position coordinate of each photographing point is obtained with a GPS or the like. A computer converts each frame image of the moving image to create an orthographic image in the state of being viewed from directly above and places it at the photographing point to create a connected image along the running path. Also, the computer creates a wide road image by combining connected images of different paths. In this road image, when the connected images overlap each other and the road marking A153 of a lower connected image is concealed, a transparency polygon POL15 is automatically set on such portions. Thus, by making effective use of the road marking included in each connected image, a road image can be created, and the position coordinate of the road marking can be easily obtained. <P>COPYRIGHT: (C)2010,JPO&amp;INPIT

Description

  The present invention relates to a technique for generating a road marking map including a marking applied to a road surface.

  Electronic map data used in car navigation or the like requires various detailed data in order to realize various functions. One example is the markings on road surfaces such as pedestrian crossings, central lines, and lane boundaries. By acquiring these signs as images in advance, it is possible to provide the user with an image close to the actual road surface, and to realize guidance that is easy to understand intuitively.

As a technique for efficiently generating an image of a road surface including a sign, Patent Document 1, Patent Document 2, and the like can be cited.
Patent Document 1 discloses a technique for generating a still image including a road surface marking from images acquired by a digital camera or the like with respect to the front and rear or the side of a vehicle. In this technique, a road marking or the like is photographed with a digital camera or the like mounted on the vehicle while traveling on the target road with the vehicle. Then, each frame image constituting the moving image is converted into an orthogonal image as viewed from directly above, and arranged according to the shooting position. An orthographic image is a road image when the viewpoint is placed at an infinite point vertically above the road. By arranging a plurality of frame images, it is possible to obtain a composite image of the road surface along the trajectory of one run (hereinafter also referred to as a path).

  Patent Document 2 discloses a technique for synthesizing a wide road image by synthesizing images obtained by two passes. In this technique, first, an affine transformation is applied to an image obtained in one pass so that what is originally drawn linearly, such as a road lane boundary line, is displayed as a straight line. Then, the image of one path is affine transformed so that the coordinates of the lane boundary line or the like that are photographed in common in the two paths coincide. Also disclosed is a technique of synthesizing three or more paths by synthesizing images while performing affine transformation for each path by a similar method.

JP 2007-249103 A Japanese Patent No. 3820428

Map data includes a road network in which roads are represented by nodes and links for route search, and data in which roads are represented by polygons in order to display a map. In the road network, since the road is represented by one or two links, the coordinates attached to the link do not know exactly which part of the road is represented. In the drawing data, the position coordinates of the outer periphery of the polygon representing the road are known, but the position coordinates of the points inside the road are not known.
For example, if there is map data in which the position coordinates of each point inside the road are obtained in detail, the lane change is determined by determining which lane on the road the vehicle is driving according to the current position of the vehicle It is possible to realize highly functional guidance such as providing guidance on the vehicle and warning that a pedestrian crossing is approaching the vehicle.

However, the prior art cannot fully utilize the road marking included in the image obtained by the synthesis.
For example, Patent Document 2 discloses a technique for synthesizing images obtained in two passes, but it is not considered that image quality of road markings in an image corresponding to each pass is superior or inferior. . That is, even when the image quality of the road marking in one image is inferior to that of the other road marking, the low-quality road marking may be used in the composite image.

In order to avoid such adverse effects, it is conceivable to preferentially use the side with the high image quality of the road marking when combining. However, when the images obtained in the two passes are compared, the image quality of one pass is good in a certain part, but the image quality of the other pass is good in the other part. Therefore, even if either one is used preferentially, there is still room for improvement in the image quality of the road markings included in the composite image.
In addition to these problems, there has been a problem that the conventional map data has insufficient accuracy to realize such high-precision and high-performance guidance. Even if the current position of the vehicle is grasped with high accuracy, it cannot be said that detailed map data has been prepared to make use of the position information.

The road marking is suitable as an object for enriching the position coordinates on the road. For example, if the position coordinates of the pedestrian crossing and the lane boundary line are obtained, it is possible to contribute to the realization of the high-performance guidance described above.
However, all the prior arts have a main purpose of obtaining a composite image of a road surface, and have not been intended to obtain a position coordinate of a road surface sign.
For example, the technique of Patent Document 2 merely represents an image with the traveling direction of the vehicle as the X axis and the movement distance as the X coordinate regardless of whether the road is a straight line or a curve, and a direction orthogonal to the X axis. Only the image is affine transformed. Since there is no guarantee that the X coordinates determined in this manner are sufficiently coincident with each other for images obtained by a plurality of passes, the technique of Patent Document 2 cannot accurately obtain the position coordinates of road markings. .
The affine transformation can also be said to be a transformation having the effect of distorting the rectangular area of the original image into a parallelogram. Therefore, in the technique of Patent Document 2, the image is deteriorated by synthesizing the image by the affine transformation, and the road surface. There is also a problem of further reducing the position coordinates of the markings.
On the other hand, the technique described in Patent Document 1 only discloses processing for an image obtained in one pass and cannot sufficiently cover the entire road.

  An object of the present invention is to solve at least a part of these problems and to generate a composite image that fully utilizes a road marking included in a road surface. It is another object of the present invention to make it possible to generate a map that includes road markings with high positional accuracy.

The present invention can be configured as a road marking map generation method for generating a road marking map including a marking on a road surface by a computer.
In the present invention, the computer firstly includes image data composed of a plurality of frame images obtained by photographing a road surface on which a sign is given while moving on a road to be imaged a plurality of times, and each frame image in the image data . acquires the position coordinate data representing the photographing position. As a result, the position coordinate data of a plurality of paths, which are movement trajectories when the photographed road is moved, and image data of continuous images along the plurality of paths are acquired .
The above-mentioned image data can be photographed by a photographing device mounted on a moving body such as a vehicle, for example. For example, a digital video camera or the like can be used as the photographing apparatus. Moreover, it is preferable that the photographing apparatus is equipped with a position measuring device that acquires position coordinate data at the time of photographing. As the position measuring device, for example, a GPS (Global Positioning System) or an inertial navigation device such as a gyro can be used alone or in combination. For convenience of processing, it is preferable to prepare a recording apparatus that acquires a captured image and position coordinate data and records them in synchronization.

Here, first, the plurality of paths can be paths having different travel positions such as different lanes. However, the lanes do not necessarily have to be different, and the image may be taken by moving a plurality of times on any lane. When the lanes are different, the lane and the path have a one-to-one correspondence. In the latter case, the path and the lane correspond to each other in a plurality of one-to-one. The present invention is applicable to any case.
In this way, when the positions of the paths are different, as will be described later, by combining images obtained from these paths, it is possible to obtain a composite image that is wider than the range obtained in one pass.
Secondly, the plurality of paths may be paths acquired at different times. For example, once a road image is taken and map data is prepared, after a lapse of time, the road is taken again by taking the same path. In this case, as will be described later, it is possible to generate a map reflecting changes in road markings by combining images obtained from these paths.

When the image data is input, the computer converts each frame image constituting the image data to obtain an orthographic image in a state where the road surface is viewed from directly above. The orthographic image may be generated using a part of each frame image.
Then, by arranging the orthographic image obtained in this way on the path based on the position coordinate data, a connected image representing the road surface of each path is generated and displayed. At this time, a part of the orthographic image may overlap. For example, the orthographic image is preferably arranged in a state where its center line is along the traveling direction of the path. In this way, as many connected images as the number of paths are obtained.

Next, the computer based on the position coordinate data, superimposed the combined images on the path corresponding to the plurality of lanes constituting one of the road, and generates a composite image of the road road. The synthesis can be performed by various methods. For example, a method of arranging each connected image without considering the error of the position coordinate data may be adopted. Further, in the case where a relative error is included in the position coordinate data between a plurality of paths, a composite image may be generated after correcting the error. As for the correction method, affine transformation may be used as described in Patent Document 2, or a method exemplified later may be employed.

  When the composite image is generated in this way, the computer generates a transparent polygon at a position and shape determined according to the connected image. The transparent polygon is a polygon for transmitting the upper connected image and displaying the lower connected image in an area where the connected images overlap each other. In the region where the transparent polygon is generated, the upper connected image is transmitted and the lower connected image is displayed. “Determined according to the connected image” means that the transparent polygon is automatically generated according to a predetermined rule. Further, the transparent polygon is determined according to the connected image, and the transparent polygon is not generated at a preset fixed position.

According to the present invention, by generating a transparent polygon, it is possible to generate a composite image by effectively utilizing road markings included in a connected image used for composition. That is, a composite image is not generated by giving priority only to any one of a plurality of connected images in a fixed manner. It is possible to switch the connected image to be prioritized for each region, such as giving priority to the connected image arranged at the lower side in the place where the transparent polygon is set, and giving priority to the upper connected image in other regions. As a result, road markings included in each connected image can be used effectively according to the rules for setting the transparent polygon.
In addition, since these transparent polygons are automatically set based on predetermined rules, the burden on the operator is reduced, and a stable composite image can be obtained without variation in quality due to operator skill. There is also.

The transparent polygon can be generated in various modes. For example, it is preferable to generate the transparent polygon within a range where the lower connected image exists. When the connected images partially overlap, an area where only the lower or upper connected image exists and an area where both overlap are generated. In such a case, if a transparent polygon is set in an area where only the upper connected image exists, the upper connected image is transmitted, resulting in a state in which the image is completely missing. Such a phenomenon can be avoided if the transparent polygon is set within the range where the lower connected image exists.
Instead of such a mode, for example, a transparent polygon generated in a region where only the upper or lower connected image exists may be treated as invalid, and a composite image may be displayed without making the image transparent.

The position and shape of the transparent polygon can be generated based on, for example, a sign included in the connected image. The sign in each connected image is extracted, and a transparent polygon is generated based on the evaluation of the positional relationship and the superiority or inferiority of the image quality. By doing so, it is possible to generate a transparent polygon that can effectively use the road marking in each connected image.
Various evaluation methods can be adopted depending on the purpose of generating the composite image.
For example, when emphasizing the reproduction of the sign, the transparent polygon is displayed in a portion where the sign included in the lower connected image is judged to have a higher evaluation value indicating the accuracy than the sign included in the upper connected image. The method of setting can be taken. By doing so, it is possible to generate a composite image using the sign obtained accurately.

The accuracy index can be obtained in consideration of position accuracy, shape, and the like.
As an aspect that considers the position accuracy, for example, evaluation data representing the accuracy of the position coordinate data at the time of shooting each pass may be acquired, and the superiority or inferiority of the position accuracy may be determined for each pass. In this aspect, when it is evaluated that the lower path has higher positional accuracy, the transparent polygon is applied to a part where the signs overlap or a part where only the lower sign exists. Should be set. By doing so, it is possible to effectively utilize a sign with high position accuracy.
As an aspect in consideration of the shape, a method can be used in which the markings overlapping on the upper side and the lower side are compared, and the transparent polygon is set so that the side having the larger area of the marking is adopted. The large area is considered that the sign has been acquired without omission. In this case, the transparent polygon may be generated in a portion where the area of the lower sign is larger than that of the upper mark. For the comparison of the areas, the type of the sign may be considered. For example, the areas may be compared only when the same type of sign is overlapped.

  Moreover, you may make it consider both position accuracy and a shape. First, with reference to a sign of a path with high position accuracy, a position shift of a sign with a low position accuracy is obtained. This shift can be evaluated by a shift between corresponding vertices between the signs or a shift between the centers of gravity of the signs. On the other hand, the shape is evaluated by the difference in area. Then, the two are comprehensively compared by obtaining a sum obtained by multiplying each of the position accuracy evaluation and the shape evaluation by a predetermined weight value. Then, a transparent polygon is generated in a portion where the overall evaluation value for the lower sign is higher than that of the upper sign. However, the accuracy index is not limited to these examples, and various methods can be adopted.

As another aspect of generating the transparent polygon based on the sign, the transparent polygon may be generated in a portion where the sign is included only in the lower connected image. By doing so, it is possible to effectively utilize the sign included in the lower connected image with a relatively light load. When the transparent polygon is set based on the sign, the sign in each connected image may be extracted in advance by an operator using a pointing device or the like in the display screen of the connected image, for example.
Further, the computer may recognize a sign for each connected image by image processing, and set a transparent polygon based on the recognition result. If the sign is automatically recognized, there is an advantage that the load on the operator can be reduced. The extraction by the operator and the automatic recognition may be used in combination. For example, after automatic recognition is performed, extraction by an operator may be performed to correct misrecognition or replenish a recognition failure.

The automatic recognition of a sign can take the following method, for example.
First, the computer generates a connected image for recognizing a sign in the following procedure.
As a processing reference, a linear distance axis representing a distance from a reference point on the path is defined in a predetermined direction on the two-dimensional plane. The reference point may be a starting point of the path or an arbitrary point provided on the path. The distance axis can be defined in any direction on the two-dimensional plane. However, it is preferable to match with the x-axis or the y-axis from the viewpoint that each processing described later is simplified.
Based on the distance from the reference point along the path, the computer arranges the orthographic image on the distance axis, thereby generating a just-connected image that represents the road surface of each path. At this time, a part of the orthographic image may overlap. For example, the orthographic image is preferably arranged with its center line along the distance axis. By doing so, a connected image representing a straight road along the distance axis can be obtained.

The computer recognizes the type and position of the sign drawn on the road surface based on the linearly connected image. The path is a travel locus when photographing the road surface, and since the position coordinate data is acquired while traveling, the position accuracy is sufficiently secured, and the position accuracy of the extracted sign can be secured. it can.
Further, by arranging the orthographic image on the distance axis, it is possible to obtain a connected image extending linearly even when the road is curved. When the connected image is generated in this way, the sign on the road exists in a certain positional relationship with respect to the distance axis. For example, pedestrian crossings and stop lines are drawn so as to be orthogonal to the distance axis, and lane boundary lines are drawn parallel to the distance axis. In this way, how to draw a sign to be extracted in the connected image is constant for each sign, so it is possible to clearly set the conditions for extracting the sign and extract the sign stably. Is possible.
In the present invention, since the orthographic image is arranged to generate the connected image, the linearly extended connected image can be relatively easily obtained by arranging the orthographic image on the distance axis.

As described above, the present invention is not limited to the mode in which the transparent polygon is set based on the sign, but can also have a mode in which the transparent polygon is generated without depending on the sign.
For example, the transparent polygon may be generated so as to cover a predetermined range at the side edge of the upper connected image. In image conversion for generating an orthographic image constituting a connected image, it is more susceptible to distortion as it goes to the side edge of the image. In addition, since the orthographic image is usually trapezoidal, a sawtooth-like jagged shape may occur at both ends of the connected image generated by arranging the orthographic image. As a result, the side edge of the connected image is often in a state where the position accuracy of the sign is lower than that of the lower connected image and the sign is also divided by the jagged shape. Therefore, if a transparent polygon is generated in a predetermined range at the side end, such a region can be removed relatively easily, and the indication of the lower connected image can be effectively used.

In the above-described aspect, the transparent polygon may be generated at both ends of the upper connected image, or may be generated only in one of them.
Further, the predetermined range of the side end can be arbitrarily generated. For example, it is possible to identify a region that is a saw-toothed jagged area and set a transparent polygon having a minimum width sufficient to remove this range. In this way, it is possible to avoid unnecessary deletion of the upper connected image.

When the road width is wide, it may not be possible to photograph the entire road with one pass. In such a case, an image of the entire road may be obtained by combining images taken with a plurality of paths having different travel positions. When taking an image with a plurality of passes, the positional accuracy is not necessarily the same for each pass. Therefore, even if the connected images of each path are arranged based on the respective position coordinate data, inconsistencies may occur in the positions of the markings. In such a case, the position may be corrected by the following method.
First, corresponding points corresponding to each path are identified in an area that is photographed in common with a connected image of two or more paths among a plurality of paths. For example, when a pedestrian crossing is photographed in common in two paths, one of the corners of the pedestrian crossing pattern can be set as a corresponding point in each connected image.
The computer compares the extracted sign types and positional relationships among multiple paths and identifies the corresponding correspondence of the sign so that any point on the sign is automatically identified as the corresponding point. May be. Alternatively, the connected image and the extracted sign may be displayed on a computer display, and the operator may view the display and indicate the corresponding point.
The deviation between the corresponding points specified in this way represents an error in position coordinates.

The computer sets one of a plurality of paths as a reference path. This setting may be performed after the corresponding point is identified or may be performed before that. Then, based on the movement vector set so that the positions of the corresponding corresponding points coincide with each other, the combined image of the road surface extending over a plurality of paths can be obtained by correcting the connection image of the path other than the reference path. Generate.
This correction is performed by translating for each region constituting the connected image based on the movement vector. This area may be obtained by dividing the generated connected image once into a size corresponding to the original orthographic image or another arbitrarily set size. In addition, when generating a connected image, if the orthographic images are merely arranged without being synthesized, the position may be corrected for each orthographic image.

In this way, since a connected image representing the road surface of each path can be generated based on the position coordinate data at the time of shooting, the connected image can be obtained in a state where positional accuracy is ensured.
Then, one of the multiple paths is set as the reference path, and this path is fixed, and the position of the other paths is corrected. It is possible to eliminate the positional accuracy error.
Further, since the synthesis of each path is performed by translating the connected image for each area, the position can be corrected without adding distortion to the orthographic image of each area. Therefore, at the time of this correction, the road surface marking can maintain the relative positional accuracy in the orthographic image of each region.
By the above operation, according to the road marking map generation method of the present invention, a composite image including road surface marking can be obtained in a state where position accuracy is ensured.

In the road marking map generation method of the present invention, the reference path can be set by various methods such as designation by an operator.
The computer may also input evaluation data on the accuracy of the position coordinate data for each path and set a reference path based on the evaluation data. For example, among the plurality of paths, there is a method of setting a path that is evaluated as having the highest position accuracy based on evaluation data as a reference path. If a composite image is generated so as to match another path with the reference path set in this way, the highest position accuracy can be ensured.
The evaluation data may be data that directly represents the position accuracy quantitatively or may be data that can be used to calculate the position accuracy.

In the present invention, a plurality of transparent polygons may be generated.
In such a case, among the generated transparent polygons, those overlapping each other may be combined. By doing so, the number of transparent polygons can be reduced, and the data structure can be simplified and the data capacity can be reduced.
In the present invention, when overlapping images for a plurality of paths are overlaid, the operator may determine which one is arranged on top or may automatically set it.

As an automatic setting method, for example, at least one of the amount of a sign in a connected image and the amount of a foreign substance other than the sign is detected by image processing using a connected image or a frame image, and based on this detection result A method for determining which path is to be arranged on the upper side can be adopted. Since the upper / lower relationship of superposition is determined, this processing needs to be performed prior to the generation of the composite image of the road surface. Note that “foreign matter” means an object that hinders shooting of road surfaces and signs, such as vehicles that pass forward or sideways during shooting, bicycles, motorcycles, vehicles parked on the road, etc. To do.
Only one or both of the amount of sign in the connected image and the amount of foreign matter other than the sign may be detected. Here, the “amount” may be evaluated by a number such as a sign or an area.

In the above-described method, the number of transparent polygons can be suppressed or the shape can be simplified by determining the vertical relationship in consideration of the amount of marking. Further, by determining the vertical relationship in consideration of the amount of foreign matter, it is possible to improve the appearance of the road image after setting the transparent polygon.
For example, consider a case where a transparent polygon is generated where a road marking on the lower connected image is hidden. In this case, the transparent polygon can be suppressed by arranging the connected image having a large amount of the sign on the upper side. Further, the appearance can be improved by arranging the connected image with a small amount of foreign matter on the upper side.
On the other hand, when generating a transparent polygon in a portion excluding the upper sign, placing a connected image with a small amount of the sign on the upper side reduces the sign that the transparent polygon should avoid. The shape can be simplified. Further, the appearance can be improved by arranging a connected image with as little foreign matter as possible on the lower side.
Furthermore, in the above-described method, since the vertical relationship of the path is automatically determined, the subjectivity of the operator does not enter, and the instability that the appearance of the road image differs depending on the operator can be suppressed.

When automatically determining the arrangement of connected images, both the amount of marking and the amount of foreign matter may be considered. In this case, it is preferable to determine the arrangement by evaluating the amount of foreign matter preferentially over the amount of marking. To prioritize the amount of foreign matter, first determine the vertical relationship of the path according to the amount of foreign matter, and consider the amount of indication when the vertical relationship is not determined, such as when the amount of foreign matter is the same for both upper and lower paths Means that.
As described above, the amount of foreign matter affects the appearance of the road image. Therefore, the appearance of the road image can be improved by preferentially considering the amount of foreign matter.

The sign and the amount of foreign matter in the connected image may be evaluated for the entire connected image of each path, or may be evaluated for the area where the connected images overlap.
Moreover, you may use both evaluation together. When both evaluations are used together. It is preferable to preferentially evaluate the amount in the overlapping area rather than the amount in the whole connected image. The priority means that the evaluation in the entire connected image is considered when the vertical relationship is not determined by the evaluation in the overlapping region. Since areas other than overlapping areas do not affect the appearance of the road image even if the top-and-bottom relationship is changed, the effect of improving the appearance is more appropriately obtained if the evaluation in the overlapping areas is emphasized. Because it is.

The sign in the connected image can be recognized by the method described above. The reflection of a foreign object image other than the sign (hereinafter also referred to as “dust image”) can be extracted by the following method, for example.
A first method for recognizing a dust image and reflecting it in recognition of a road marking is as follows.
First, the computer obtains an optical flow between successive frame images. The optical flow is a vector representing movement from a plurality of feature points in the previous frame image to corresponding feature points in the next frame image. A continuous frame image does not necessarily need to be a frame image captured continuously, and the captured frame image may be defined by a frame image group after thinning out at a predetermined distance interval or time interval. Good.
Based on the optical flow thus obtained, the CPU extracts an image other than the marking, that is, a dust image from the frame image. This is because when a foreign object different from the road marking is reflected, the optical flow also has a different size and direction from the optical flow of the road marking portion. Reflection extraction may be evaluated based on both the magnitude and direction of the optical flow, or may be evaluated based on either one.

  Next, the CPU generates an orthographic image reflecting the extraction result of the reflection. In addition to the orthographic image generated without considering the dust image, an orthographic image of the dust image may be generated. Further, instead of the orthographic image generated without considering the dust image, the orthographic image may be generated based on an image obtained by removing the dust image from the frame image. By any method, it is possible to specify an area where a dust image is present in a connected image. After that, the evaluation value may be obtained by measuring the number or area of dust images.

A second method for recognizing the dust image and reflecting it in the recognition of the road marking is as follows.
The CPU cuts out a portion where the orthogonal images arranged on the distance axis overlap. Then, the gradation difference of each point between the orthogonal images of the overlapping portion is obtained. The gradation difference is a difference between color components recorded at each point. In the case of a color image, a gradation difference may be obtained for each RGB component, but it is preferable to obtain a lightness (V) difference after decomposing into three elements of HSV. Since the road markings should overlap in the overlapping portion between the orthogonal images, the gradation difference with respect to the road marking is less than a predetermined value. On the other hand, foreign objects other than road markings usually have large gradation differences because the shape and size of the reflection on the orthographic image are different at different shooting times and shooting locations. Therefore, it is possible to extract a dust image by extracting a point where a gradation difference exceeding a predetermined threshold is obtained. When the dust image is extracted in this way, it is possible to specify an area where the dust image is present in the connected image.

  In the case of using the second method, it is preferable to obtain the gradation difference after applying a smoothing filter to the orthogonal image. This is because if the gradation difference is strictly determined for each pixel constituting the image, a large gradation difference may occur in the road marking portion due to a position error when generating an orthographic image. By applying the smoothing filter, the gradation change in the image can be dulled, so that the influence of the position error and the like can be alleviated and the dust image extraction accuracy can be improved.

The dust image extraction may be applied to the entire frame image, but may be applied to only a part of the frame image in order to improve the processing speed. For example, it is possible to target a portion that is an orthographic image generation target.
As another aspect, a predetermined area excluding a predetermined range including a path may be targeted. This is because photographing is performed while moving along the path, and it is unlikely that foreign objects other than road markings appear in the area around the path. This is because, in the travel lane in which the host vehicle travels, it is possible to travel at a greater distance so that the vehicle traveling ahead does not enter the image creation range. On the other hand, in lanes other than the traveling lane, there is currently no means for avoiding the appearance of foreign objects.
Furthermore, by combining the two, a predetermined region excluding a predetermined range including a path may be targeted in a portion that is an orthographic image generation target.

In the present invention, the example in which the transparent polygon is generated in the portion of the sign hidden by the upper connected image has been described above. However, the method for generating the transparent polygon is not limited to such a part.
For example, when extracting a portion where foreign objects other than the marking appear in the upper connected image, that is, a dust image, by the above-described method, the region is made transparent so as to cover an area corresponding to at least a part of the extracted foreign matter. Polygons may be generated. By doing so, the appearance of the road image can be improved.
When generating the transparent polygon that covers the dust image, it is preferable to generate the transparent polygon within the range in which the connected image exists on the lower side, as in the case of generating the transparent polygon in the marking portion. This is because if a transparent polygon is set in a portion where there is no connected image, a road image will appear as if there was a hole in that portion.

Further, when generating a transparent polygon so as to cover a predetermined range of the side edge of the upper connected image and hiding a saw-toothed jagged portion generated on both sides of the connected image, the transparent polygon covering at least a part of the foreign matter is You may integrate so that the clearance gap with the transparent polygon (henceforth a side edge transparent polygon) for concealing this jagged part may be filled up. By doing so, the number of transparent polygons can be suppressed and the shape can be simplified. Further, by integrating so as to fill the gap, if there is a dust image that could not be extracted in the gap, the dust image can also be hidden by the transparent polygon.
Integration can be done in various ways. For example, a perpendicular line for projecting the side-end transparent polygon may be drawn to combine the two. Further, only the transparent polygons of foreign matters within a predetermined distance from the side end transparent polygons may be combined.

Conversely, the transparent polygon can be generated so that only the road marking is left.
That is, after extracting the road marking included in the upper connected image, a transparent polygon that covers the side end side of the connected image with respect to the extracted road marking is generated. However, it is preferable that the transparent polygon is generated within a range that does not cover other road markings. Since there is no road marking in the region on the side end side from the extracted road marking, it is not always necessary to leave a connected image. In the above method, a transparent polygon is generated in such a part, and unnecessary parts of the upper connected image are deleted together, so that it is easy to delete the dust image without having to extract whether or not there is a dust image. can do.

In addition, in a portion where there is no road marking that can be projected on the path, a transparent polygon may be generated so as to cover the connected image in a range excluding a portion within a predetermined distance from the path in a direction intersecting the path Good. In this way, it is possible to easily delete the dust image even at a location where no road marking exists.
The present invention does not necessarily have all the above-described features, and some of them may be omitted as appropriate, or some features may be appropriately combined.
The present invention is not limited to the road marking map generation method described above, and may be configured as a road marking map generation device that generates a road marking map including a marking applied to a road surface by the road marking map generation method.

  Further, the above road marking map generation method may be configured as a computer program for causing a computer to realize the method, or may be configured as a computer-readable recording medium on which the computer program is recorded. Examples of the recording medium include a flexible disk, a CD-ROM, a magneto-optical disk, an IC card, a ROM cartridge, a punch card, a printed matter on which a code such as a barcode is printed, an internal storage device of a computer (a memory such as a RAM or a ROM). ) And external storage devices can be used.

  By generating the transparent polygon, it is possible to generate a composite image by effectively utilizing the road marking included in the connected image used for the synthesis. In addition, road markings included in each connected image can be effectively used according to the rules for setting the transparent polygon.

It is explanatory drawing which shows the structure of the road surface imaging | photography system as an Example. It is explanatory drawing which shows the structure of the road marking map production | generation apparatus as an Example. It is explanatory drawing which shows the intermediate data in the production | generation process of a road marking map. It is explanatory drawing which shows the production example of the road image in an Example. It is explanatory drawing which shows the outline | summary of the alignment process. It is explanatory drawing which shows the procedure of the alignment in case an intersection exists. It is a flowchart of a connection image generation process. It is a flowchart of an alignment process. It is a flowchart of a reference | standard path | pass setting process. It is a flowchart of a connection image movement process. It is explanatory drawing which shows the process example (1) of position alignment process. It is explanatory drawing which shows the process example (2) of an alignment process. It is explanatory drawing which shows the process result of the process (2) of position alignment process. It is explanatory drawing which shows the acquisition method of the absolute position coordinate of a road marking. It is explanatory drawing which shows the outline | summary of a transparent polygon setting process. It is a flowchart of a transparent polygon setting process. It is explanatory drawing which shows the example of a road image before setting a transparent polygon. It is explanatory drawing which shows the example of a road image after the setting of the transparent polygon. It is a flowchart of a paint recognition process. It is explanatory drawing which shows the content of the vertical arrangement | positioning process. It is explanatory drawing which shows the content of a relative coordinate conversion process. It is a flowchart of an automatic transparency polygon production | generation process. It is explanatory drawing which shows the method of overlap determination. It is explanatory drawing which shows the setting method of a root area | region. It is a flowchart of the transparent polygon setting process for paint. It is explanatory drawing which shows the example (1) of setting of the transparent polygon for paint. It is explanatory drawing which shows the example (2) of setting of the transparent polygon for paint. It is explanatory drawing which shows the example (3) of setting of the transparent polygon for paint. It is a flowchart of a jagged cut process. It is explanatory drawing which shows the content of the transparent polygon coupling | bonding process. It is explanatory drawing which shows the example of a setting of a transparent polygon. It is a flowchart of a priority path determination process. It is explanatory drawing which shows the example of determination of a priority path | pass. It is explanatory drawing which illustrates the reflection of a foreign material. It is a flowchart which shows the reflection method of the recognition result of a dust image. It is a flowchart of a dust image recognition process (1). It is an example of an optical flow. It is an example of the optical flow when there is a traveling vehicle. It is an example of the optical flow when there is a stopped vehicle. It is a flowchart of a dust image recognition process (2). It is explanatory drawing which shows the example of recognition of a dust image. It is explanatory drawing which shows the example of a transparent polygon setting by the method of an Example. It is a flowchart (1) of dust image deletion polygon generation processing. It is a flowchart (2) of dust image deletion polygon generation processing. It is explanatory drawing which shows the process example of a dust image deletion polygon production | generation process (2). It is a flowchart of a dust image erasure polygon generation process (2). It is explanatory drawing which shows the example of application of automatic transparency polygon production | generation process (2).

Embodiments of the present invention will be described in the following order.
A. System configuration:
A1. Road surface photography system:
A2. Road marking map generator:
B. Outline of processing:
B1. Intermediate data structure:
B2. Processing example:
B3. Outline of alignment processing:
C. Road marking map generation method:
C1. Connected image generation processing:
C2. Alignment processing:
C3. Standard path setting process:
C4. Connected image movement processing:
C5. Transparent polygon setting process:
C6. Paint recognition process:
C7. Automatic transparency polygon generation processing:
D. effect:
E. Variations:
E1. Determining the preferred path:
E2. Automatic transparency polygon generation processing (1):
E3. Automatic transparency polygon generation processing (2):

A. System configuration:
In this embodiment, a method for generating a map including road markings (hereinafter referred to as a “road marking map”) using road surface images taken by a video camera mounted on a vehicle will be described.
The system of the present embodiment includes a road surface photographing system and a road marking map generating device. The road surface photographing system is a system for photographing an image of a road surface with a video camera while traveling on the road. In the present embodiment, the target road is traveled a plurality of times with different travel trajectories, and images are respectively captured.
The road marking map generation device is a device that generates a road marking map based on an image of a road surface photographed by a road surface photographing system. First, a connected image of a part of the lane on the road surface is generated by arranging the photographed image after converting it into an orthographic image on each of the above-described traveling trajectories. And the image of the whole road is produced | generated by arrange | positioning the image of a some driving | running | working locus so that a position coordinate may match. Further, a road surface sign is extracted from the connection image thus generated.
Hereinafter, the system configuration of the road surface photographing system and the road marking map generating device will be described.

A1. Road surface photography system:
FIG. 1 is an explanatory diagram showing a configuration of a road surface photographing system as an embodiment.
The road surface photographing system 100 is a system mounted on a vehicle. The system configuration will be described based on the block diagram below.
The video camera 120 captures an image of a running road surface.
The position measurement unit 110 is a device that measures position coordinates during shooting. The position measurement unit 110 includes a GPS (Global Positioning System) 114 and an IMU (Inertial).
A measurement unit (116) 116, a distance measuring instrument (DMI) 118, and a controller 112 are provided. The GPS 114 is a global positioning system. The IMU 116 is an inertial measurement device including a triaxial gyro and an acceleration sensor inside. The DMI 118 is a device that measures the distance traveled by detecting the rotation of a wheel.

The controller 112 receives signals from the GPS 114, the IMU 116, and the DMI 118, and sequentially outputs position coordinates at the time of shooting. The position coordinates can take an arbitrary coordinate system, but in this embodiment, latitude and longitude and altitude are used.
In addition, after obtaining these signals, a self-estimated position accuracy σ, which is an evaluation value of the position coordinate measurement accuracy, is also output. In general, it is known that the detection accuracy of the GPS 114 varies depending on the arrangement of artificial satellites used for detecting position coordinates, the reception status of radio waves, the presence of multipath by receiving radio waves reflected on buildings, and the like. Yes. In differential positioning, the detection accuracy is also affected by the operating status of the reference station.
The self-estimated position accuracy σ can be arbitrarily defined. For example, the self-estimated position accuracy σ may be calculated using a precision reduction rate (DOP (Dilution of Precision)) determined by the arrangement of the satellites of the GPS 114. The self-estimated position accuracy σ may be analyzed when the acquired data is processed by a road marking map generation device to be described later.

  The recording device 130 records the output signals of the video camera 120 and the position measuring unit 110 in synchronization. In this embodiment, the recording device 130 is configured by a device in which a recording hard disk 140 is added to a general-purpose personal computer. In the hard disk 140, as shown in the figure, image data 142, synchronization data 144, and measurement data 146 are recorded. The image data 142 is a moving image file of an image taken with a video camera. The measurement data 146 is position coordinates obtained by the position measurement unit 110. The synchronization data 144 is data that associates the acquisition times of the image data 142 and the measurement data 146 with each other. By referring to the synchronization data 144 and the measurement data 146, the position coordinates of the shooting point can be obtained for each frame of the image data 142.

The data structure for recording at the time of shooting is not limited to the structure described above. For example, the measurement data 146 may be data that sequentially stores the position coordinates of each frame of the image data 142. By doing so, the synchronization data 144 can be omitted. In order to acquire such data, for example, a method in which the recording device 130 outputs a synchronization signal to the position measuring unit 110 for each frame of the video camera 120 and acquires the position coordinates at that time can be employed.
The state mounted on the vehicle is schematically shown in the upper part of the figure.
The video camera 120 is installed in front of the vehicle so that a front image can be taken. A wide-angle lens may be attached to widen the angle of view.
The antenna 114A of the GPS 114 is installed on the upper roof of the vehicle. In the present embodiment, two main and sub antennas 114A are installed in front of and behind the vehicle so that radio waves from GPS artificial satellites can be reliably received and sufficient positional accuracy can be ensured. Only one of them may be used.
The IMU 116, DMI 118, and controller 112 were installed at the rear of the vehicle. The DMI 118 is mounted so that the rotation of the rear wheel can be detected.
Since the recording device 130 and the hard disk 140 can be installed at any location in the vehicle interior, illustration is omitted.

A2. Road marking map generator:
FIG. 2 is an explanatory diagram showing a configuration of a road marking map generating apparatus as an embodiment. It is an apparatus for generating a road marking map based on an image of a road surface photographed by a road surface photographing system. In this embodiment, a method of advancing the process interactively or semi-automatically while receiving an instruction by a command from an operator as appropriate, instead of generating a road marking map completely automatically.
In the figure, functional blocks of the road marking map generating apparatus 200 are shown. In this embodiment, the road marking map generating apparatus 200 is constructed in software by installing a computer program for realizing each function shown in a general-purpose personal computer. Some of these functional blocks may be provided by an OS (Operating System). Each of these functional blocks can also be configured in hardware. In addition, here, for convenience of explanation, it is described as a stand-alone operating device, but each functional block may be distributed and prepared in a plurality of computers connected via a network.

  The main control unit 201 performs integrated control of each functional block. The data input unit 204 inputs image data 142, synchronization data 144, and measurement data 146 from the hard disk 140 that records various data acquired by the road surface photographing system 100. In this embodiment, a method of transferring these data by reconnecting the hard disk 140 to the road marking map generating device 200 from the road surface photographing system 100 is adopted. However, a method of transmitting data via a network, a DVD, A method of transferring data using a recording medium such as the above may be adopted. Since the measurement data includes the position coordinates obtained by the position measurement unit 110, the data input unit 204 corresponds to the input unit in the present invention.

The command input unit 202 inputs a command from an operator through operations such as a keyboard and a mouse provided in the computer.
The display control unit 203 displays the processing result of the road marking map generation device 200 on a computer display, or displays a screen for an operator to instruct various commands. The functions of the command input unit 202 and the display control unit 203 may be provided by a computer OS.

The trajectory data calculation unit 205 generates data representing a travel trajectory (hereinafter also referred to as “pass”) when the image data 142 is captured based on the measurement data 146. In this embodiment, the trajectory data calculation unit 205 corrects the measurement data 146 in which the position coordinates obtained by the road surface photographing system 100 are recorded based on detection information provided from a reference station whose position coordinates are known. Thus, trajectory data is generated. Since the technique for correcting the position coordinates using the information of the reference station is well known, the description thereof is omitted. This processing can improve the accuracy of position coordinates.
However, it is not essential to use data from the reference station. The position coordinates obtained from the measurement data 146 may be used as they are. In such a case, the trajectory data calculation unit 205 can be omitted.

The image conversion unit 206 generates an orthographic image by converting each frame image of the image data 142 into an orthographic projection, that is, a state as viewed from directly above.
A one-pass composition unit 207 serving as a connected image generation unit determines an orthographic image of each frame image obtained by the image conversion unit 206, and a position at which a representative point in the orthographic image is determined based on position coordinates at the time of shooting. By arranging so as to come to the coordinates, an image of the road surface along the traveling locus (path) at the time of photographing is synthesized. The image synthesized in this way is called a connected image. The combined connection image is stored in the processing data storage unit 210.
In this embodiment, each road is photographed by traveling a plurality of times with different traveling trajectories. The 1-pass image composition unit 207 generates a composite image for each pass. As a result, a plurality of connected images are generated according to the number of paths.

Note that images of a plurality of passes may be taken at different times. For example, after taking an image of a road surface at a certain time and preparing a map that includes road markings, when the predetermined time has elapsed, the image is taken again on the same road, and the previous image and this time The change of the road marking may be reflected on the map by synthesizing the image.
When the object to be photographed is a road with multiple lanes, the image may be taken by moving once for each lane, or may be taken by moving several times on any lane. In the former case, the lane and the path correspond one-to-one. In the latter case, the path and the lane correspond to a plurality of one-to-one. In this embodiment, unless otherwise specified, a case where the vehicle travels once for each lane, that is, a case where the lane and the path correspond one-to-one will be described as an example.

The alignment processing unit 220 serving as a composite image generation unit converts a plurality of connected images generated by the one-pass image combining unit 207 into an alignment process, that is, corrects positional coordinate errors between the connected images, thereby generating a road surface image. By performing the process of arranging them so as to match, an orthogonal image of the entire road (hereinafter also referred to as “road image”) is generated. The alignment process is performed according to an instruction from the operator. The processing contents will be described later.
The road image obtained by the alignment is stored in the processing data storage unit 210.

  A transparent polygon setting unit 221 as a transparent polygon generation unit sets a transparent polygon on the obtained road image according to an operator's instruction. When performing the above-described alignment, a part of the orthographic image corresponding to the adjacent path may overlap. In the overlapped portion, the road marking may be clearly copied toward the orthographic image arranged on the lower side. In such a case, the transparent polygon is a polygon for designating an area to be subjected to processing for making a part of the upper orthographic image transparent so that the lower image is displayed. By setting the transparent polygon, it is possible to provide a map that can accurately grasp the road marking.

The automatic transparent polygon setting unit 224 automatically sets the transparent polygon described above. By doing so, there is an advantage that the load on the operator can be reduced and the transparent polygon can be set with stable quality without depending on the skill of the operator.
Either one of the transparent polygon setting unit 221 and the automatic transparent polygon setting unit 224 may be selectively used according to an operator instruction, or both may be used in combination. As an example of the combined use, for example, a mode in which the transparent polygon setting unit 221 corrects or adds the transparent polygon after the processing by the automatic transparent polygon setting unit 224 is performed.
The transparent polygon set by the transparent polygon setting unit 221 and the automatic transparent polygon setting unit 224 is stored in the predetermined data storage unit 210.

The paint recognition unit 223 recognizes a road surface sign (hereinafter also referred to as “paint”). In the present embodiment, the sign is recognized based on the connected image generated by the one-pass image combining unit 207. The paint recognition result is stored in the processing data storage unit 210.
The road marking map generating device can output a road marking map based on the road image generated as described above. For example, a road image may be output as a printable file. Moreover, you may output a road image as electronic data so that a road marking map may be produced | generated as an electronic map. In addition, prior to these outputs, a process for obtaining position coordinates and shape data of road markings based on road images may be performed.

B. Outline of processing:
B1. Intermediate data structure:
FIG. 3 is an explanatory diagram showing intermediate data in the process of generating a road marking map. These data are sequentially stored in the processing data storage unit 210 (see FIG. 2).
In this embodiment, data acquired by the video camera 120 and the position measuring unit 110 while traveling on a road is stored in the hard disk 140 by a personal computer as the recording device 130. The stored data includes image data 142, measurement data 146, and synchronization data 144 for synchronizing them.
In the present embodiment, it is possible to process a plurality of images acquired at different times. However, in the following, a case will be described as an example in which shooting is performed with different travel trajectories at the same time.

  The measurement data 146 is a record of position coordinate data at the time of shooting. In this embodiment, the trajectory data 210a is calculated by correcting the measurement data 146 with reference to the reference station data 150. This is a process performed by the trajectory data calculation unit 205 described above with reference to FIG. The reference station data 150 is data representing a detection result by GPS at a reference point whose position coordinates are known. For example, reference point data provided by the Geospatial Information Authority of Japan can be used. The trajectory data 210a obtained here represents the trajectory (hereinafter also referred to as “path”) when an image of the road surface is captured in each processing, in absolute coordinates including latitude, longitude, and altitude. Used as data.

On the other hand, a road surface texture 210c is generated from the image data 142, the synchronization data 144, and the trajectory data 210a. In addition, road surface trajectory data is generated from the synchronization data 144 and the trajectory data 210a.
In this embodiment, each road is traveled a plurality of times, and an image of the road surface is taken. Therefore, the road surface track data 210b and the route track data 210b are generated for a plurality of paths for each road.

A coupled image 210d is generated using the road surface texture 210c and the route trajectory data 210b. The connected image 210d is an image generated by the one-pass image composition unit 207 in FIG. That is, the connected image 210d is a road surface image of each path generated by arranging each road surface texture 210c based on the position coordinates represented by the route locus data 210b. The connected image 210d is also generated for a plurality of paths for each road.
The connected image 210d can also be generated as one image file obtained by combining the road surface texture 210c. In this embodiment, for convenience of subsequent processing, information (hereinafter also referred to as “registered data”) for generating the connected image 210d by arranging the road surface texture 210c is not generated as a composite image. The image is stored in association with each image of the road surface texture 210c. Such information can include position coordinates for placing the road surface texture 210c, posture (angle) at the time of placement, information for specifying the adjacent road surface texture 210c, vertical relationship with the adjacent road surface texture 210c, and the like.

Using the connected image 210d thus obtained, processing such as alignment and transparent polygon setting is performed. These processes are performed by the alignment processing unit 220, the transparent polygon setting unit 221 and the automatic transparent polygon setting unit 224 shown in FIG. By this process, the linked images 210d for a plurality of paths can be synthesized to obtain a road image 210e for each road.
The road image 210e may be generated as a composite image, or information for generating the road image 210e by arranging the road surface texture 210c may be stored in association with each image of the road surface texture 210c. Good. In this embodiment, the latter method is adopted. Information such as the position coordinates at which each road surface texture 210c is arranged and the posture (angle) at the time of arrangement are stored as road image registration data 210f. In addition, since a process for correcting the position error is performed on the route trajectory data 210b in the alignment process, information indicating the correction process for the original data is stored as the trajectory registration data 210g.

  In addition, the data of the connected image 210d (including the road surface texture 210c and the route trajectory data 210b) are also stored. It is preferable to store the image data 142 and the trajectory data 210a as the original data. If the connected image 210d is stored in the form of a composite image, the road image 210e is combined with the connected image 210d. Therefore, the image quality is deteriorated as compared with the original data due to repeated combining. There is a risk. On the other hand, as in the present embodiment, by leaving data close to the original data including the road surface texture 210c, it is possible to generate the road image 210e using these data. Therefore, it is possible to suppress the image processing being superimposed on the image data, such as repeated synthesis, and to improve the image quality of the road image 210e.

B2. Processing example:
Next, in order to facilitate understanding of the outline of processing in the present embodiment, a processing example will be shown.
FIG. 4 is an explanatory diagram illustrating a generation example of a road image in the embodiment. FIG. 4A shows an example of generation of a connected image obtained along one path, and FIG. 4B shows an example of a road image obtained by arranging connected images of a plurality of paths. Is shown.
Straight lines L <b> 41 to L <b> 44 in FIG. 4A represent travel trajectories (paths) when a road image is captured while traveling on the road surface photographing system 100. The PIC 41 in FIG. 4A is a linked image generated based on image data obtained by traveling on the path L43. In this embodiment, since the image is taken using the wide-angle lens, it is possible to obtain a connected image that only covers a plurality of lanes even in one pass. The fact that both ends of the connected image are serrated in a saw-tooth shape is due to the influence of shape distortion that occurs when each frame of image data is orthographically projected. The connected image PIC41 is generated by arranging orthogonal images (road surface texture) having the number of frames corresponding to the number of jagged peaks.
Such a connected image is obtained for each of the paths L41 to L44 in the drawing.

  FIG. 4B shows a road image PIC42 obtained by synthesizing connected images for the paths L41 to L44. It can be seen that a road image is generated that is wider than that in FIG. When combining a multi-pass connected image, if there is an error in the position coordinates of each pass, a shift occurs between the connected images. If these deviations exist, signs such as a pedestrian crossing and a lane boundary line in FIG. 4B are also displayed in a state of being shifted in the middle. In this embodiment, the composition is performed while correcting the positional coordinate error between the connected images of each path. This process is called alignment. By performing alignment in this way and synthesizing the connected images, as shown in FIG. 4B, a road image in which signs such as a pedestrian crossing and a lane boundary line are matched can be obtained.

B3. Outline of alignment processing:
FIG. 5 is an explanatory diagram showing an outline of the alignment process. In the present embodiment, alignment is performed by moving the connected images in parallel so that the positions of corresponding points designated by the operator are aligned based on the signs photographed in common with the plurality of connected images.
FIG. 5A shows a processing method when only one corresponding point is designated. In the figure, two linked images PIC51 and PIC52 are drawn. Each of these includes a diamond sign, that is, a warning sign for a pedestrian crossing. However, in the state on the left side of FIG. 5 (a), the linked images PIC51 and PIC52 have a relative position error, so that the positions of the markings are shifted.

The operator designates corresponding points using a pointing device such as a mouse while looking at the display screen. In the example of the figure, a state is shown in which P51 and P52 corresponding to the apex of the notice of pedestrian crossing are designated. These corresponding points P51 and P52 are originally points that should overlap at the same position if the connected images PIC51 and PIC52 have no position error. Therefore, in this embodiment, the connected images PIC51 and PIC52 are translated as indicated by arrows in the drawing so that the corresponding points P51 and P52 coincide.
At this time, a method was adopted in which one of the connected images PIC51 and PIC52 is translated and the other is translated. In the example of the figure, an example in which the connected image PIC52 is moved with reference to the connected image PIC51 is shown. By moving in this way, it is possible to obtain the road image PIC53 in a state in which the shift of the warning sign is eliminated.

FIG. 5B shows a processing method when a plurality of corresponding points are designated. In the figure, two linked images PIC54 and PIC55 are drawn. Each of these includes a warning sign for a pedestrian crossing. However, in the state on the left side of FIG. 5B, since the linked images PIC54 and PIC55 have a relative position error, the positions of the markings are shifted.
In this state, it is assumed that the operator has specified two sets of corresponding points. A pair of corresponding points P54 and P53 and a pair of corresponding points P56 and P55. In the linked image PIC54, the entire warning sign M52 included in the linked image PIC55 is drawn, and a part of the warning sign M51 included in the linked image PIC54 has disappeared. Even in such a state, since it is clear that the corresponding points P55 and P56 correspond, it is possible to designate them as corresponding points.
When a plurality of sets of corresponding points are designated in this way, the linked image PIC 55 is moved so that the corresponding points match with the linked image PIC 54 as a reference. However, the first movement amount for matching the corresponding point P53 with P54 and the second movement amount for matching the corresponding point P55 with P56 are not necessarily the same. Therefore, in the region between the corresponding points P53 and P55, the first movement amount and the second movement amount are linearly interpolated to set the movement amount of each point. By doing so, it is possible to obtain the road image PIC 56 in a state in which the shift of the notice sign is eliminated.

FIG. 5B also shows an example of setting a transparent polygon.
In this example, half of the notice sign M51 in the linked image PIC54 is missing. If alignment is performed in this state, in this example, the connected image PIC54 is displayed so as to be superimposed on the upper side of the PIC55, so the notice sign M52 of the connected image PIC55 is covered by the connected image PIC54. As a result, the sign M52 drawn in the complete state in the connected image PIC55 cannot be utilized in the road image PIC56.
Therefore, in such a case, the transparent polygon TP50 is set so as to surround the notice sign M52 according to an instruction from the operator. At the place where the transparent polygon TP50 is set, the upper connected image is made transparent and displayed as if it was cut out. As a result, in the part of the transparent polygon TP50, the notice sign M52 drawn on the connection image PIC55 arranged below the connection image PIC54 is displayed.
In this embodiment, by making the transparent polygons settable in this way, the signs drawn in the respective connected images can be used effectively in the road image.

  FIG. 6 is an explanatory diagram showing a procedure for alignment when an intersection exists. In order to avoid complication of the figure, only the positional relationship of the paths of the connected images is shown here. In the figure, roads around two intersections are drawn. It is assumed that connected images are obtained along the paths BP61 and BP62 on the vertical road, respectively. For the horizontal road, it is assumed that a connected image is obtained along the paths BP63b, BP64b, and NP61b indicated by broken lines.

In this embodiment, when positioning between a plurality of paths, any one path is set as a reference path, and the other paths are translated to match the reference path. A path other than the reference path is hereinafter referred to as a standard path. Although the reference path and the standard path can be set by an arbitrary method, in this embodiment, as described later, a path with high position accuracy is set as the reference path.
In the example of FIG. 6, there is only a single path for each vertical road, so the paths BP61 and BP62 are the reference paths.
For the side road, in the section D61, the side having higher position accuracy among the paths BP63b and NP61b is set as the reference path, and for the section D62, the side in BP64b and NP61b having the higher position accuracy is set as the reference path. Here, it is assumed that the paths BP63b and BP64b are set as reference paths, respectively. Further, the superiority or inferiority is determined by comparing the positional accuracy between the paths BP63b and BP64b. This is because the paths BP63b and BP64b are reference paths of the sections D61 and D62, respectively, but are continuous paths arranged on one road, and therefore it is necessary to perform alignment between these paths. In the example of FIG. 6, it is assumed that the path accuracy of the path BP63b is higher than that of the path BP64b.
As a result, for the horizontal path, the alignment priority is determined in the order of the reference path BP63b> reference path BP64b> standard path NP61b.

Next, each path is aligned according to the above-described priority. Assume that the vertical paths BP61 and BP62 have already been aligned.
First, alignment of the reference path BP63b is performed. It is assumed that the corresponding point P63b on the reference path BP63b is designated by the operator's instruction, and the point P63a is designated as its original position. As a result, the reference path BP63b is moved so that the corresponding point P63b coincides with the point P63a, and the reference path BP63a indicated by the solid line is obtained.
Although not shown, the connected image corresponding to the reference path BP63b also moves in accordance with the reference path BP63a. In the present embodiment, the connected images are displayed by arranging road surface textures along the reference path BP63b, and these road surface textures are not synthesized. Therefore, when the movement to the reference path BP63a is performed, a connected image of the reference path BP63a can be obtained by translating the position of each road surface texture along the reference path BP63a.

Next, the reference path BP64b is aligned. It is assumed that corresponding points P65b and P64b on the reference path BP64b are designated by the operator's instruction, and points P65a and P64a are designated as their original positions. This corresponding point is designated based on the connected image of the reference path BP63a. That is, the alignment of the reference path BP64b is affected according to the result of the process of aligning the reference path BP63b with the reference path BP63a.
When the corresponding point is designated, the reference path BP64b is moved so that the corresponding points P65b and P64b coincide with the points P65a and P64a, and the reference path BP64a indicated by the solid line is obtained. In accordance with this, the road surface texture constituting the connected image of the reference path BP64b is also translated on the reference path BP64a.

Finally, the standard path NP61b is aligned. It is assumed that corresponding points P68b, P67b, and P66b on the standard path NP61b are designated by an operator's instruction, and points P68a, P67a, and P66a are designated as their original positions. This corresponding point is designated based on the connected image of the reference paths BP63a and BP64a. That is, the alignment of the standard path NP61b is affected by the result of the process of aligning the reference path BP63b with the reference path BP63a and the process of aligning the reference path BP64b with the reference path BP64a.
When the corresponding point is designated, the standard path NP61b is moved so that the corresponding points P68b and P67b coincide with the points P68a and P67a, and the corresponding points P67b and P66b coincide with the points P67a and P66a. Moved. Since these three points are not on a straight line, as a result, the standard path NP61b is moved to the broken line-shaped standard path N61a. In accordance with this, the road surface texture constituting the connected image of the standard path NP61b is also translated on the standard path NP61a.

In the present embodiment, when there are a plurality of paths as shown in FIG. 6, alignment is performed preferentially from a path with high position accuracy by the procedure described above. By doing so, it is possible to perform alignment while ensuring sufficient overall position accuracy.
For example, in the processing of FIG. 6, it is assumed that alignment is performed in the order of low position accuracy, that is, the standard path NP61b, the reference path BP64b, and the reference path BP63b. In this case, the alignment of the reference path BP64b is affected by the alignment of the standard path NP61b, and the position accuracy decreases. The alignment of the reference path BP63b is affected by the alignment of the standard path NP61b and the reference path BP64b, and the position accuracy is lowered. Therefore, if the alignment is performed in the order from the lowest position accuracy, the overall position accuracy decreases due to the interaction between the paths.
In the present embodiment, on the contrary, the alignment is performed in the descending order of positional accuracy. Therefore, it is possible to perform the entire alignment without deteriorating the position accuracy of the path having the highest position accuracy.

C. Road marking map generation method:
Hereinafter, the road marking map generation method described with reference to FIGS. 1 to 6 will be described in detail with reference to an example in which an operator gives an instruction as necessary.
First, a connected image generation process, that is, a process of obtaining a connected image 210d of each path based on the road surface texture 210c and the road surface trajectory data 210b in FIG. 3 will be described.
Next, registration processing, that is, processing for positioning the connected image 210d with respect to a plurality of passes, reference path setting processing, and connected image movement processing performed during the positioning processing will be described.
Also, the transparent polygon setting process will be described. Regarding the setting of the transparent polygon, first, in order to facilitate understanding of the processing outline, a case where the operator designates will be described as an example.

In the series of processes described above, if the road marking map generating apparatus displays the position of the marking in the connected image, the operator can easily specify the corresponding point and the position of the transparent polygon. It is also possible to automatically identify corresponding points and set transparent polygons. A paint recognition process will be described as a process for enabling such a process.
Then, based on the result of the paint recognition process, an automatic transparent polygon setting process for automatically setting a transparent polygon will be described.

C1. Connected image generation processing:
FIG. 7 is a flowchart of linked image generation processing. In terms of hardware, this is a process executed by the CPU of the road marking map generating apparatus 200. This corresponds to the processing of the image conversion unit 206 and the one-pass synthesis unit 207 shown in FIG.
When the process is started, the CPU first reads frame data (step S10). The frame data is an image of each frame constituting the image data 142 photographed by the video camera 120 of the road surface photographing system 100 (FIG. 1).

An example of frame data is shown in the figure. Since the video camera 120 is installed toward the front of the road surface photographing system 100, the frame data includes a road ahead of the vehicle, a vehicle ahead, and the like. In this embodiment, since it is desired to generate an image of the road surface, a partial region of this frame data is cut out and used. A region A71 in the figure represents a cut-out region set to include only the road surface. In this embodiment, the area A71 is set so as to acquire an image of an area 5 to 7 m ahead of the vehicle. The relative position of the area A71 in each frame is constant.
The area A71 is not limited to the above example, and can be set arbitrarily. Since the video camera 120 captures images at a constant frame rate, the frame data is an image group obtained by intermittently capturing the road surface. Therefore, it is preferable to determine the range of the area A71 so that the road can be reproduced as a continuous image when the group of images photographed intermittently are arranged. For example, if the vertical width of the area A71 is narrowed, a gap is likely to be generated between an area cut out from certain frame data and an area cut out from the next frame data when the vehicle speed is high. On the other hand, if the vertical width of the area A71 is increased, miscellaneous images different from the road image such as the preceding vehicle, the sky, and the building are likely to be included. The area A71 may be set in consideration of these influences.

Next, the CPU converts the acquired frame data into an orthogonal image (road surface texture) (step S12). The outline of the processing is shown in the figure. On the upper side is an example of frame data. Here, an example is shown in which only the road surface state is photographed, and the left and right lane regulation lines L71 and L72 and the sign M7 are photographed. Since this is an image of the front, the lane restriction lines L71 and L72 that are essentially parallel are shown in a square shape under the influence of perspective (perspective).
As described above, a partial area A71 of this frame data is cut out and used.
The lower part illustrates a state in which the image of the area A71 is orthographically converted. In order to convert the road into an image viewed from directly above, the left and right lane regulation lines L71 and L72 are converted into parallel line segments as shown in the figure. The sign M7 is similarly converted into a shape as seen from directly above.

An orthographic projection conversion method will be described.
First, it is assumed that a vehicle on which the road surface photographing system 100 is mounted is traveling on a horizontal plane, and a road as a subject is also on the same horizontal plane.
At this time, the road image, that is, the two-dimensional coordinates on the screen of the frame data is m = [u, v] ^T. Further, the three-dimensional coordinate of the world coordinate system fixed to the ground is M = [X, Y, Z] ^T. A vector obtained by adding one element to each of these coordinates as a direct product is defined as the following equation (1).

The relationship between the three-dimensional coordinate M and the two-dimensional coordinate m of the projected image is modeled by the following relational expressions (2) and (3).

Where s is the scale factor;
[Rt] is an external parameter matrix;
R is a rotation matrix;
t is a translation matrix;
A is an internal parameter matrix.
The internal parameter matrix A is an internal parameter considering the focal length of the video camera 120 and the like, and represents a mapping parameter from the real image coordinate system (xy coordinate system) to the frame coordinate system (uv coordinate system).
α and β are scale factors in the u-axis and v-axis directions, respectively, and γ is a parameter represented by the skew of the two image axes;
[U ₀ , v ₀ ] ^T is the coordinate (principal point coordinate) of the principal point of the image.

Assuming that the pixel size of the image is (k _u , k _v ), the angle between the u axis and the v axis is θ, and the focal length is f, α, β, and γ are expressed by the following equation (4).

The external parameter matrix [Rt] is an external parameter depending on the installation position, installation posture, and the like of the video camera 120, and represents a mapping parameter from the world coordinate system (XYZ coordinate system) to the real image coordinate system (xy coordinate system). . In the world coordinate system, the road surface directly below the video camera 120 is the origin, the horizontal axis perpendicular to the traveling direction of the vehicle is the X axis, the vertical axis is the Y axis, and the horizontal axis in the traveling direction is the Z axis.
The parallel movement vector t is a movement vector of the image principal point of the real image with respect to the origin in the world coordinate system.
When the height of the video camera 120 (the height of the main image point of the actual image) is h, the translation vector t is expressed by the following equation (5).

In the world coordinate system, if the rotation angle (yaw angle) in the heading direction of the real image is φ, the pitch angle is ω, and the roll angle is κ, the rotation matrix R is expressed by the following equation (6).

The internal parameter matrix A is obtained by a prior measurement.
The yaw angle φ, pitch angle ω, roll angle κ, and height h of the image principal point are obtained by the following procedure. First, in an initial state, that is, a state where the vehicle is installed on a horizontal ground, reference values of the yaw angle φ ₀ , the pitch angle ω ₀ , the roll angle κ ₀ , and the height h ₀ are measured. Next, while driving, the change in the vehicle attitude angle and the change in vehicle height are recorded with a gyroscope, an acceleration sensor, etc., and this change is reflected in the above-mentioned reference value. The angle φ, the pitch angle ω, the roll angle κ, and the height can be obtained.

The orthographic projection conversion is performed by using the equation (2) based on these parameters, and a road image in the frame coordinate system (uv coordinate system) is converted into a projected road image in the world coordinate system (XYZ coordinate system). Can be converted. The procedure is as follows.
First, it is assumed that the road surface as a subject is an image of a horizontal plane (Y = 0). At this time, the relationship of the following equation (7) is established from the equation (2).

As a result, the world coordinates (X, Z) and the scale parameter s for the pixel (u, v) can be obtained by the following equation (8).

Next, the CPU of the road marking map generating apparatus 200 performs correction in consideration of the inclination of the road surface that is the subject.
First, from the position coordinate data (X ₀ , Y ₀ , Z ₀ ) of each point where the frame data is acquired, and the position coordinates (X _i , Y _i , Z _i ) of a plurality of points near the road surface that is the subject, The slope of the road surface that is the subject is calculated. In this embodiment, it is assumed that the gradient is uniform.
Specifically, the height change Δh is obtained from position coordinate data in the vicinity of the world coordinate point (X, Y, Z) of the shooting point. That is Δh = _{Y-Y 0.} At this time, assuming a uniform gradient, the depth Z ′ of the points in the world coordinate system (X ′, Y ′, Z ′) on the road surface can be obtained by the following equation (9).

When the corrected depth Z ′ on the road surface is determined, the relationship between the frame coordinate point (u, v) and the world coordinate point (X ′, Y ′, Z ′) is expressed by the following equation (10) from the equation (2). It becomes as follows.

Thus, X ′ and Y ′ of the world coordinate point can be calculated by the following equation (11).

As described above, an orthographic image (road surface texture) can be obtained by mapping the point (u, v) on the frame data to (X ′, Z ′), respectively. As shown in FIG. 7, when the frame data is cut out in a rectangular area A71 and orthographically projected, a trapezoidal orthographic image (road surface texture) A72 spreading upward is obtained.
In the present embodiment, for the convenience of the subsequent processing, the orthographic image (road surface texture) is generated in two ways of low resolution / high resolution. A high-resolution orthographic image (road texture) (hereinafter referred to as a “high-resolution image”) is generated with the same resolution as the original image, ie, an image generated by using the cut-out area A71 of the original frame data as it is. It is an image that was made. A low-resolution orthographic image (road texture) (hereinafter referred to as “low-resolution image”) is an image whose resolution is lower than that of the original data. The resolution of the low-resolution image is preferably set to such a value that the road marking map generating apparatus 200 can display the image with a light load, and can be arbitrarily set such as half the resolution of the original image.

Next, the CPU of the road marking map generating apparatus 200 arranges the obtained orthographic image (road surface texture) and synthesizes a one-pass image (step S14). An example of 1-pass image synthesis is shown in the figure. In this example, orthographic images (road surface texture) A72 [0] to A72 [5] are synthesized.
In each orthographic image (road surface texture) A72, a point corresponding to the origin of the frame coordinate system (uv coordinate system) may be arranged based on the position coordinates at the time of shooting each frame data. Since the frame data is a forward image of the position of the vehicle, the orthographic image (road surface texture) needs to be arranged at a point moved forward by a certain distance from the vehicle position. In this embodiment, the positional relationship between the vehicle position and the frame coordinate system is calculated and arranged for each frame data. In addition, the orthographic image (road surface texture) is sequentially arranged from the old image to the new image in time series.

By arranging the orthographic image (road surface texture) in this way, the lane boundary lines L71 and L72 and the marking M7 on the road surface are reproduced.
In the present embodiment, at the stage of the connected image generation processing, the orthographic image (road surface texture) is kept arranged and displayed without being combined with one image. Therefore, what is generated by the one-pass image synthesis process (step S14) is not a synthesized image but information that determines the arrangement of each orthogonal image (road surface texture). However, in this process, a method of combining an orthographic image (road surface texture) into one image can be adopted.

C2. Alignment processing:
FIG. 8 is a flowchart of alignment processing. In terms of hardware, this is a process executed by the CPU of the road marking map generating apparatus 200. This corresponds to the processing of the alignment processing unit 220 shown in FIG.
When the process is started, the CPU first inputs a designation from the operator regarding a road to be processed (hereinafter referred to as “target road”) (step S20). Then, a connected image corresponding to the target road is input (step S22). In this embodiment, each road is traveled a plurality of times while changing the travel position, and a road surface image is taken. Therefore, a connected image is generated based on the path corresponding to each run. In step S22, the plurality of connected images are read.
Next, the CPU sets a reference path (step S30). The reference path is a path that serves as a reference when positioning a plurality of paths. In the present embodiment, among the paths corresponding to the target road, the evaluation value of the position accuracy, that is, the one having the highest self-estimated position accuracy is selected. The reference path setting method will be described later.

When the reference path is set, the CPU performs processing for setting corresponding points for each path in accordance with the operation of the operator (step S40).
In this embodiment, as shown in the figure, a method of displaying a linked image of a reference path and a standard path on a display, and an operator operating a pointing device such as a mouse to set corresponding points in this screen. Was taken. In the example of the figure, the vertex of the pedestrian crossing warning sign having a rhombus shape in the standard path image is designated as the corresponding point, and then the corresponding vertex is designated in the reference path image. The corresponding points are not limited to one point, and a plurality of points can be designated.
When a paint recognition process (to be described later) is performed and a sign in each connected image is extracted, the CPU displays the extracted sign on the display, and the sign that the operator should use as a corresponding point from this is displayed. You may make it select. Further, it is possible to identify a corresponding sign based on the positional relationship between the signs extracted between connected images, and to automatically set corresponding points and transparent polygons.

In this embodiment, a low-resolution image is used for displaying the connected image. By doing so, there is an advantage that when the corresponding point is designated, the display can be smoothly moved, enlarged and reduced, and the work efficiency can be improved.
When the corresponding point is designated, the CPU performs a process of moving the standard path connected image to match the reference path connected image so that the corresponding points match each other, and ends the alignment process (step S50). ).
As described above, in the present embodiment, the connected image is not generated as a single composite image, but an orthographic image (road surface texture) is arranged and displayed. Therefore, in the process of step S50, the movement process of a connected image is performed by moving each orthogonal image (road surface texture). Along with the movement process, a process of replacing each orthogonal image from a low resolution image to a high resolution image is performed. Processing for rearranging the orthographic image may be performed using the high-resolution image.
The contents of the connected image moving process will be described in detail later.

C3. Standard path setting process:
FIG. 9 is a flowchart of the reference path setting process. This process corresponds to step S30 of the alignment process (FIG. 8), and is a process for setting a reference path having the highest self-estimated position accuracy when aligning a plurality of paths.
When starting the processing, the CPU inputs the position accuracy at each point where the frame image is acquired for each path of the target road (step S31). At the time of shooting, as shown in the figure, a frame image is shot along the path at points P91, P92, P93, etc., and position accuracy AC1 in the east-west direction and position accuracy AC2 in the north-south direction are recorded for each point. Yes.

In general, it is known that the detection accuracy of the GPS 114 varies depending on the arrangement of artificial satellites used for detecting position coordinates, the reception status of radio waves, the presence of multipath by receiving radio waves reflected on buildings, and the like. Yes. In differential positioning, the detection accuracy is also affected by the operating status of the reference station. The position accuracy is a quantitative evaluation of these effects. The position accuracy can be arbitrarily defined. For example, a precision reduction rate (DOP (Dilution of Precision)) or the like may be used.
The CPU calculates the self-estimated position accuracy σ for each path based on the position accuracy of each point (step S32).

The self-position estimation accuracy may be a value determined based on a difference between GPS and IMU, DMI, or the like. In this case, for example, a standard deviation of the deviation amount may be used. Alternatively, the sum of the square of the standard deviation in the east-west direction and the square of the standard deviation in the north-south direction may be obtained, and this square root may be used as the self-position estimation accuracy. As described above, when the GPS, IMU, DMI, and other values are used, the self-position estimation accuracy increases as the deviation increases. That is, the smaller the value of the self-estimated position accuracy is, the higher the accuracy is.
When the self-estimated position accuracy σ of each path is obtained, the CPU sets the path having the minimum value as the reference path (step S33). If there is only a single path for the target road, that path is unconditionally set as the reference path. Let the self-estimated position accuracy of this reference path be σ _B.

If the self-estimated position accuracy σ _{B of} the reference path set in step S33 is lower than the predetermined threshold σ _TH (step S34), the reference path setting process is terminated.
On the other hand, if the self-estimated position accuracy σ _B is equal to or greater than the predetermined threshold σ _TH , an error display is performed (step S35), and the process ends. In this case, it means that the position accuracy of the reference path is not sufficiently ensured, and therefore the position accuracy is not sufficiently ensured even if the alignment process is performed.
As described above, the predetermined threshold σ _TH can be arbitrarily set based on the position accuracy to be secured as the road marking map.

It is only the self-estimated position accuracy σ _B of the reference path that is the object of judgment whether or not to perform error display (step S35). This is because, with respect to other standard paths, even if the self-estimated position accuracy is low, it is possible to improve the position accuracy by performing alignment with reference to the reference path.
However, it can be said that the correction in the alignment process is preferably as small as possible for any path. Therefore, in step S34, the self-estimated position accuracy of all the paths is compared with the threshold value _σTH, and if any one of the paths has an accuracy lower than the threshold value, an error display may be performed. .
However, if the standard path is required to have the same position accuracy as the reference path, there is a possibility that error display is frequently performed. In order to avoid such an adverse effect, a higher threshold σ _TH may be used in the standard path than in the reference path. In other words, for the standard path, the positional accuracy requirement is relaxed compared to the reference path. By doing so, it is possible to avoid frequent error display while guaranteeing the minimum position accuracy for the standard path.

C4. Connected image movement processing:
(1) Flow chart:
FIG. 10 is a flowchart of the connected image movement process. This corresponds to the processing in step S50 of the alignment processing (FIG. 8).
When the process is started, the CPU inputs standard path data and corresponding point data to be moved (step S51). The standard path data is trajectory data including a point sequence in which position coordinates when a frame image is taken are sequentially recorded. The corresponding point data is the coordinate value of the corresponding point designated by the operator in the screen on which the reference path and the standard path are displayed in step S20 of FIG.

Next, the CPU calculates a movement vector for each point where the orthographic image (road surface texture) is arranged on the standard path (step S52).
An example of movement vector calculation is shown in the figure. In this example, it is assumed that corresponding points P101 and P103 are designated for the standard path NP10. On the standard path, an orthographic image (road surface texture) is arranged as shown by a trapezoid in the drawing.
As points corresponding to the corresponding points P101 and P103, it is assumed that corresponding points P102 and P104 are designated on the reference path. The CPU obtains a movement vector for the corresponding points based on these designation results. In the illustrated example, a movement vector V10 from the corresponding point P101 to P102 of the standard path and a movement vector V11 from the corresponding point P103 to P104 are obtained.

  The corresponding point is not necessarily specified on the standard path NP10 because the operator specifies a point such as the apex of the sign that the operator can easily correspond with the standard path and the standard path. When the corresponding point is specified at a location deviated from the standard path NP10, the movement vector V10a is obtained at a location deviated from the standard path NP10 as indicated by a broken line in the drawing. Therefore, the movement vector V10 may be obtained by moving in the vertical direction to the standard path NP10 so that the starting point of the movement vector V10a is on the standard path NP10.

  When the movement vectors V10 and V11 at the corresponding points are obtained, the CPU obtains the movement vectors at the respective points located between the corresponding points P101 and P103 by interpolating these. For example, as shown in the figure, when the movement vector is obtained at the shooting point PP10 of the frame image, the movement vectors V10 and V11 are translated so that this point is the starting point, and a line segment connecting the end points of both vectors is obtained. Is internally divided by the ratio of the distance between corresponding points P101 to PP10 and the distance between P103 to PP10. By doing so, it is possible to obtain a movement vector VP10 starting from the point PP10 and ending at this internal dividing point.

For points that do not exist in the section between the two movement vectors V10 and V11, the movement vector at the closest position is used as it is. In the example in the figure, the movement vector V10 is used as it is in the section on the right side of the point P101, and the movement vector V11 is used as it is in the section on the left side of the point P103.
Further, when only one corresponding point is specified and only one movement vector is given, this movement vector is used.

The CPU translates the orthographic image (road surface texture) according to the movement vector obtained by the above processing (step S53), and ends the connected image movement processing. In the example of the figure, an example is shown in which the road surface texture TX11 arranged at the point PP10 of the standard path NP10 is translated to the position of the road surface texture TX12 according to the movement vector VP10.
Along with this processing, the position of the point PP10 on the standard path NP10 is also corrected by the movement vector VP10. Therefore, in the process of step S53, the locus of the standard path NP10 is also corrected along with the movement of the road surface texture.

(2) Positioning processing example (1):
FIG. 11 is an explanatory view showing a processing example (1) of alignment processing. Each of FIG. 11A to FIG. 11C shows a state where the linked images for the standard path NP11 and the reference path BP11 are displayed in an overlapping manner. FIG. 11A shows a state in which the connected image of the standard path NP11 is arranged above the connected image of the reference path BP11. As described above, the connected image is configured by arranging a large number of road surface textures, but for the sake of convenience of explanation, one road surface texture TX11 is illustrated with an outline.
The operator designates the corresponding point P111 in the standard path NP11 in this screen. The corresponding point P111 can be arbitrarily set. In this embodiment, one of the end points of the white stripe diagonal stripe pattern of the separation band sign M11 is selected as the corresponding point P111.

FIG. 11B shows a state where the connected images of the reference path BP11 are arranged on the upper side. In this state, the positions of the standard path NP11 and the reference path BP11 are shifted. Therefore, when the connected image of the reference path BP11 is displayed on the upper side, the position of the corresponding point P111 is deviated from the diagonal stripe pattern of the white line of the separation band sign M12.
FIG. 11C shows a state in which the corresponding point P112 is specified with the connected image of the reference path BP11 facing upward. That is, in the image with the reference path BP11 on the upper side, the end point of the white stripe of the separation band sign M11 may be selected as the corresponding point P112.
When the corresponding point P112 is designated, the movement vector V11 is obtained so as to go from the corresponding point P111 of the standard path NP11 to the corresponding point P112 of the reference path BP11. If the road texture TX11 is moved according to the movement vector V11, the corresponding point P111 coincides with the corresponding point P112, and the positions of the separation band signs M11 and M12 can also coincide.

  In the alignment process, not only the road surface texture TX11 but also other road surface textures constituting the standard path NP11 are similarly moved according to the movement vector V11. Although an example of processing in which only one corresponding point is specified is shown here, a plurality of corresponding points may be specified. For example, in the example of the figure, it is conceivable to use the crossing stripe pattern, the stop line, the end point of the lane boundary line, or the like as the corresponding point.

(3) Positioning processing example (2):
FIG. 12 is an explanatory view showing a processing example (2) of alignment processing. The state where the connected images of the standard path NP12 and the reference path BP12 are overlapped is shown. For convenience of explanation, both road markings are shown in a visible state. Before the alignment, the positions of the standard path NP12 and the reference path BP12 are deviated, and thus the positions of the markings such as the lane boundary line are deviated.
Here, the operator selects one of the end points of the lane boundary line indicated by a broken line as a corresponding point. For the standard path NP12, the end point of the lane boundary line L122 is selected as the corresponding point P122, and for the reference path BP12, the end point of the lane boundary line L121 is selected as the corresponding point P121. As a result, a movement vector V12 from the corresponding point P122 of the standard path NP12 toward the corresponding point P121 of the reference path BP12 is determined.

FIG. 13 is an explanatory diagram showing a processing result of the positioning processing (2).
As described above, the position of the lane boundary line can be adjusted by moving the connected image of the standard path NP12 according to the movement vector V12. The result of the alignment is the lane boundary line L13.
In addition, by this alignment process, the standard path is also aligned with the position of the reference path. In the present embodiment, an image of a road surface is generated by aligning a plurality of paths originally traveling at different positions. At this time, as can be seen from the comparison between FIG. 12 and FIG. 13, by moving the standard path in parallel according to the movement vector set based on the corresponding point, the positional relationship of the road marking and the position of the path between the plurality of paths. Relationships can be matched very well.

(4) Acquisition of absolute coordinates:
FIG. 14 is an explanatory diagram showing a method for obtaining the absolute position coordinates of the road marking. In the illustrated example, the road surface texture TX142 on the standard path NP14 and the road surface texture TX141 on the reference path BP14 are illustrated. In the road surface textures TX141 and TX142, signs M141 and M142 are included, respectively.
The road surface textures TX141 and TX142 are arranged so that their representative points coincide with a point P141 on the reference path BP14 and a point P143 on the standard path NP14.

In the road surface texture TX141, the position of the vertex P142 of the sign M141 can be specified by relative coordinates (x142, y142) with the representative point as the origin. Therefore, if the absolute coordinates of the representative point, that is, the position coordinates (X141, Y141) where the road surface texture TX141 is arranged are known, the absolute position of the vertex P142 of the sign M141 is added by adding the relative coordinates described above. Coordinates can be acquired.
Similarly, in the road surface texture TX142, the position of the vertex P145 of the sign M142 can be specified by relative coordinates (x145, Y145) with the representative point as the origin. Accordingly, if the absolute coordinates of the representative point, that is, the position coordinates (X143, Y143) where the road surface texture TX142 is arranged are known, the relative position described above is added to the absolute position of the vertex P145 of the sign M142. Coordinates can be acquired.

For the road surface texture TX142, it is assumed that the position P143 of the representative point is moved to the point P144 according to the movement vector V14 by the alignment process. At this time, the absolute position coordinates of the point P144 after alignment can be obtained by adding the components (VX14, VY14) of the movement vector V14 to the position coordinates (X143, Y143) of the point P143 before movement. Furthermore, if the relative coordinates (x145, Y145) of the point P145 are added to the absolute position coordinates of the point P144 obtained in this way, the absolute position coordinates of the vertex P145 of the sign M142 after the alignment processing are acquired. Can do.
Here, the method of obtaining the absolute position coordinates for the vertices of the signs M141 and M142 in the road texture has been shown. However, each arbitrary point in the road texture is a relative coordinate based on the representative point of the road texture. Since it can be specified, the absolute position coordinates of an arbitrary point can be obtained by a similar method.

C5. Transparent polygon setting process:
(1) Process overview:
FIG. 15 is an explanatory diagram showing an outline of the transparent polygon setting process. In the transparent polygon setting process, the transparent polygon is set on the superimposed road image according to the operator's instruction so that the orthographic image corresponding to the adjacent path overlaps at the upper orthographic image. This is a process in which part of the image is made transparent so that the lower orthographic image can be seen through. Here, the processing contents will be described by taking as an example a case where the operator manually sets.
In the center of the figure, a state in which the orthographic image P151 is superimposed on the orthographic image P152 is shown in a perspective view. The lower orthographic image P152 includes the pedestrian crossing A154 in a divided state, and includes the stop line A153 in a complete state. In the upper orthographic image P151, the pedestrian crossing A152 is included in a complete form, and the stop line A151 is included in a divided state. Each divided part is shown surrounded by a broken line.

When the orthographic images P151 and P152 are overlapped in this state, they are displayed as shown on the left side. That is, in the part where both overlap, only the image of the upper orthographic image P151 is displayed, so the pedestrian crossing A152 is displayed in a complete state, but the stop line A151 is shown in a divided state. It is.
If the vertical relationship between the orthogonal images P151 and P152 is changed, the stop line A153 can be displayed in a complete state, but the pedestrian crossing A154 is displayed in a divided state. . Thus, it is not possible to display both the pedestrian crossing and the stop line in a complete state only by the vertical relationship between the orthogonal images P151 and P152.

  Therefore, in this embodiment, the transparent polygon POL15 is set. In this example, the upper orthographic image P151 is set so as to cover the divided stop line A151. In the transparent polygon POL15, the upper orthographic image P151 is displayed in a transparent state. Therefore, as shown on the right side of the figure, when the orthographic images P151 and P152 are overlapped, the lower orthographic image P152 is displayed inside the transparent polygon POL15, and the upper orthographic image is displayed in other portions. An image P151 is displayed. As a result, the stop line A153 included in the lower orthographic image P151 and the pedestrian crossing A152 included in the upper orthographic image P152 are displayed, and both the stop line and the pedestrian crossing can be displayed in a complete form. it can.

(2) Flow chart:
FIG. 16 is a flowchart of the transparent polygon setting process. In terms of hardware, this is a process executed by the CPU of the road marking map generating apparatus 200. This corresponds to the processing of the transparent polygon setting unit 221 shown in FIG.
When the process is started, the CPU inputs designation of the target road from the operator (step S100), and inputs a connected image corresponding to the target road (step S102). When a plurality of paths correspond to the target road, a plurality of connected images corresponding to these paths are input.

The CPU displays these connected images and inputs designation of a priority path based on the operation of the operator (step S104). The priority path refers to a path having the best road image among a plurality of paths, and refers to a path positioned at the top when overlapping connected images of a plurality of paths. The priority path is different from the reference path used in the alignment process. The reference path means the one with the best position accuracy, but just because the position accuracy is good does not mean that the road surface image is good. Regardless of the top-to-bottom relationship of the overlay of connected images between multiple paths, alignment can be performed without any problem, so the alignment reference path and priority path are independent of each other. And can be set.
In this embodiment, the priority path can be arbitrarily set while the operator compares the linked images of the paths. The path with the roughest road surface image may be designated as the priority path. In such a case, only the number of transparent polygons to be described later increases.

When the priority path is set, the CPU sets a transparent polygon according to the operation of the operator (step S106).
An example of transparent polygon setting is shown in the figure. In this example, the road surface texture TX161 along the priority path and the road surface texture TX162 along the other paths are shown.
At the time of shooting, a rectangular image becomes a trapezoid by orthographic image conversion. Therefore, when the road surface textures TX161 and TX162 are arranged, it becomes a saw blade shape as shown in the figure. From the saw blade portion, the appearance of the road surface image is deteriorated and only the divided road surface image can be obtained. Therefore, the portion is unnecessary for the purpose of obtaining a complete road surface image. Therefore, in the example of the figure, in the portion where the road surface textures TX161 and TX162 overlap, the transparent polygon POL161 is set at the left end portion of the road surface texture TX161 having a saw blade shape so that the saw blade portion is not displayed. ing.

On the other hand, transparent polygons are not set in the portions where the road surface textures TX161 and TX162 do not overlap, in the example shown in the figure, the regions A161 and A162 at both ends. This is because in this portion, the images obtained by the road surface textures TX161 and TX162 are the only image information. If transparent polygons are set in the regions at both ends, the information on the road surface image included in this portion cannot be used. In the present embodiment, the information on the road surface image included in the road surface texture can be effectively used by not setting the transparent polygon in the portion that does not overlap with the other road surface texture.
Such a setting may be realized by an operation in which the operator sets a transparent polygon by simply avoiding a portion where the road surface texture does not overlap, but in the transparent polygon setting process (step S106), the transparent polygon You may make it restrict | limit a setting position. In other words, the transparent polygon setting operation by the operator may be accepted only for the portion where the road surface texture overlaps.

When there is a sign hidden by the road surface texture TX161, the operator sets the transparent polygon so that the sign can be visually recognized. In the example shown in the figure, the transparent polygon POL 162 is set so as to cover the arrow mark. The arrow mark is an image included in the texture arranged on the lower side of the texture TX161.
In order to set the transparent polygon POL162 that covers the sign in this way, the vertical relationship is temporarily changed so that the road texture TX161 is positioned below the other road texture, or the road texture TX161 is not displayed. Just do it. With these operations, the sign hidden in the road texture TX161 is made visible, the transparent polygon POL162 is set so as to cover the sign, and the display of the road texture TX161 is restored.

When the setting of the transparent polygon is completed by the above processing, the CPU outputs the setting result and ends the transparent polygon setting processing.
(3) Processing example:
FIG. 17 is an explanatory diagram showing an example of a road image before setting a transparent polygon. In this example, a road image generated by performing alignment of connected images obtained along two paths P171 and P172 is shown. The shape of both ends of the saw blade in the connected image of the path P172 and the connected image of the path P171 are opposite to each other in the direction in which the vehicle of the road surface photographing system 100 travels on these paths P171 and P172. This is because the opposite is true.

In a portion where the connected image of the path P172 overlaps with the connected image of the path P171, a saw-toothed boundary at the end B17 of the connected image of the path P172 appears, degrading the image quality of the road image. However, in FIG. 17, for the convenience of illustration, the shape of the end portion B <b> 17 is emphasized with a saw-blade contour.
Further, since the road surface image of the path P172 is unclear at the end, for example, in the region A171, the stripe pattern of the pedestrian crossing is distorted. In the area A172, the stop line is divided. In the area A173, the character “bus only” indicating that the route is a preferential traffic zone such as a route bus (so-called bus lane) is broken so that it cannot be read. In the area A174, the broken-line lane boundary line is divided in the middle.

In order to avoid these influences, in FIG. 17, the transparent texture POL <b> 17 including the regions A <b> 171 to A <b> 174 and the end B <b> 17 is set as indicated by a one-dot chain line in the drawing.
When the transparent polygon POL17 is set in this way, the road surface texture on the path P172 side is in a transparent state inside the transparent polygon POL17, and the road surface texture on the path P171 side arranged below is visually recognized.

FIG. 18 is an explanatory diagram showing an example of a road image after setting a transparent polygon. Due to the action of the transparent polygon described above, the lower image is displayed in the area A181, so that the divided state of the pedestrian crossing shown in FIG. 17 is eliminated. Similarly, in the area A182, the stop line is displayed in a complete state. Further, as exemplified in the region B18, the saw-tooth profile at the end of the road surface texture is not visually recognized, and the image quality of the entire road image is improved.
In the area A183, the bus-specific characters are clearly readable. In the area A184, the lane boundary line is displayed in a complete state.
As described above, in this embodiment, by setting the transparent polygon, the image quality of the road image can be improved and the image quality of the marking on the road surface can be improved.

C6. Paint recognition process:
(1) Overall processing:
FIG. 19 is a flowchart of the paint recognition process. In terms of hardware, this is a process executed by the CPU of the road marking map generating apparatus 200. This corresponds to the processing of the paint recognition unit 223 shown in FIG.
When the process is started, the CPU reads the image of the road to be processed, that is, road surface texture and path data, from the processing data storage unit 210 (step S300).
Next, the CPU performs vertical arrangement processing (step S302), generates a vertical arrangement image, and stores it in the processing data storage unit 210.

  FIG. 20 is an explanatory diagram showing the contents of the vertical arrangement processing. FIG. 20A shows a connected image in normal processing. A one-dot chain line arrow PA20 in the figure represents a path when an image is taken. In normal processing, the road surface texture Tx20 is arranged along the path PA20. The position coordinates of the path PA20 are obtained in an absolute coordinate system XY such as latitude and longitude. Therefore, when the path PA20 and the road surface texture Tx20 are arranged in the absolute coordinate system, the connected image may be displayed obliquely as shown in FIG. In this example, a straight road is illustrated, but when the road is curved, the connected image is also curved.

FIG. 20B shows an example of vertically arranged images. Since the road is straight, the image in FIG. 20A is rotated in the direction of arrow A20.
The vertically arranged image can be generated by the following procedure. First, a distance axis is set in the longitudinal (y) direction of the two-dimensional coordinate xy. The distance axis is an axis representing a moving distance from the start point when an image is taken along the path PA20. When the path PA20 is linear, the traveling direction of the path PA20 is displayed upward. When the path is curved, the path is straightened.

  For each road texture Tx20, since the position coordinate data at the time of shooting and the movement distance from the start of shooting are obtained (see the position measuring unit 110 in FIG. 1), the road texture on the distance axis is based on these data. Tx20 is arranged. The road surface texture Tx20 is arranged such that the representative point in the image is placed on the distance axis and the left-right symmetry axis is parallel to the distance axis. By doing so, it is possible to display a connected image in which the traveling direction is linearly extended in the vertical direction regardless of whether the path is curved. According to this connected image, as shown in the figure, the pedestrian crossing and the stop line are drawn in a direction perpendicular to the distance axis (left and right direction in the figure), and the lane boundary line is in a direction along the distance axis (up and down direction in the figure). Drawn in. In the vertically arranged image, road markings are drawn in a fixed positional relationship in this way, so that there is an advantage that these can be easily recognized. Here, an example is shown in which the distance axis is arranged vertically, but the distance axis can be arranged in any direction such as laterally or diagonally.

Returning to FIG. 19, the paint recognition process will be described.
When the vertical arrangement process is completed, the CPU performs a crossing-related paint extraction process (step S310). This is a process of extracting signs in the vicinity of intersections such as pedestrian crossings, bicycle crossings, and stop lines. In the present embodiment, a vertically arranged image stored in the processing data storage unit 210 is used, and a sign drawn there is extracted by image processing, and conditions based on the positional relationship and length of line segments included in the sign are included. Judgment type is judged by judgment.

When the crossing-related paint process ends, the CPU performs various paint extraction processes (step S350). Indices to be extracted in this process are shown in the figure.
The “boundary line” is a lane boundary line drawn with a solid line or a broken line.
“Arrow” is an arrow shown in each lane in the vicinity of the intersection in order to indicate regulation of the traveling direction within the intersection.
Unlike a pedestrian crossing, the “zebra” is a striped pattern that is marked on the median strip or where there are more lanes for turning left and right.
“U-turn” is a U-shaped arrow drawn on a U-turn prohibited road.
“Turning prohibited” is a cross marked with a U-turn arrow.
“Regulation” means time marking of traffic regulation. For example, traffic mode restrictions such as a sign “17-19” (meaning from 17:00 to 19:00) drawn together with a sign such as a bus lane.

The “number” is a number such as speed regulation.
“Pedestrian crossing notice” is a diamond-shaped symbol drawn in front of the pedestrian crossing.
The “deceleration zone” is a striped pattern drawn by arranging a plurality of line segments in a direction perpendicular to the path on the road surface in parallel along the traveling direction of the path in order to promote the deceleration of the vehicle speed. .
“Road surface painting” is a pavement area such as red, which is provided for safety of traffic at sharp curves and other places where the driver should be alerted.
The “bus lane character” is a character “bus lane” attached to a lane used as a bus lane. In the present embodiment, the bus lane is illustrated, but the present invention is not limited to the bus lane, and may be a general character displayed on the road surface.
The “end symbol” is a “0” -shaped symbol indicating an end point such as a bus lane.
In the present embodiment, these signs are targeted, but these are merely examples, and more signs may be targeted, and some of them may be excluded from the extraction process.

In various paint extraction processes (step S350), pattern matching is performed using a model prepared in advance. In this embodiment, two types of artificial model images and OCR model images are used. These models are stored in the processing data storage unit 210 in advance.
An artificial model image is a model generated by computer graphics. It is suitable as a model for matching relatively simple markings such as arrows, U-turns, and turning prohibition.
An OCR model image is a model generated based on an image that is manually cut out by an operator from a captured image. For example, it is suitable as a model for matching signs having complicated shapes such as numbers and bus lanes. Although it is not impossible to generate these models by computer graphics, in order to generate a model by combining the shape of characters with actual road markings, it is necessary to trace the captured image after all. As a result, it is not much different from using the OCR model.

The CPU performs the above processing in units of linear connected images, and then performs relative coordinate conversion processing (step S370) for conversion to the absolute coordinate system, stores the paint recognition result in the processing data storage unit 210, and paints The recognition process ends.
The recognized paint may be stored as image data, but may also store a type and an existing area. The existence area refers to a geometric shape including the recognized paint. For example, a rectangle circumscribing a pedestrian crossing, a bicycle crossing band, an arrow, or the like can be used as the existence area. The existence region may be represented by a centroid position and a diagonal length, or may be represented by coordinate values of two vertices located diagonally. The existence area is not limited to a rectangle, and any shape such as various polygons and circles can be used.

(2) Relative coordinate conversion processing:
FIG. 21 is an explanatory diagram showing the contents of the relative coordinate conversion process. In the present embodiment, the recognition results of the various signs described above are all obtained using linear images in which road textures are arranged along the distance axis. Therefore, as a recognition result, only a relative position in the coordinate system displaying this linear image is acquired. The relative coordinate conversion process is a process of converting this relative position into an absolute coordinate system position corresponding to the position coordinate at the time of photographing.
The left side of the figure shows a state in which the sign is recognized by a linear connected image. Here, rectangular existence areas M391 and M392 are shown. The positions of these existence areas M391 and M392 are represented by the position coordinates of the centroids G391 and G392 as representative points. These coordinates are given by a coordinate system xn, yn for displaying a linear connected image. In the coordinate systems xn and yn, for example, the distance axis NP39 is preferably defined as the yn axis.

In the connected image, the representative points of the road surface texture Tx39 are arranged on the distance axis according to the position at the time of shooting. The relative coordinates from the representative point of each point in each road surface texture Tx39 are known. Accordingly, since the relative coordinates within the road surface texture Tx39 of the centroids G391 and G392 of the respective existence regions are obtained, the coordinates of the perpendicular legs R391 and R392 taken from the centroids G391 and G392 on the distance axis can also be obtained.
The right side of the figure shows a state converted to the absolute coordinate system XY. Since the road is not necessarily a straight line, in the absolute coordinate system, the path RP39 may be drawn in a curved line according to the position coordinates acquired at the time of shooting.

The positions and orientations of the respective existence areas M391 and M392 are obtained by the following method.
First, according to the distance from the reference point of the curved path RP39, the positions of the perpendicular legs R391 and R392 on the path RP39 of the existing region are obtained, and the normal vectors V391 and V392 are obtained from the points R391 and R392, respectively. Is drawn, and the end points are set as the center of gravity G391 and G392. The magnitudes of the normal vectors V391 and V392 are equal to the distance between the points R391 and G391 and the distance between the points R392 and G392 in the linear connected image, respectively. The directions of the existence regions M391 and M392 are arranged so that the longitudinal direction is orthogonal to the normal vectors V391 and V392. In other words, the relative positional relationship between the existence areas M391, M392, the centroids G391, G392, and the perpendicular legs R391, R392 is expressed by a linear connected image (left side in the figure) and an absolute coordinate system (right side in the figure). The existence areas M391 and M392 are arranged so as to be the same.
By this conversion process, the position of each extracted sign can be expressed by the position coordinates of the absolute coordinate system.

C7. Automatic transparency polygon generation processing:
(1) Overall processing:
FIG. 22 is a flowchart of automatic transparent polygon generation processing. This is a process in which the CPU automatically executes the contents corresponding to the transparent polygon setting process (see FIG. 16) described above.
When this process is started, the CPU inputs a road image to be processed and a paint recognition result (step S400). These results are stored in the processing data storage unit 210. Then, the CPU determines whether or not the connected images overlap in the road image (step S402).

FIG. 23 is an explanatory diagram showing a method for determining overlap. In the drawing, road surface textures TX231 and TX232 are arranged along two paths RT231 and RT232, respectively, and a road image is generated.
The overlapping of these connected images can be determined by determining whether or not the road surface textures TX231 and TX232 overlap each other. However, in this case, overlap determination is performed for all combinations of the road surface texture TX231 arranged in the path RT231 and the road surface texture TX232 arranged in the path RT232, or road surface textures whose position coordinates are within a certain range, respectively. It is necessary to perform overlap determination with This is because even when the connected images of the paths RT231 and RT232 overlap each other as shown in the drawing, not all the road surface textures TX231 and TX232 overlap each other. For example, the leftmost road surface texture TX231 and the rightmost road surface texture TX232 do not overlap. Therefore, if an overlap determination is to be made in units of road texture, it is necessary to determine whether or not there is an overlap with the above-mentioned many combinations, and the processing for that and the management of the determination results become very complicated.

In this embodiment, the overlap determination can be easily performed by the following method.
First, the CPU sets a polygon that includes a connected image for each of the paths RT231 and RT232. In the example in the figure, for the path RT231, the vertexes P23 at both ends of each road surface texture TX231 are obtained, and these are connected by line segments to set a polygon A231 that includes a connected image.
Similarly, for the path RT232, vertices Q23 at both ends of each road texture TX232 are obtained, and these are connected by line segments to set a polygon A232 that includes a connected image.
Then, it is determined whether or not the polygons A231 and A232 set in this way are overlapped. Since the presence / absence of overlapping of polygons can be determined by various methods, detailed description thereof is omitted here.

Returning to FIG. 22, the automatic transparency polygon generation processing will be described.
The CPU can determine step S404 according to whether or not the polygons A231 and A232 overlap each other by the method described with reference to FIG. If the connected images do not overlap (step S404), there is no need to set a transparent polygon, so the CPU ends the automatic transparent polygon generation process.
If the connected images overlap (step S404), the CPU determines a priority path (step S406). The priority path refers to a path having the best road image among a plurality of paths, and refers to a path positioned at the top when overlapping connected images of a plurality of paths. The priority path can be set by any method. For example, a reference path that is used when a road image is synthesized may be unconditionally handled as a priority path. In addition, a path with a large number or area of paint (road markings) included in each connected image may be used as the priority path.

When the priority path is determined in this way, the CPU sets a route area (step S408). The root region refers to a region in the connected image excluding the jagged portions at both ends, that is, a region that is in a smooth state where there is no portion that is bent acutely at the edge line in the direction along the path. Step S408 is a process of defining the root area for the connected image, and is not a process of deleting the jagged areas at both ends.
FIG. 24 is an explanatory diagram showing a route area setting method. The state where road surface textures Ta24 to Tf24 are arranged along the path is shown. The road surface texture Ta24 is a trapezoid composed of vertices Pa1 to Pa4. Similarly, the road surface textures Tb24, Td24, Te24, and Tf24 are trapezoids including vertices Pb1 to Pb4, Pd1 to Pd4, Pe1 to Pe4, and Pf1 to Pf4. In order to avoid complication of illustration, the symbol of the vertex of the road surface texture Tc24 is omitted.

The CPU obtains intersections P1, Q1 between the lower base Pa1-Pa4 of the road texture Ta24 and the oblique sides Pb1-Pb2, Pb4-Pb3 of the road texture Tb24. Similarly, the intersection points P2 and Q2 of the road surface textures Tb24 and Tc24 and the intersection points P3 and Q3 of the road surface textures Td24 and Te24 can be obtained.
The intersections of the road texture Te24 and Tf24 are P4 and Q4. However, the intersection point P4 is an intersection point of the oblique sides Pf1-Pf2 of the road surface texture Tf24, but the intersection point Q4 is an intersection point with the upper base Pf2-Pf3. In this embodiment, since the intersection with the oblique side of the road surface texture is used in setting the route area, the intersection Q4 is not adopted.
The CPU establishes a route area RTA24 indicated by a bold line in the figure by connecting the intersections P1, P2,..., P3, P4 and Q1, Q2,.
The route area is not limited to this example, and various settings are possible, and the route area may be set using an intersection (point Q4 in the figure) with a side other than the oblique side.

Returning to FIG. 22, the automatic transparency polygon generation processing will be described.
When the route area setting is completed (step S408), the CPU performs a paint transparent polygon setting process (step S410). This is a process of determining whether or not to set the transparent polygon based on the overlapping state of the paint (road marking) in the connected image and setting its position and shape. The details of the transparent polygon setting process for painting will be described later.
Next, the CPU performs a jagged cut process (step S430). This process is a process of setting the transparent polygon so as to hide the jagged areas at both ends of the connected image. The processing contents will be described in detail later.
When the transparent polygon is set in this way, the CPU performs a combination process of the set transparent polygon (step S450). This is a process of combining overlapping objects among a plurality of transparent polygons. By performing this process, the number of transparent polygons can be reduced, and the capacity thereof can be reduced.
Hereinafter, the paint transparent polygon setting process (step S410), the jagged cut process (step S430), and the transparent polygon combination process (step S450) will be sequentially described.

(2) Transparent polygon setting process for paint:
FIG. 25 is a flowchart of the transparent polygon setting process for painting. This process corresponds to step S410 in the automatic transparency polygon generation process (FIG. 22). In this process, the CPU determines whether or not to set a transparent polygon and sets its position and shape based on the paint recognition result of each connected image.
First, the CPU extracts, as a processing target paint, paint existing in the overlapping area with the priority path from the paths arranged on the lower side in the road image, that is, paths other than the priority path (step S411). In addition, the overlapping paint is extracted from the paint existing in the overlapping area of the priority path (step S412). The overlapping paint is a paint of the same type as the processing target paint among the paints included in the priority path, and it overlaps at least partially with the processing target paint.

In the present embodiment, the processing object paint and the priority paint are extracted from the overlapping region of the upper and lower paths. This is because the setting of the transparent polygon is unnecessary because the paint is not obscured by other connected images except in the overlapping region. Also, if the paint of the area where only the upper connected image exists is extracted as the processing target, and the transparent polygon may be mistakenly set here, the upper connected image will be transmitted, so some will be missing This is because the road image in the state is generated.
In this embodiment, the extraction of the overlapping paint is limited to the same type of paint as the processing target paint, but the overlapping paint may be extracted regardless of the type of paint.
If the overlapping paint has been extracted (step S413), the areas of the processing object paint and the overlapping paint are compared (step S414). If the overlapping paint is larger (step S414), the CPU determines that the overlapping paint should be displayed on the road image. Since the overlapping paint exists in the priority path, it is determined that the transparent polygon is unnecessary.

On the other hand, when there is no overlapping paint (step S413), or when the area of the overlapping paint is smaller than the processing target paint (step S414), the CPU uses the paint existing below as a road image. Judge that it should be displayed. Therefore, the transparent polygon is set based on the position and shape so that the processing target paint can be displayed (step S415).
The CPU executes the above processing for all the processing target paints and all the passes (step S416), and ends the paint transparent polygon setting processing.

FIG. 26 is an explanatory diagram showing a setting example (1) of the transparent polygon for paint.
FIG. 26A shows a state in which the route area OR26 of the priority path is overlaid on the lower road surface texture TX26. The route area OR26 includes dashed lane boundary lines L261 and L262 as paint. The lower road surface texture TX26 includes an arrow A26 indicating the restriction of the traveling direction.
From the lower road surface texture TX26, an arrow A26 is extracted as a processing target paint. Since there is no overlapping paint there, no overlapping paint is extracted from the root region OR26. Therefore, since there is no overlapping paint (step 413 in FIG. 25), it is determined that a transparent polygon should be set. As a result, the CPU sets a rectangular transparent polygon R26 including the arrow A26.

The transparent polygon R26 can be set by various methods. In this embodiment, the region where the arrow A26 exists is used. In this embodiment, when recognizing a paint, the size and position are output in the form of an existing area that is a geometric shape including the paint. Therefore, in this embodiment, a shape obtained by dividing the existence area by several percent and using it as the transparent polygon is used. In this way, a transparent polygon having a position and size suitable for painting can be set relatively easily.
In addition to this, the transparent polygon may be generated, for example, by extracting the outline of the paint and enlarging the extracted shape by several percent.

FIG. 26B is an explanatory diagram showing a method (1) for expanding an existing area. In order to avoid complication of the figure, the lower side is represented in the form of the route area UR26 instead of the road surface texture Tx26 of FIG.
In this embodiment, the transparent polygon is set by enlarging the existing area at a preset enlargement ratio. However, as a result of the enlargement, if the transparent polygon covers the lane boundary lines L261 and L262 existing in the vicinity, the lane boundary lines L261 and L262 are transmitted. Therefore, when the enlarged transparent polygon is covered with another paint in the vicinity, as shown by the transparent polygon R262 in FIG. 26B, the CPU performs other paint (lane boundary line L261 in the example in the figure). , L262), the enlargement factor in the direction along the path or in the width direction is suppressed. FIG. 26B shows an example in which the enlargement ratio in the width direction is suppressed so as not to get over the lane boundary line L261.

FIG. 26C is an explanatory diagram showing a method (2) for enlarging an existing area. In this embodiment, the transparent polygon is set within the overlapping range of the upper and lower root areas. This is because if the transparent polygon is set in a region where only the upper route region exists, there is a possibility that the lower image does not exist and a part of the road image may be lost.
Therefore, when there is a possibility of exceeding the boundary line of the lower route area UR26 as a result of enlarging the existence area, as shown in the transparent polygon R263 in FIG. Suppresses the expansion rate. In this way, a transparent polygon can be set within the overlapping area of the root area.

FIG. 27 is an explanatory view showing a setting example (2) of the transparent polygon for paint.
FIG. 27A shows a road surface texture Tx272 along the lower path and a road surface texture Tx271 along the upper path. A lane boundary line L272 is included on the lower side, and a lane boundary line L271 is included on the upper side. These are the same kind of paint and overlap. Therefore, in the paint transparent polygon setting process (FIG. 25), the lower lane boundary line L272 is extracted as the processing target paint, and the upper lane boundary line L271 is extracted as the overlapping paint (steps S411 and S412 in FIG. 25).
If there is overlapping paint, a transparent polygon is set when the lower area is large (step S415 in FIG. 25). In the example of FIG. 27, since the lower lane boundary line L272 has a larger area, a transparent polygon is set here. The reason why the area of the upper lane boundary line L271 is small is that a part of the area is recognized as being missing. By setting the transparent polygon in this way, the lane boundary line L272 is displayed in the road image, so that it is possible to display accurate paint in a more realistic form.

FIG. 27B shows a setting example of the transparent polygon TP27. Again, the transparent polygon TP27 is set by enlarging the existence area of the lower lane boundary line L272.
First, the enlargement ratio of the existing area is suppressed so as not to cover other paint (arrow A27 in the example in the figure) existing in the upper root area OR27. In the example of the figure, the enlargement ratio in the width direction is suppressed above the lane boundary line L272. Since there is no other paint in the downward direction, the transparent polygon TP27 has been shown as an example expanded to the boundary of the lower route region UR27. However, the downward expansion rate also shows the lane boundary line L272. It is sufficient to set within the range that can be displayed reliably.
In the direction along the path (the left-right direction in the figure), the enlargement ratio is determined so as not to protrude from the overlapping range of the upper route region OR27 and the lower route region UR27. Assuming that the upper root area OR27 is cut out in the illustrated range, the enlargement ratio is limited so that the left side of the transparent polygon TP27 does not protrude from the boundary of the root area OR27, as illustrated.

FIG. 28 is an explanatory view showing a setting example (3) of the transparent polygon for paint.
FIG. 28A shows a road texture Tx282 along the lower path and a road texture Tx281 along the upper path. The lower side includes a pedestrian crossing CR282, and the upper side includes a pedestrian crossing CR281. These are the same kind of paint and overlap. Accordingly, in the paint transparent polygon setting process (FIG. 25), the lower pedestrian crossing CR282 is extracted as the processing target paint, and the upper pedestrian crossing CR281 is extracted as the overlapping paint (steps S411 and S412 in FIG. 25).
If there is overlapping paint, a transparent polygon is set when the lower area is large (step S415 in FIG. 25). In the example of FIG. 28, since the lower crosswalk CR282 has a larger area, a transparent polygon is set here.

FIG. 28B shows a setting example of the transparent polygon TP28. Also here, the transparent polygon TP28 is set by enlarging the existence area of the lower crosswalk CR282.
As in the other examples, the transparent polygon TP28 is set in a range that does not protrude from the lower root area UR28. Since the transparent polygon TP28 is intended to transmit the upper pedestrian crossing CR281, the enlargement ratio is preferably set so as to include the pedestrian crossing CR281 as illustrated.

(3) Jagged cut processing:
FIG. 29 is a flowchart of the jagged cut processing. This process corresponds to step S430 in the automatic transparency polygon generation process (FIG. 22).
The jagged cut process is a process for setting a transparent polygon to a saw-toothed jagged portion at both ends of a connected image. However, in this embodiment, the transparent polygons are not set for all the jagged portions of the upper and lower connected images, but are set only for the overlapping portions. Even if it is set to a part where there is no other connected image (a part where the connected images do not overlap), only a part of the image is deleted, and the paint contained in the connected image cannot be used effectively. is there.

When the process is started, the CPU reads the side line of the route area of the priority path (step S431). An example of processing is shown in the figure. It is assumed that a rectangular route region indicated by a thick line is set for the road texture Tx291 of the priority path. The root area can be set by the method described above with reference to FIG.
The side line refers to a boundary line in the direction along the path among the boundary lines of the route region. In the illustrated example, the CPU reads the side lines SL291 and SL292.

Next, the CPU selects a side line included in the route area of the lower path from the two side lines extracted in step S431 (step S432). An example of processing is shown in the figure.
In the figure, an upper road surface texture Tx291, a lower road surface texture Tx292, and a lower route area UR29 are shown. Of the two side lines SL291 and SL292 extracted from the priority path, the lower route area UR29 includes the side line SL292, which is selected.

The CPU sets the transparent polygon by offsetting the selected side line to the outside (step S433).
An example of processing is shown in the figure. The figure shows the upper road surface texture Tx291. In order to avoid complication of the drawing, the illustration of the lower road surface texture is omitted. The CPU can generate the transparent polygon TP29 by moving the selected side line SL292 outward as indicated by the arrow A29.
The amount of outward movement can be set arbitrarily. For example, the side line 292 and the upper road surface texture Tx291 may be translated until they no longer intersect. In this case, it is preferable to set the transparent polygon TP29 by moving it by a predetermined width after it has not crossed in order to ensure that the jagged portion is transmitted.
As another aspect, a method may be used in which the side line 292 is moved until it is covered with another paint or exceeds the boundary of the lower root area.

  By these methods, a transparent polygon can be set at a jagged portion. In image conversion for generating an orthographic image constituting a connected image, it is more susceptible to distortion as it goes to the side edge of the image. In addition, the jagged portion may divide the paint included in the lower linked image. If a jagged portion at the side edge is set as in the present embodiment, such a region can be removed relatively easily, and it is possible to effectively utilize the indication of the lower connected image.

(4) Transparent polygon combination processing:
FIG. 30 is an explanatory diagram showing the contents of the transparent polygon combining process. The transparent polygon combining process is a process corresponding to step S450 of the automatic transparent polygon generation process (FIG. 22).
FIG. 30A shows a lower road surface texture Tx302, an upper road surface texture Tx301, and transparent polygons TP301 to TP303. A plurality of transparent polygons may be set in this way. In this example, the transparent polygons TP301 and TP302 partially overlap each other.

When set in this way, the three transparent polygons TP301 to TP303 may be output as setting results.
However, if some of them overlap, the number of transparent polygons can be reduced by combining these transparent polygons as a single polygon, and the data capacity can be reduced.
FIG. 30B shows an example in which the transparent polygon combining process is performed. The transparent polygons TP301 and TP302 in FIG. 30A are combined to form one transparent polygon TP304. Since the polygons can be combined by various known methods, description thereof is omitted.

(5) Processing example:
FIG. 31 is an explanatory view showing an example of setting a transparent polygon.
FIG. 31A shows a concatenated image of the lower path NP31 and the upper path BP31. In the example of the figure, as shown in a region B31, a jagged portion at the road texture side end of the upper path BP310 appears, and the image quality of the road image is impaired. Moreover, as shown to area | region A31, the broken lane boundary line is parted by the upper jagged part.

FIG. 31B shows a setting example of the transparent polygon. The transparent polygon was set while being sandwiched between the lines L313 and L314 so as to pass through the upper jagged portion. Further, when the lower connected image overlaps with another connected image, a transparent polygon can be set in a portion sandwiched between the lines L311 and L312. In this embodiment, since the transparent polygon is set in the range where the upper and lower connected images overlap, no transparent polygon is set between the lines L311 and L312.
FIG. 31C shows a display example of the result of setting the transparent polygon. As can be seen from comparison with FIG. 31 (a), the jagged portion of the upper path disappears and the appearance of the road surface image is improved. Further, as a result of the jagged portion being transmitted through the divided lane boundary line, only the lane boundary line of the lower connected image is clearly displayed without being divided.

D. effect:
According to the road surface photographing system 100 and the road marking map generating device 200 of the embodiment described above, the road surface texture obtained by orthogonal transformation of the frame image acquired while traveling on the road is arranged, thereby driving. A connected image with high positional accuracy can be obtained along a trajectory (path). Furthermore, a road surface image of the entire road can be obtained by aligning and synthesizing connected images obtained along a plurality of paths. At this time, a road surface image is generated while ensuring the overall position accuracy by adopting a method that uses the path with the highest position accuracy of each path when taking an image as a reference path and aligning other paths with this reference path. can do.
In the present embodiment, the connected images of the respective paths are limited until the road surface texture is arranged, and these are not combined as a single image. Therefore, it is possible to easily synthesize a multi-pass connected image by translating the arrangement in units of road surface texture.

  In this embodiment, both the generation of the connected image of each path and the synthesis of the combined images of a plurality of paths do not need to perform affine transformation on the road surface texture, and are performed by simple parallel movement. Therefore, it is possible to avoid deterioration in image quality due to complicated image processing, and to obtain a road surface image displayed with a clear road marking. Further, since the translation is performed by parallel movement, a relative coordinate system based on the representative point in the road texture is maintained before and after the generation and synthesis of the connected images. As a result, if the absolute position coordinates of the representative point are obtained, the absolute position coordinates of each point in the road texture can be easily obtained, and the absolute position coordinates of the road marking can also be obtained.

  In this embodiment, by setting transparent polygons, it is possible to generate road images by effectively utilizing road markings (paints) included in the connected images of each path. Further, in this embodiment, since the transparent polygon can be automatically set, there is an advantage that the load on the operator can be reduced and the setting result of the transparent polygon can be obtained stably.

E. Variations:
E1. Determining the preferred path:
(1) Priority path determination method:
In the embodiment, a case where the operator designates a priority path, that is, a path to be arranged on the upper side when overlapping connected images is illustrated (see step S406 in FIG. 22). In the modification, an example in which a priority path is automatically determined is shown.

FIG. 32 is a flowchart of the priority path determination process. This process can be executed as a process replacing step S406 in FIG.
In this process, the CPU inputs the connected image of the processing target path and the paint recognition result (step S321). As the recognition result of paint, the result obtained by the processing shown in FIG. 19 can be used.
Next, the CPU recognizes a dust image for each path (step S322). A dust image refers to an image of a portion of a connected image where a foreign object other than a road marking is reflected, such as a stopped vehicle. A dust image recognition method will be described later.

The CPU determines a priority path based on the recognition result of the paint and dust image (step S323). In this embodiment, the priority path is determined according to the following four conditions.
Condition 1: A path having a larger dust image area in the overlapping region is determined as a lower side (a path having a smaller dust image area is set as a priority path);
Condition 2: A path with less paint in the overlapping area is determined to be the lower side (a path with much paint is a priority path);
Condition 3: A path having a larger dust image area in all regions is determined as a lower path (a path having a smaller dust image area is set as a priority path);
Condition 4: A path with less paint in all areas is determined as a lower path (a path with more paint is a priority path);

The above conditions are applied in order of priority of condition 1 to condition 4. That is, Condition 2 is applied when the priority path is not determined under Condition 1 (Step S324). The same applies to conditions 3 and 4 (step S324).
These conditions are determined with the following intention. First, the amount is evaluated by using the area as an evaluation value for a dust image and the number as an evaluation value for paint. Then, the evaluation (condition 1 and condition 2) in the region where the connected images overlap is prioritized over the evaluation of the entire connected image (condition 3 and condition 4). This is because the overlapping region has a larger influence on the generation of the transparent polygon and the appearance of the road image. In the evaluation of the overlapping area and the entire connected image, the evaluation of the dust image is given priority over the evaluation of the paint. This is because the garbage image has a greater influence on the appearance of the road image.

FIG. 33 is an explanatory diagram of an example of determining a priority path. The case where the connection image T33A for the path P33A and the connection image T33B for the path P33B in FIG. As shown in the figure on the left side, the two overlap in the overlapping region OL. In the connected image T33A, the dust image G33A is recognized in a portion that is out of the overlapping region. In the connected image T33B, the dust image G33B in the overlapping region and the dust image G33C in a portion outside the overlapping region are recognized.
For these images, first, a priority path is determined according to the above-mentioned “condition 1: a path having a larger dust image area in the overlapping region is determined as a lower side (a path having a smaller dust image area is set as a priority path)”. . In the overlapping area, the dust image area is larger in the path P33B because the dust image G33B exists. Therefore, as shown in the upper right, the connected image T33A is determined as the priority path, and the connected image T33B is placed below the connected image T33B. When overlapping in this way, as shown in the upper right figure, the garbage image G33B is hidden by the connected image T33A, and the appearance of the road image is improved.

On the other hand, the lower right diagram shows an example in which the path P33B on the side where the dust image area is large contrary to the condition 1 is set as the priority path. The connected image T33A is placed below the connected image T33B. In this case, as shown in the lower right figure, the garbage image G33B exists in the road image, and the appearance of the road image is impaired.
Assuming that the priority path could not be determined under conditions 1 and 2, "Condition 3: The path having the larger dust image area in all regions is determined as the lower side (the path having the smaller dust image area is set as the priority path). ”To determine the priority path.

In this case, the area of the dust image G33A in the connected image T33A is compared with the areas of the dust images G33B and G33C in the connected image T33B. If the area of the dust image G33A is larger, the linked image T33A is arranged on the lower side as shown in the lower right figure. If the areas of the dust images G33B and G33C are larger, the linked image T33B is arranged on the lower side as shown in the upper right figure.
The whole connected image is evaluated for the following reason. The part of the dust image outside the overlapping region of the connected images does not actually affect the appearance of the road image. However, in reality, garbage images are not completely recognizable. Therefore, a connected image having a large area of dust images recognized as a whole connected image includes a large amount of dust images that are not recognized. Condition 3 determines the priority path based on the amount of dust images of the entire connected image from this point of view.

(2) Reflecting dust image recognition results:
As described above, a dust image recognition result is used to determine the priority path. Since both the dust image and the paint recognition result are used as the priority path determination condition, it is preferable that the paint is recognized based on the dust image recognition result.
Therefore, in the following, first, a dust image will be exemplified, a method for reflecting a dust image recognition result in paint recognition, and then two specific dust image recognition methods will be described.

FIG. 34 is an explanatory view illustrating the reflection of a foreign object.
FIG. 34A illustrates a frame image. A frame A34 in the center in the figure is a part used for generating road surface texture (see FIG. 7). In this example, it is assumed that the vehicle is traveling in the lane on which the arrow P34c is drawn. In this frame, paint such as an arrow P34c and a lane boundary line P34b is shown, and a rear part P34a of a truck traveling in the right lane is shown as a foreign object.
FIG. 34B shows a connected image generated using a frame image in which a foreign object is reflected. The part in which the foreign object is reflected as shown in FIG. 34A is represented as a part that cannot be recognized as paint in the connected image, as shown in a region G34 in the figure. Hereinafter, the foreign matter reflected in the frame image or the foreign matter shown in the connected image is referred to as a dust image.

Next, a method of reflecting the dust image recognition result in the paint recognition will be described.
FIG. 35 is a flowchart showing a method of reflecting the recognition result of the dust image. The result of recognizing the dust image can be reflected by a pre-processing performed prior to paint extraction or by a post-processing performed after paint extraction.
In the flowchart in the figure, the reflection method in the case of pre-processing on the left side and post-processing on the right side is shown.
In executing these processes, it is assumed that a dust image in a frame image or a connected image is recognized separately by dust recognition processing (step S351), and the result is stored in dust image data. The contents of the dust recognition process (step S351) will be described later.
Here, an example is shown in which the dust recognition process is performed independently, but it may be incorporated in the process of pre-processing or post-processing.

In the case of preprocessing, the CPU reads a frame image (step S300A), and deletes the dust image from the frame image using the dust image data (step S301). By this process, for example, in the image shown in FIG. 34A, the reflection P34a at the rear of the track is deleted.
The CPU extracts paint according to the procedure described in the embodiment (see FIG. 19) using the frame image from which the dust image has been removed. That is, vertical arrangement processing is performed (step S302), cross-related paint is extracted (step S310), and various paints are extracted (step S350). Then, a relative coordinate conversion process (see FIG. 21) is performed (step S370).
According to the preprocessing method, the paint is extracted using the frame image from which the dust image is removed, so that erroneous recognition due to the dust image can be suppressed.

In the post-processing, the CPU reads the frame image (step S300A), and in accordance with the procedure described in the embodiment (FIG. 19), the vertical arrangement process (step S302), the crossing related paint extraction (step S310), and various paint extractions (step S350) and relative coordinate conversion processing (step S370) are performed. Since the processing so far is performed using a connected image including a dust image, misrecognition due to the dust image is also included.
Next, the CPU reads the dust image location using the dust image data (step S352). The presence position of the dust image is obtained by orthographically projecting the dust image recognized in the frame image (see FIG. 7) and performing alignment processing (see FIG. 8). . Since dust images are not used for paint extraction, vertical arrangement processing is unnecessary, and relative coordinate conversion processing for converting the result of vertical arrangement into a position in the absolute coordinate system is unnecessary.
When the dust image existence area is thus obtained, the CPU checks the positional relationship between the paint and the dust image existence area, and deletes the paint overlapping the existence area (step S353). This is because it is determined that such a paint has erroneously recognized a dust image.

According to the post-processing method, since the part of the recognized paint that overlaps with the dust image is removed, it is possible to prevent the paint that erroneously recognized the dust image from remaining.
In FIG. 35, the pre-processing and the post-processing have been described as separate independent processes. The reflection of the dust recognition result may be performed by either the pre-processing or the post-processing, or the post-processing may be further applied after performing the pre-processing.
Hereinafter, the details of the dust image recognition process in step S351 will be described in detail. As a dust image recognition method, there are an optical flow method and a gradation difference method.

(3) Dust image recognition processing (1) to optical flow:
FIG. 36 is a flowchart of the dust image recognition process (1). An example of processing using optical flow is shown.
The CPU first reads two pieces of frame data arranged at an interval of about 1 m from the photographing position (step S361). Frame images taken while traveling are obtained at distance intervals shorter than 1 m, but in order to reduce the processing load, frame images are thinned out at intervals of about 1 m and two consecutive images are read. did. The reason why it is set to about 1 m is that the frame images are not strictly taken at intervals of 1 m.
The CPU calculates the distance between the frames based on the position coordinates together with the reading of the frame image (step S362).

Next, the CPU extracts left and right inspection areas from each frame image (step S363). The outline of the inspection area is shown in the figure. The inspection areas are the left and right areas A36L and A36R excluding the central part A36C in the rectangular area used for generating the road surface texture in the frame image FL36. The reason for limiting to the area used for generating the road surface texture is that the recognition of the paint is not affected even if a dust image exists in the other area. The reason why the central portion A36C is excluded is that there are no other stopped vehicles or other foreign objects in the lane in which the vehicle is traveling. This is because, in the travel lane in which the host vehicle travels, it is possible to travel at a greater distance so that the vehicle traveling ahead does not enter the image creation range.
The inspection area is not obtained by analyzing the frame image, but may be a fixed area in the frame image. Since shooting is performed with the camera fixed to the vehicle, a portion of the frame image FL36 that can be used for texture is fixed, and a region A36C in front of the vehicle that does not require recognition of a dust image is also fixed. This is because that. The area A36C in front of the vehicle can be set arbitrarily, but can be set to be an area corresponding to the lane width in which the host vehicle travels, for example.

The CPU generates an optical flow based on the images of the left and right inspection areas (step S364). The optical flow is a feature point tracking vector indicating where a feature point existing in the inspection area of the previous frame image moves in the next frame image.
The magnitude of the optical flow also depends on the interval between two frame images used for generation. The CPU normalizes the optical flow to a value that does not depend on the frame image interval. In this example, as a normalization, a method of converting to a value corresponding to a value obtained from a frame image having an interval of 1 m is adopted. Normalization can be performed by the following method, for example.
(Vx, Vy) = (OFx / DIST, OFy / DIST);
Vx, Vy ... x, y components of the normalized movement vector;
OFx, OFy ... optical flow x and y components;
DIST: interval between two frame images;
The CPU obtains the following three types of evaluation values for each normalized movement amount vector obtained by normalizing each optical flow in the inspection region (step S365).
| V |: the size of the normalized movement amount vector;
Vx: x component of the normalized movement amount vector;
Vy: y component of the normalized movement amount vector;

Then, a point where any evaluation value becomes abnormal, that is, a point where any evaluation value exceeds a preset threshold range is extracted (step S366). For example, if the magnitude | V | of the normalized movement amount vector is smaller than a preset minimum value | V | min or larger than a maximum value | V | max, it is determined that there is an abnormality. . Similarly, the x component and the y component are also determined to be abnormal if they are smaller than the minimum values Vxmin and Vymin or larger than the maximum values Vxmax and Vymax.
The threshold range may be set as an allowable range of the absolute value of the evaluation value. In this case, it is determined that there is an abnormality when the magnitude | V | of the normalized movement amount vector exceeds a preset threshold value | V | th. Similarly, regarding the x component and the y component, if the absolute values | Vx | and | Vy | exceed the respective threshold values | Vx | th and | Vy | th, they are determined to be abnormal.

A threshold value for determining whether or not the evaluation value is abnormal can be arbitrarily set. For example, it is possible to obtain a normalized movement amount vector of a screen on which no foreign object is shown and set based on a range of evaluation values obtained therefrom. In addition, here, a uniform threshold value is used for all points in the inspection area, but different threshold values may be used for the right and left inspection areas. The threshold value may be changed for each portion obtained by further subdividing the inspection area.
In step S366, the point at which any one of the magnitude of the normalized movement amount vector, the x component, and the y component is determined to be abnormal is extracted, but the points where all the points are abnormal are extracted. Other evaluation methods can also be taken. Next, the CPU generates a convex polygon including the point determined to be abnormal, and stores this as a dust image (step S367).
The CPU repeatedly executes the above processing for all the frame data (step S368), and ends the dust image recognition processing.

FIG. 37 shows an example of an optical flow. The portions surrounded by solid lines on both sides of the area A37C in front of the traveling lane of the host vehicle are inspection areas A37L and A37R. The optical flow V37 is shown in the form of dots in the drawing. In this example, the optical flow V37 is obtained over the entire image regardless of the inspection areas A37L and A37R. Since no foreign matter is present in the inspection areas A37L and A37R, the optical flow is a vector indicating the amount and direction of the road marking to move in the screen as the vehicle travels. A movement of about 1 m is only a vector having a magnitude represented in a dot shape as illustrated.
After normalizing this optical flow, if the magnitude of the normalized movement amount vector, the distribution of the x component, and the y component are obtained, step S366 of the dust recognition process (FIG. 36) is performed based on the maximum value and the minimum value. It is possible to set a threshold value used in

FIG. 38 shows an example of an optical flow when there is a traveling vehicle. An example in which the vehicle C38 overtakes the lane on the left side of the host vehicle is shown. The optical flow V38 is shown for the inspection area on the left side. Since the feature point of the vehicle overtaking the own vehicle moves at a speed faster than the movement of the own vehicle, it moves forward with respect to the own vehicle, that is, in the upper right direction in the screen. Therefore, the optical flow V38 is a vector pointing in the upper right direction.
As a result, in the portion corresponding to the vehicle A38, the evaluation value (size, x component, y component) of the optical flow V38 exceeds a preset threshold value and is determined to be abnormal.
The dust image A38 is recognized by generating a convex polygon including a point determined to be abnormal. The upper and left sides of the vehicle C38 are not included in the dust image A38 because these portions are outside the inspection area. A curve L38 in the dust image A38 represents a classification according to which evaluation value has exceeded the threshold among the magnitude of the normalized movement amount vector, the x component, and the y component. In the modified example, if any evaluation value exceeds the threshold value, all the images are determined as dust images, and therefore the curve L38 has no particular meaning.

FIG. 39 is an example of an optical flow when there is a stopped vehicle. The example in case the vehicle C39 has stopped in the lane of the left side of the own vehicle was shown. An optical flow V39 is shown for the inspection area on the left side. The stopped vehicle C39 is not different from the road marking in that the moving speed is 0, but is different from the road marking, and is a three-dimensional figure. Therefore, an optical flow different from the road marking is obtained in the frame image. The overtaking (FIG. 38) provides an optical flow that is diagonally upward to the right on the whole, whereas an optical flow that is diagonally downward to the left is obtained on the right side of the vehicle on the stopped vehicle, and leftward on the rear side of the vehicle. Other complex optical flows.
The result of normalizing these optical flows V39 and generating convex polygons including abnormal points exceeding the threshold range is the dust image A39. In this example, the entire vehicle C39 cannot be recognized as a dust image, but it can also be recognized as a dust image by adjusting the threshold range.

(4) Dust image recognition processing (2)-gradation difference:
FIG. 40 is a flowchart of the dust image recognition process (2). An example of processing using gradation difference is shown.
First, the CPU extracts an overlapping area of consecutive texture images in the connected image (step S501). An example of processing is shown in the figure. As shown on the left side, attention is paid to adjacent textures T401 and T402 in the connected image. Both overlap in the hatched part. Therefore, as shown on the right side, the CPU cuts out a region A401 overlapping the texture T402 from the texture T401. Similarly, a region A401 is cut out from the texture T402.

Next, the CPU applies a smoothing filter to blur the image of the overlapping area (step S502). Any smoothing filter can be applied.
The reason for smoothing is as follows. In this dust recognition process, as will be described later, it is determined whether or not the image is a dust image based on the gradation difference for each pixel in the overlapping region. This is because road markings should overlap each other in the overlapping area, and a gradation difference should occur only in the dust image portion. However, since various errors are actually included at the time of photographing or in the processing process, the position of the road marking does not always coincide completely in the overlapping region. If the gradation difference is calculated for each pixel in this state, the road marking portion may be erroneously recognized as a dust image. Therefore, in step S502, a smoothing filter is applied to the image of the overlapping area so that a large gradation difference does not occur even when the position of the road marking is shifted in the overlapping area.

Using the image thus smoothed, the CPU obtains a gradation difference between the overlapping areas, and extracts pixels whose gradation difference exceeds a threshold value (step S503).
An example of processing is shown in the figure. It is assumed that the color component of the pixel Pxa of the overlapping image A401 is expressed as (Ha, Sa, Va) in the H (hue) S (saturation) V (lightness) system. It is assumed that the color component of the pixel Pxb corresponding to the overlap image A402 is represented as (Hb, Sb, Vb).
Then, a pixel whose gradation difference exceeds the threshold is extracted.
For the hue, pixels satisfying | Ha−Hb |> threshold THH are extracted.
For saturation, pixels satisfying | Sa−Sb |> threshold THS are extracted.
For brightness, pixels satisfying | Va−Vb |> threshold THV are extracted.
Here, a method of extracting pixels whose brightness difference exceeds a threshold value is adopted. This is because white or yellow is used for the road marking, so that the lightness gradation difference is most prominent between the road marking and other dust images. Of course, pixels whose tone difference regarding hue and saturation exceeds a threshold value may also be extracted. Further, it is possible to take a method of extracting only pixels that exceed a threshold value in all of lightness, saturation, and hue.

The CPU combines the extracted pixels to specify a dust image area (step S504).
The bonding can be performed, for example, by the following method. First, adjacent portions of pixels extracted as dust images are combined. Next, each pixel is thickened by predetermined pixels vertically and horizontally. As a result of this fattening, if there is a combined pixel, that portion is also recognized as one polygon. By doing so, the CPU can generate a dust image area. A convex polygon may be formed by filling a concave portion of a polygon obtained by combining pixels.
Finally, the CPU deletes a region having a minute area that is equal to or smaller than the threshold (step S505). This is because a stopped vehicle or other foreign object is normally recognized as a large dust image, and even if a small area dust image is left, it does not have a significant adverse effect on the recognition of paint. However, this process can be omitted.

FIG. 41 is an explanatory diagram showing an example of dust image recognition. 41 (a) and 41 (b) are overlapping regions cut out from two textures. In both cases, dust images G412 and G413 are shown on the right side. In FIG. 41A, another dust image G411 is shown.
41 (c) and 41 (d) show the results of applying a smoothing filter to FIGS. 41 (a) and 41 (b), respectively. It can be seen that the contours of road markings are blurred compared to the original data. 41 (e) and 41 (f) show the results of obtaining the gradation differences between FIGS. 41 (c) and 41 (d), respectively. The range where the lightness gradation difference exceeds the threshold value THV is dust images G414 to G417 indicated by solid lines. The dust images G414, G416, and G415, G417 are the same. Here, they are merely shown superimposed on each other so that the comparison with FIG. 41 (c) and FIG. 41 (d) can be seen.

For example, when only FIG. 41 (f) is viewed, the portion of the dust image G416 appears as an appropriate road marking. However, since there is the dust image G414 of FIG. 41E superimposed on this, the gradation difference is an abnormal value exceeding the threshold value.
On the other hand, the dust images G415 and G417 are reflections of foreign matters. In this example, the entire reflection of the foreign object cannot be recognized as a dust image, but the entire image can be recognized depending on the adjustment of the threshold value.
FIG. 41 (g) shows the recognition result for another texture. As a result of obtaining the gradation difference by the above method, the dust image G418 is recognized. However, since these dust images G418 are very small, the CPU excludes them from the dust images (see step S505 in FIG. 40).

If the priority path determination method shown in the modification is used, the number of transparent polygons can be suppressed or the shape can be simplified by determining the vertical relationship in consideration of the amount of paint. Further, by determining the vertical relationship in consideration of the amount of dust, the appearance of the road image after setting the transparent polygon can be improved.
Further, in the above-described method, since the vertical relationship of the path is automatically determined, the subjectivity of the operator does not enter, and instability that the appearance of the road image differs depending on the operator can be suppressed.

E2. Automatic transparency polygon generation processing (1):
In the embodiment, an example is shown in which a transparent polygon is generated at a place where paint exists (see FIG. 25).
FIG. 42 is an explanatory diagram showing an example of setting a transparent polygon by the method of the embodiment. In the example, the connection image T42B corresponding to the path P42B is superimposed on the connection image T42A corresponding to the path P42A. In the area A42, the paint PT42 indicating the speed limit (50 km / hour) appears by generating the transparent polygon TP42. As described above, by generating a transparent polygon in a portion where paint is present, it is possible to utilize the paint included in the lower connected image as a road image.
However, in this method, as shown in the region B42, a dust image in which a foreign object appears in the road image may remain. The presence of such a garbage image impairs the appearance of the road image.
In the modified example, in order to improve the appearance of the road image, a method of erasing the dust image shown in the region B42 with a transparent polygon is shown. Such a transparent polygon for erasing a dust image is called a dust image erasing polygon.
The following process is performed after the paint transparent polygon setting process (step S410), the jagged cut process (step S430), and the transparent polygon combination process (step S450) in the automatic transparent polygon generation process (FIG. 22). Preferably it is done.

FIG. 43 is a flowchart (1) of the dust image erasure polygon generation process. The CPU reads the recognition result of the dust image in the upper path from the dust image data (step S601). The dust image data is data storing a dust image existing portion recognized by a method such as optical flow and gradation difference (FIGS. 36 to 41).
The CPU polygonizes each dust image based on the dust image data (step S602). This polygon is hereinafter referred to as a garbage polygon. An example of the garbage polygon GP431 is shown in the figure.

Next, the CPU groups dust polygons in the vicinity and generates a dust polygon including the dust polygons in the same group (step S603).
Grouping can be performed by the following procedure. First, dust polygons to be processed are extracted, and the nearest dust polygon is extracted. If the distance between the dust polygons is equal to or less than a predetermined value, the same group ID is assigned to both. If a group ID has already been assigned to any dust polygon, this group ID may be used in common. On the other hand, if the predetermined value is exceeded, a group ID is assigned to the target dust polygon alone. The above processing may be repeatedly executed until a group ID is assigned to all dust polygons.
In the example in the figure, the dust polygon GP431 indicated by a broken line shows a state in which the distance to the nearest dust polygon is equal to or less than a predetermined value. As a result, the same group ID is sequentially assigned to these dust polygons. Further, the dust polygon GP432 is grouped independently because the distance d43 to the nearest dust polygon exceeds a predetermined value.
When the grouping is completed in this way, the CPU generates a dust polygon GP433 including the dust polygon GP431 existing in the same group.

Next, the CPU reads the route area of the lower path (step S604). As described with reference to FIG. 24, the root region is a region in the connected image excluding the jagged portions at both ends, that is, a region that is in a smooth state in which there is no portion that is sharply bent at the edge line in the direction along the path. Say.
FIG. 44 is a flowchart (2) of the dust image erasure polygon generation process.
The CPU extracts an overlapping portion between the dust polygon generated by the above process and the root area (step S605).

An example of processing is shown in the figure. The dust polygon GP432 is extracted as it is because it is entirely included in the route region R43. Since the dust polygon GP433 partially protrudes from the root area RT43, a part in the root area RT43 (the hatched part in the figure) is extracted. The dust polygon GP434 is excluded because it entirely protrudes from the route region RT43.
In this way, if the extracted dust polygon is used, the transparent polygon can be generated only in the portion where the route area exists on the lower side, so that the road image appears to have a hole. Can be avoided.

Next, the CPU combines the jagged cutting transparent polygon and the dust polygon to correct the shape (step S606).
An example of processing is shown in the figure. It is assumed that the transparent polygon TP29 is set for the connected image Tx43 by the jagged cut processing shown in FIG. In this state, it is assumed that a dust polygon GP435 indicated by a broken line is obtained. In the processing of step S606, the dust polygon GP435 is projected onto the transparent polygon TP29. The dust polygon GP435 may obtain points P431 and P432 having a maximum width in the direction along the transparent polygon TP29, and a perpendicular line may be placed on the transparent polygon TP29 therefrom.
In this way, the CPU combines the dust polygon GP435 and the transparent polygon TP29 and shapes the dust polygon GP436 into a smooth shape by thinning out the constituent points P433 forming the recesses of the dust polygon GP435. obtain.
This processing may be applied only when the distance between the dust polygon GP435 and the transparent polygon TP29 is equal to or less than a predetermined value.

If the transparent polygon is generated by the above processing, the dust image can be erased by the transparent polygon, and the appearance of the road image can be improved.
Further, in the modified method, dust polygons including nearby dust polygons are generated, so that dust images that cannot be recognized by the dust image recognition process can be erased collectively. Furthermore, since the dust polygon generated based on the dust image and the transparent polygon generated by the jagged cut processing are combined, it is possible to erase the dust images existing in the gap between the two.

E3. Automatic transparency polygon generation process (2): Next, another aspect of erasing a dust image with a transparency polygon will be described.
In the processing shown in FIGS. 43 and 44, a method of recognizing a dust image and generating a transparent polygon there is adopted. On the other hand, in the mode shown below, a method is adopted in which unnecessary portions other than the paint in the upper connected image are erased with the transparent polygons without determining whether or not they are dust images.
FIG. 45 is an explanatory diagram showing a processing example of the dust image erasure polygon generation processing (2).
The left half of the connected image of the path PS45 from the start point S45 to the end point E45 is shown. However, in order to avoid complication of the figure, only a part of the texture Tx45 constituting the connected image is shown, and the other parts are not shown. In the figure, hatched portions represent paint such as lane boundary lines. A point G45 represents a dust image.

In this process, first, perpendicular lines L45 [1] to L45 [18] with respect to the path PS45 are generated at equal intervals from the start point S45 to the end point E45. The perpendicular interval d45 can be set to an arbitrary value such as an interval of 0.5 m. The interval may be given by dividing the path PS45 by a predetermined number.
The CPU generates constituent points of the transparent polygon in the order of the perpendicular lines L45 [1] to L45 [18] by the following method. First, paint is searched for as indicated by an arrow A45 from the side end side of the connected image with respect to the perpendicular line L45 [1]. When there is no paint as shown by the perpendicular line L45 [1], a transparent polygon constituent point P45 [1] is generated at a point separated from the path PS45 by a predetermined distance OS451. Similarly, the point P45 [2] is generated for the perpendicular L45 [2].
The predetermined distance OS451 can also be set arbitrarily. In this example, the predetermined distance OS451 is set to 1.5 m based on the half width of the road lane. This is because the lane including the path PS45 is the lane that was traveled when the connected image was captured, and it is considered that no foreign matter exists.

Paint exists in the perpendicular line L45 [3]. In this case, the constituent point P45 [3] is generated at a position away from the paint by a predetermined distance OS452. The distance OS452 is a distance set to an arbitrary value in order to avoid the transparent polygon from being applied to the paint. In this example, the distance OS452 is set to 0.2 m. Since the paint exists for the vertical line L45 [4], the point P45 [4] is generated similarly.
There is no paint on the vertical line L45 [5]. However, paint is present on the immediately preceding perpendicular L45 [4], and the composing point P45 [4] is set at a position relatively distant from the path PS45. Therefore, if the composing point is generated at the position of the distance OS451 from the path PS45 as in the case of the perpendicular line L45 [1], the transparent polygon may have an irregular shape. Therefore, when paint is present on the immediately preceding vertical line L45 [4], the constituent point P45 [5] is located at a position as far away from the path PS45 as the constituent point P45 [4] on the vertical line L45 [4]. ] To be generated.

There are two paints on the perpendicular lines L45 [6] and L45 [7], but since the search is performed in the direction of the arrow A45, the configuration points P45 [6] and P45 [7] are based on the side end side paint. Is generated. Thereafter, the configuration points P45 [8] to P45 [18] are set similarly.
By connecting the constituent points P45 [1] to P45 [18] obtained in this way, and connecting both ends thereof with a line outside the transparent polygon obtained by the jagged cut processing (FIG. 29), a transparent polygon POL45 is obtained. .
In this example, in order to avoid the boundary line of the transparent polygon POL45 from intersecting with the paint, the boundary line is constituted by a line parallel or perpendicular to the path PS45. When the boundary line and the paint intersect with each other while allowing an oblique boundary line, a method of correcting the shape of the boundary line so as to avoid painting may be adopted.

FIG. 46 is a flowchart of the dust image erasure polygon generation process (2). This is a process for generating the transparent polygon described with reference to FIG. This process is also performed after the paint transparent polygon setting process (step S410), the jagged cut process (step S430), and the transparent polygon combination process (step S450) in the automatic transparent polygon generation process (FIG. 22). Is preferred.
The CPU reads the upper path paint from the paint recognition result (see FIG. 19) (step S660). Then, a perpendicular line that divides the path into equal intervals is generated (FIG. S661). This is processing for generating the perpendicular L45 shown in FIG.

The CPU searches for the intersection with the paint from the side end side of the connected image for the perpendicular to be processed (step S662). If there is an intersection, as indicated by the perpendicular line L45 [3] in FIG. 45, the point offset from the intersection by the distance OS452 is stored as a constituent point.
If there is no intersection (step S663), it is determined whether there is an intersection with the paint on the adjacent perpendicular (step S665). When there is an intersection point on the adjacent perpendicular line L45 [4] as in the perpendicular line L45 [5] in FIG. 45, a point having the same offset as the constituent point on the adjacent perpendicular line is stored as the constituent point. If there is no intersection with paint on the adjacent vertical line as shown by the vertical line L45 [1] in FIG. 45, a point offset from the path by the distance OS451 is stored as a constituent point (step S667).

The CPU repeatedly executes the processes in steps S662 to S667 until there is no perpendicular to be processed (step S668). Then, a transparent polygon including a texture outside the obtained constituent points is generated (step S669). Specifically, as described with reference to FIG. 45, both ends of the line connecting the constituent points may be connected to the line outside the transparent polygon obtained by the jagged cut processing (FIG. 29).
FIG. 47 is an explanatory diagram showing an application example of the automatic transparency polygon generation process (2).
FIG. 47A shows a state before the transparent polygon is generated. It is assumed that the connected image of the path P472 is arranged on the connected image of the path P471. The paint PT47 is recognized in the connected image of the path P472. In this state, when a transparent polygon is generated by the method described with reference to FIGS. 45 and 46, a transparent polygon POL47 that covers the road side end side of the paint PT47 is generated.

FIG. 47 (b) shows a state where the transparent polygon POL47 is applied. In the area A47, the dust image is erased by the transparent polygon POL47, and the connected image corresponding to the path P471 appears. There is no foreign matter and no dust image in the lane that was traveled when the connected image was generated on the path P471. Therefore, by allowing the connected image of the path P471 to appear, the dust image can be deleted, and the appearance of the road image can be improved.
In the processing described above, it is possible to generate the transparent polygon and erase the dust image with a very light load without recognizing the dust image in the connected image.
When this processing is applied, for the upper connected image, transparent polygons are set so that unnecessary portions are erased all together while leaving paint, so that the connected image with many dust images is It is preferable that a connected image with few dust images is arranged on the lower side represented by the transparent polygon. Therefore, it is preferable to change the conditions 1 and 3 of the priority path determination process (FIG. 32) described above as follows.

Condition 1: A path having a smaller dust image area in the overlapping region is determined as a lower side (a path having a larger dust image area is set as a priority path);
Condition 3: A path having a smaller dust image area in all regions is determined as a lower path (a path having a larger dust image area is set as a priority path);
Although various embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and it goes without saying that various configurations can be adopted without departing from the spirit of the present invention.
For example, the connected image may be generated as a single image obtained by combining road textures. In this case, when synthesizing a plurality of paths, the connected image may be divided into a plurality of regions corresponding to the road texture, and then translated for each region.
In this embodiment, an example of using an image taken with a video camera mounted on a vehicle is shown. However, not only a vehicle but also various other moving bodies can be used, and a method of taking a picture while walking is adopted. Also good.

In the embodiment, an example in which images obtained by traveling at different positions at the same time are combined is shown. The present embodiment can also be used in a mode in which images obtained at different times are compared to generate a road image reflecting a change applied to the road. Even in this case, the method for generating a connected image and the method for generating a road image by combining them are the same as in the embodiment.
When setting a transparent polygon, the same method as in the embodiment can be applied. However, it is preferable to determine the priority path based on the time when the image is acquired. The newly acquired side is set as the priority path. This is because it is preferable to give priority to the newly photographed image in consideration of the purpose of reflecting the change in the paint on the road surface in the road image. By doing so, when a paint that has not existed before is newly applied, it is possible to reflect it on the road image without setting a transparent polygon.

  In the embodiment, when the paint exists only in the lower connected image, the transparent polygon is set as the setting target (steps S413 and S415 in FIG. 25). However, in this aspect, the transparent polygon is excluded from the setting target. Is preferred. That is, when “No” determination is made in step S413 of FIG. 25, it is preferable to shift the processing to step S416. If the previously drawn paint is deleted, the paint will be present only on the lower (older) connected image, and no paint will be present on the upper (newly acquired side) By removing such an area from the setting target of the transparent polygon, such deletion of paint can be reflected without any trouble.

DESCRIPTION OF SYMBOLS 100 ... Road surface imaging | photography system 110 ... Position measurement part 110 ... Measurement data 112 ... Controller 114 ... GPS
114A ... Antenna 116 ... IMU
118 ... DMI
DESCRIPTION OF SYMBOLS 120 ... Video camera 130 ... Recording apparatus 140 ... Hard disk 142 ... Image data 144 ... Synchronization data 146 ... Measurement data 150 ... Base station data 200 ... Road marking map generator 201 ... Main control part 202 ... Command input part 203 ... Display control part 204 ... Data input unit 205 ... Track data calculation unit 206 ... Image conversion unit 207 ... 1-pass image synthesis unit 210a ... Track data 210b ... Road surface track data 210c ... Road surface texture 210d ... Linked image 210e ... Road image 210f ... Road image registration data 210g ... Registration data for trajectory 210c ... Data (road surface texture 210 ... Processing data storage unit 220 ... Alignment processing unit 221 ... Transparent polygon setting unit

Claims (18)

A road marking map generation method for generating a road marking map including a marking given to a road surface by a computer ,
(A) The computer includes image data composed of a plurality of frame images obtained by capturing the road surface on which the sign is given while moving on the imaging target road a plurality of times, and each frame image in the image data Obtaining position coordinate data representing a shooting position;
(B) The computer converts each frame image constituting the acquired image data to obtain an orthographic image in a state where the road surface is viewed from directly above;
(C) the computer, on the basis of the orthoimage in the position data, by placing on the path a moving locus when you move the photographed road, connected image representing the road surface of the path Generating
A step; (d) computer, which on the basis of the position data, superimposed the combined images on the path corresponding to the plurality of lanes constituting one of the road to generate a composite image before Symbol road surface,
(F) the computer, position and shape of the transparent polygon for by transmitting an upper connected image displays the connected image in the lower side in the overlapping region is the connection between images, determined according to the connected image A road marking map generation method comprising the step of generating a road marking map. A road marking map generation method according to claim 1,
In the step (f), the road marking map generating method in which the computer generates the transparent polygon within a range where the lower connected image exists. A road marking map generation method according to claim 1 or 2,
In the step (f), the computer, the road marking map generating method for generating the position and shape of the transparent polygon based on the indication contained in the connected image. A road marking map generation method according to claim 3,
In the step (f), the computer includes the transparent polygon in a portion where the sign included in the lower connected image has a higher evaluation value indicating the accuracy than the sign included in the upper connected image. A road marking map generation method for generating A road marking map generation method according to claim 3,
In the step (f), the road marking map generation method in which the computer generates the transparent polygon in a portion where the marking is included only in a lower connected image. The road marking map generation method according to any one of claims 3 to 5, further comprising:
(E) said computer, wherein for each connected image by image processing, has a step of recognizing the indication,
In the step (f), the road marking map generation method in which the computer generates the transparent polygon based on the recognition result. A road marking map generation method according to any one of claims 1 to 5,
In the step (f), the computer, the road marking map generating method for generating a transparent polygon so as to cover a predetermined range of the side end of the connection image of the upper. A road marking map generation method according to any one of claims 1 to 7,
The step (d)
(D1) the computer specifying a corresponding corresponding point in an area that is photographed in common in a connected image of two or more paths among a plurality of paths constituting one road ;
(D2) the computer sets one of the plurality of paths as a reference path;
(D3) The computer translates for each predetermined region in the connected image constituting a path other than the reference path, based on the movement vector set so that the positions of the corresponding corresponding points match, road marking map generated how having a generating a composite image of the road surface spanning the plurality of paths. The road marking map generation method according to any one of claims 1 to 8, further comprising:
(G) the computer, the step (f) of the plurality of transparent polygons generated in, road marking map generated how having a step of attaching those mutually overlapping. A road marking map generation method according to any one of claims 1 to 9,
The step (d)
(D4) Before the generation of the composite image of the road surface , the computer performs at least the amount of the sign in the connected image and the amount of foreign matter other than the sign by image processing using the connected image or the frame image. Detecting one,
(D5) the computer, the detection result on the basis of the road surface marking map generated how having a the step of determining whether to place the orthoimage on any path on the upper side. A road marking map generation method according to claim 10,
In the step (d4), the computer detects both the amount of the sign and the amount of the foreign matter,
In the step (d5), the computer, the road marking map generating method for determining the placement of the orthophoto image based on the amount of the indicia when the arrangement can not be determined in the orthoimage based on the amount of the foreign substance . A road marking map generating method according to claim 10 or 11,
In the step (d4), the computer calculates an amount in the entire connected image and an amount in a region where the connected images overlap, for at least one of the sign and the foreign matter,
In the step (d5), the computer determines the arrangement of the orthographic image by preferentially evaluating the amount in the region where the connected images overlap each other than the amount in the entire connected image. Marking map generation method. A road marking map generation method according to any one of claims 10 to 12,
The step (f)
The step of extracting a portion in which the foreign object is reflected in the upper connected image;
The computer, road marking map generated how having a generating a transparent polygons covering a region corresponding to at least a portion of the foreign substance. A road marking map generation method according to claim 13,
The step (f)
(F1) the computer generating the transparent polygon so as to cover a predetermined range of a side edge of the upper connected image;
(F2) said computer, at least a portion transparent polygons covering the road surface marking that having a the step of integrating to fill the gap between the generated transparent polygon in the step (f1) of the foreign substance Map generation method. A road marking map generation method according to any one of claims 1 to 12,
The step (f)
(F3) the computer extracting a sign included in the upper connected image;
(F4) the computer, the extracted the upper indicia was in a range not fogged other indicia, that having a generating a transparent polygons covering the side end side of the connection image than the indication Road marking map generation method. A road marking map generation method according to claim 15,
The step (f) further includes
(F5) In a portion where the sign does not exist in the connected image of the road surface along the path, the computer displays the connected image in a range excluding a portion within a predetermined distance from the path in a direction intersecting the path. road marking map generated how having a step of generating a transparent polygons covering. A road marking map generating device for generating a road marking map including a marking applied to a road surface,
Image data composed of a plurality of frame images obtained by photographing the road surface on which the sign is given while moving on a road to be imaged a plurality of times, and position coordinate data representing a shooting position of each frame image in the image data An acquisition unit for acquiring
An image conversion unit that converts each frame image constituting the acquired image data to obtain an orthographic image in a state in which the road surface is viewed from directly above;
A connected image generation unit that generates a connected image representing a road surface of the path by arranging the orthographic image on a path that is a movement locus when the photographed road is moved based on the position coordinate data. When,
Based on the position data, it superimposed the combined images on the path corresponding to a plurality of lanes constituting one of the roads, and the composite image generation unit for generating a composite image before Symbol road surface,
Transparent polygon generation for generating a transparent polygon for transmitting the upper connected image and displaying the lower connected image in a region where the connected images overlap with each other at a position and shape determined according to the connected image A road marking map generating device comprising a section. A computer program for generating a road marking map including a marking applied to a road surface,
Image data composed of a plurality of frame images obtained by photographing the road surface on which the sign is given while moving on a road to be imaged a plurality of times, and position coordinate data representing a shooting position of each frame image in the image data With the steps to get
A procedure for converting each frame image constituting the acquired image data and executing a process of obtaining an orthographic image in a state where the road surface is viewed from directly above;
Based on the position coordinate data, the orthographic image is arranged on a path that is a movement locus when the photographed road is moved, thereby executing a process of generating a connected image representing the road surface of the path. Procedure and
A step of executing a process based on said position data, superimposed the combined images on the path corresponding to a plurality of lanes constituting one of the road to generate a composite image before Symbol road surface,
A process of generating a transparent polygon for transmitting the upper connected image and displaying the lower connected image in a region where the connected images overlap with each other at a position and a shape determined according to the connected image is executed. A computer program for causing a computer to execute the procedure .