JP7115869B2 - Map generation system - Google Patents

Description

The present invention relates to map generation systems.

For example, Patent Literature 1 describes a system in which a map generation device side generates a map based on an image captured by an imaging unit mounted on a vehicle. More specifically, the in-vehicle device transmits the image captured by the imaging unit and the travel locus of the vehicle to the map generation device. The map generation device generates a map by pasting images along the received travel locus.

JP-A-2009-237901

In the above system, the imaging unit captures images at a constant cycle (frame rate) when capturing moving images. When generating a map based on such images, for example, depending on the speed of the vehicle, gaps may occur when the images are pasted together, or multiple redundant images with most of the images overlapping may be generated by the map generating apparatus. or sent to Therefore, in this technical field, there is a demand for efficient transmission of an image captured by an imaging unit from an in-vehicle device to a map generation device.

A map generation system according to the present invention is a map generation system that includes an in-vehicle device that is mounted on a vehicle and that captures an image of the surroundings of the vehicle, and a map generation device that generates a map based on the image captured by the in-vehicle device. The in-vehicle device includes an imaging unit that captures images of the surroundings of the vehicle at a predetermined imaging cycle, a driving condition acquisition unit that acquires the driving conditions of the vehicle, a trajectory recognition unit that recognizes the driving trajectory of the vehicle, and a condition setting unit for setting extraction conditions for images captured by the imaging unit; an image extraction unit for extracting images satisfying the extraction conditions from the images captured by the imaging unit; to the map generating device through communication, the map generating device includes a receiving unit that receives the traveling locus and the image transmitted from the transmitting unit through communication, and the received traveling locus and the image: and a map generator for generating a map based on the map.

According to the present invention, an image captured by an imaging unit can be efficiently transmitted from an in-vehicle device to a map generation device.

1 is a diagram showing a schematic configuration of a map generation system according to a first embodiment; FIG. FIG. 2(a) is a diagram showing a camera image and a road surface equivalent area. FIG. 2(b) is a diagram showing a cut-out image of a road surface. FIG. 2(c) is a diagram showing a road surface image. FIG. 2(d) is a diagram showing a road surface orthoimage. FIGS. 3(a) and 3(b) are diagrams showing the overlap of road surface images depending on the speed of the vehicle. It is a graph which shows the change of an overlap rate. 4 is a flow chart showing the flow of image transmission processing performed in an in-vehicle device. It is a figure which shows schematic structure of the map generation system which concerns on 2nd Embodiment. 4 is a flow chart showing the flow of image transmission processing performed in an in-vehicle device. FIG. 10 is a diagram showing a modified exposure pattern; FIG. 11 is a diagram showing a schematic configuration of a map generation system according to a third embodiment; FIG. 4 is a flow chart showing the flow of image transmission processing performed in an in-vehicle device. FIG. 12 is a diagram showing a schematic configuration of a map generation system according to a fourth embodiment; FIG. FIG. 10 is a diagram showing how a road surface image is pasted to generate a road surface orthoimage. 4 is a flow chart showing the flow of image transmission processing performed in an in-vehicle device.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant description is omitted.

(First embodiment)
A first embodiment of the map generation system will be described. A map generation system 1 shown in FIG. 1 is a system that generates a map based on an image captured by a camera 101 mounted on a vehicle V (hereinafter referred to as a “camera image”). The map generation system 1 includes an in-vehicle device 100 and a map generation device 200 .

Here, an overview of the map generation processing performed by the map generation system 1 will be described. The in-vehicle device 100 cuts out a road surface clipped image C including the road surface of the travel path L as shown in FIG. 2(b) from the camera image A captured by the camera 101 shown in FIG. 2(a). When receiving the road surface cutout image C from the in-vehicle device 100, the map generation device 200 performs bird's eye conversion on the road surface cutout image C to generate a road surface image D as shown in FIG. 2(c). The map generating device 200 generates a road surface orthoimage E as a map by pasting the road surface images D in sequence, as shown in FIG. 2(d). The configuration of each part of the map generation system 1 will be described in detail below.

(details of in-vehicle equipment)
The in-vehicle device 100 is mounted in a vehicle V. As shown in FIG. The in-vehicle device 100 includes a camera (imaging unit) 101 , a vehicle sensor (driving condition acquisition unit) 102 , a GPS receiving unit 103 , an ECU 104 and a transmitting unit 105 .

The camera 101 is imaging equipment that captures an image of the external environment of the vehicle V. As shown in FIG. The camera 101 is provided inside the windshield of the vehicle V. As shown in FIG. The camera 101 images the front of the vehicle V at a predetermined imaging cycle (frame rate). In addition, the camera 101 captures an image of the front of the vehicle V so as to include the road surface of the road on which the vehicle V travels. The camera 101 outputs the captured camera image to the ECU 104 .

The vehicle sensor 102 is a sensor that acquires the running condition of the vehicle V. FIG. In this embodiment, the vehicle sensor 102 detects the speed of the vehicle V as the driving condition. That is, vehicle sensor 102 includes a vehicle speed sensor that detects the speed of vehicle V. FIG. As the vehicle speed sensor, for example, a wheel speed sensor that is provided for a wheel of the vehicle V or a drive shaft that rotates integrally with the wheel and that detects the rotational speed of the wheel is used. Vehicle sensor 102 outputs detected vehicle speed information to ECU 104 .

GPS receiver 103 measures the position of vehicle V (for example, the latitude and longitude of vehicle V) by receiving signals from three or more GPS satellites. The GPS receiver 103 outputs the measured positional information of the vehicle V to the ECU 104 .

The ECU 104 is an electronic control unit having a CPU [Central Processing Unit], a ROM [Read Only Memory], a RAM [Random Access Memory], and the like. The ECU 104 implements various functions by, for example, loading programs stored in the ROM into the RAM and executing the programs loaded into the RAM by the CPU. The ECU 104 may be composed of a plurality of electronic units. The ECU 104 functionally includes an image extraction unit 111 , an extraction cycle setting unit (condition setting unit) 112 , and a trajectory recognition unit 113 .

As shown in FIG. 2(a), the image extracting unit 111 extracts the camera images A that meet the extraction condition set by the extraction cycle setting unit 112, among a plurality of camera images A captured by the camera 101 at a predetermined image capturing cycle. Image A is extracted sequentially. Details of the extraction conditions will be described later. The image extraction unit 111 extracts the camera image A based on the extraction condition each time the camera image A is captured at a predetermined imaging cycle. Hereinafter, when it is necessary to distinguish between camera images A to be extracted and camera images A not to be extracted among a plurality of camera images A captured by the camera 101, camera images A to be extracted, that is, camera images satisfying the extraction conditions. A is called an extracted camera image A1.

In addition, as shown in FIG. 2(b), the image extracting unit 111 extracts a portion of the extracted camera image A1 corresponding to the road surface area B (see FIG. 2(a)) set in front of the vehicle V. is clipped (extracted) to generate a road clipped image C. In addition, in the images shown in FIGS. 2A and 2B, the road L on which the vehicle V travels and the white lines L1 attached to both sides of the road L are shown.

Here, the road surface equivalent area B is a preset area in front of the vehicle V. As shown in FIG. For example, the road surface equivalent area B is a range of 8 m to 16 m in front of the vehicle V from the installation position of the camera 101 (that is, an area with a depth of 8 m) and a range of ±3.5 m on the left and right (that is, an area with a width of 7 m). is set. The mounting direction of the camera 101 is set so that the camera image A includes the road surface equivalent area B. As shown in FIG. Assuming that the road surface of the traveling road L is captured in the portion corresponding to the road surface equivalent area B in the extracted camera image A1, the image extraction unit 111 cuts out the image of the portion corresponding to the road surface equivalent area B to obtain the road crossing. Generate an output image C.

The image extraction unit 111 outputs the cut road surface cut image C to the transmission unit 105 . That is, the image extraction unit 111 sequentially extracts the road surface cutout image C from the extracted camera image A1 that satisfies the extraction condition among the camera images A captured by the camera 101 at a predetermined imaging cycle, and outputs the extracted road surface image C to the transmission unit 105 .

Note that the image extracting unit 111 is not limited to extracting the extracted camera image A1 based on the extraction condition and then extracting the road surface clipped image C from the extracted camera image A1. For example, the image extracting unit 111 cuts out a road surface cut-out image C from the camera image A each time the camera 101 captures an image, and after cutting out the road surface cut-out image C, the camera image A (extracted camera image A1) satisfying the extraction condition is extracted. ) may be extracted.

The extraction period setting unit 112 sets extraction conditions based on the speed of the vehicle V detected by the vehicle sensor 102 . This extraction condition is a condition for the image extracting unit 111 to extract the extracted camera image A1 from which the road surface clipped image C is extracted from the plurality of camera images A captured by the camera 101, as described above. In this embodiment, this extraction condition indicates whether or not the camera image A captured at the current time (current imaging timing) is to be extracted. That is, the extraction condition includes the time for extracting the camera image A as the extracted camera image A1.

Here, when the road surface image D is generated from the camera image A captured at each time, when the speed of the vehicle V is high, the road surface orthoimage E is generated as shown in FIG. The overlapping area of the road surface image D becomes smaller. In this case, depending on the settings of each part, when the road surface image D is pasted, the image may be missing. Further, when the speed of the vehicle V is slow, as shown in FIG. 3B, when the road surface orthoimage E is generated, the overlapping area of the road surface image D becomes large. In this case, the cut-out road surface image C redundant to the generation of the road surface orthoimage E is transmitted from the in-vehicle device 100 to the map generation device 200 . For this reason, the extraction cycle setting unit 112 in this embodiment sets extraction conditions so that the size of the overlapping area of the road surface image D is appropriate when the map generation device 200 generates the road surface orthorectified image E.

Specifically, the extraction cycle setting unit 112 calculates the overlapping rate of the two road surface images D when generating the road surface orthoimage E based on the two road surface images D. FIG. One of the two road surface images D is the road surface image D generated based on the latest (closest) extracted camera image A1 extracted by the image extraction unit 111 . The road surface image D, which is the other of the two road surface images, is the road surface image D assumed to have been generated based on the camera image A captured at the current time (current imaging timing). In this manner, the extraction cycle setting unit 112 calculates the overlapping rate between the most recently generated road surface image D and the hypothetical road surface image D generated based on the camera image A at the current time. The overlapping rate here changes according to the speed of the vehicle V. FIG. For this reason, the extraction cycle setting unit 112 calculates the overlapping rate in consideration of the speed of the vehicle V. FIG.

In addition, the extraction cycle setting unit 112 determines whether or not the calculated overlapping rate is less than a predetermined overlapping rate threshold. When the calculated overlapping rate is less than the predetermined overlapping rate, the extraction cycle setting unit 112 determines that the camera image A captured at the current time (current imaging timing) is the extracted camera image A1 to be extracted. do. The extraction cycle setting unit 112 sets, as an extraction condition, that the camera image A captured at the current time is to be extracted.

On the other hand, if the calculated overlap rate is not less than the predetermined overlap rate, the extraction cycle setting unit 112 determines that the camera image A captured at the current time (current imaging timing) is not an extraction target. . After this determination, the extraction cycle setting unit 112 again generates the image based on the most recently generated road surface image D and the camera image A at the current time, as described above, after the predetermined imaging cycle has elapsed. The overlap rate with the road surface image D when assuming that is calculated. In this manner, the extraction cycle setting unit 112 sets the most recently generated road surface image D and the road surface image D assumed to have been generated based on the camera image A at the current time, for each predetermined imaging cycle. Calculate the duplication rate of Then, the extraction cycle setting unit 112 determines whether or not the camera image A captured at the current time is to be extracted based on the calculated duplication rate, and sets extraction conditions.

More specifically, the extraction cycle setting unit 112 sets the overlap rate based on the amount of movement of the vehicle V in one imaging cycle of the camera 101 and the road surface equivalent region B when the road surface cut image C is cut out. can be calculated. The extraction cycle setting unit 112 can calculate the amount of movement of the vehicle V based on the speed of the vehicle V detected by the vehicle sensor 102 and the preset imaging cycle of the camera 101 (predetermined imaging cycle). can. Here, the road surface equivalent area B is, as an example, a range from 8 m to 16 m in front of the vehicle V from the installation position of the camera 101 (that is, an area with a depth of 8 m) and a range of ±3.5 m left and right (that is, a width of 7 m). area).

For example, assume that camera image A captured at time t[s] is extracted as extracted camera image A1, and road surface image D is generated from extracted camera image A1. Suppose that the vehicle V travels 1 m forward at time t+0.1 [s], which is the next imaging cycle of the camera 101 . In this case, the road surface image D generated from the extracted camera image A1 captured at time t[s] and the road surface image D generated from the camera image A captured at time t+0.1 [s] The overlapping rate with is about 87%, as shown in FIG. In this case, the overlapping rate is equal to or higher than the preset overlapping rate threshold value (40%). Therefore, the extraction cycle setting unit 112 does not determine the camera image A captured at time t+0.1 [s] as an extraction target.

For example, the road surface image D generated from the extracted camera image A1 captured at time t [s] and the Assuming that it is generated from the camera image A, the overlapping rate with the road surface image D is less than the overlapping rate threshold (40%). Therefore, the extraction cycle setting unit 112 determines that the camera image A captured at time t+0.5 [s] is to be extracted. Then, the extraction cycle setting unit 112 sets, as an extraction condition, that the camera image A captured at time t+0.5 [s] is to be extracted.

In this manner, the extraction cycle setting unit 112 is assumed to have been generated based on the road surface image D generated based on the latest extracted camera image A1 and the camera image A captured at a timing later than the road surface image D. The overlap rate with the assumed road surface image D is calculated, and the extraction timing (extraction condition) of the camera image A is determined according to the calculated overlap rate. The extraction cycle setting unit 112 outputs the set extraction conditions to the image extraction unit 111 . Note that the overlap rate threshold used when setting the extraction condition may be a constant value, or may be changed according to the place where the vehicle V is traveling, or the like.

The trajectory recognition unit 113 recognizes the travel trajectory and orientation of the vehicle V (orientation of the vehicle V). In the present embodiment, the trajectory recognition unit 113 recognizes the position and orientation of the vehicle V at each imaging timing when the camera 101 performs imaging as the travel trajectory and orientation. The trajectory recognition section 113 recognizes the position of the vehicle V at each imaging timing based on the position information measured by the GPS reception section 103 . The trajectory recognition unit 113 recognizes the attitude of the vehicle V at each imaging timing based on the change in the positional information. Note that the trajectory recognition unit 113 may detect the attitude of the vehicle V based on the detection result of a gyro sensor or the like that detects the yaw rate (rotational angular velocity) of the center of gravity of the vehicle V around the vertical axis. The trajectory recognition unit 113 outputs the recognized position and orientation of the vehicle V to the transmission unit 105 .

When the road surface cut-out image C is input, the transmission unit 105 transmits the vehicle image at the time when the input road surface cut-out image C and the extracted camera image A1 used as the basis for generating the road surface cut-out image C were captured. The position and orientation of V are transmitted to the map generation device 200 by communication. The transmission unit 105 transmits the cut-out road surface image C and the like to the map generation device 200 by various communication methods such as wireless communication or wired communication. That is, the transmission unit 105 transmits a portion of the extracted camera image A1 extracted based on the extraction condition to the map generation device 200. FIG.

(Details of the map generation device)
The map generation device 200 is provided in a facility such as a center that manages map information, for example. The map generating device 200 has a receiving section 201 and a map generating section 202 .

The receiving unit 201 acquires the cut-out road surface image C and the position and orientation of the vehicle V transmitted from the transmitting unit 105 through communication.

Like the ECU 104, the map generator 202 is composed of an electronic control unit having a CPU, a ROM, a RAM, and the like. The map generator 202 generates a map based on the cut-out road image C received by the receiver 201 and the position and orientation of the vehicle V. FIG. Specifically, the map generation unit 202 performs bird's eye conversion on the received road surface clipped image C to generate a road surface image D (see FIG. 2(c)). Then, the map generation unit 202 pastes the generated road surface image D onto the coordinate system of the map. Note that the map generator 202 moves and rotates the road surface image D based on the position and orientation of the vehicle V when pasting the road surface image D, and then pastes it.

The map generating unit 202 performs this pasting every time the receiving unit 201 receives the cut-out road surface image C or the like, and generates a road surface orthorectified image E as a map. By performing pasting based on the cut-out road surface image C generated based on the extraction conditions, the overlap (overlapping) between the road surface images D becomes less than the overlap rate threshold. The road surface orthoimage E generated by the map generation unit 202 is used for various purposes such as creation of map information.

(Flow of image transmission processing)
Next, the flow of image transmission processing performed in the in-vehicle device 100 will be described with reference to FIG. The camera 101 performs imaging at a predetermined imaging cycle. The processing shown in FIG. 5 is started at each imaging cycle (at each imaging time) when the camera 101 performs imaging.

As shown in FIG. 5, the trajectory recognition unit 113 recognizes the position and orientation of the vehicle V at the current time (S101). The vehicle sensor 102 detects the speed of the vehicle V at the current time (S102). When it is assumed that the extraction cycle setting unit 112 generates the road surface image D generated based on the most recent extracted camera image A1 and the camera image A at the current time while considering the speed of the vehicle V. and the road surface image D are calculated (S103).

The extraction cycle setting unit 112 determines whether the calculated overlapping rate is less than the overlapping rate threshold (S104). If the overlapping rate is not less than the overlapping rate threshold (S104: NO), the in-vehicle device 100 terminates the current process, and restarts the process shown in FIG. 5 after a predetermined imaging cycle has elapsed.

On the other hand, if the overlap rate is less than the overlap rate threshold (S104: YES), the extraction cycle setting unit 112 sets the camera image A captured at the current time (current imaging timing) as the extracted camera image A1 to be extracted. Then, an extraction condition is set to the effect that this camera image A is to be extracted. The image extracting unit 111 extracts the camera image A captured at the current time as an extracted camera image A1 based on the extraction conditions, and cuts out the extracted extracted camera image A1 to generate a road cut-out image C (S105). ). The transmission unit 105 transmits the generated road cut-out image C and the position and orientation of the vehicle V at the current time recognized in S101 to the map generation device 200 by communication (S106).

As described above, the map generation system 1 of the present embodiment transfers road surface data from the in-vehicle device 100 to the map generation device 200 so that the overlap rate of the road surface image D when generating the road surface orthoimage E is less than the overlap rate determination threshold. A clipped image C can be transmitted. As a result, the map generation system 1 can efficiently transmit the image (road cut-out image C) captured by the camera 101 from the in-vehicle device 100 to the map generation device 200 .

(Second embodiment)
A second embodiment of the map generation system will be described. The in-vehicle device 100 according to the first embodiment determines whether or not to transmit the cut-out road surface image C or the like while confirming the overlapping rate at each time of the imaging cycle of the camera 101 . The in-vehicle device according to the second embodiment estimates the next transmission timing of the road surface cut-out image C and the like based on the speed and acceleration of the vehicle V at the time when the road surface cut-out image C and the like is transmitted.

Specifically, as shown in FIG. 6, a map generation system 1A in the second embodiment includes an in-vehicle device 100A and a map generation device 200. FIG. In the following description, the same reference numerals are given to the same components as those of the map generation system 1 of the first embodiment, and detailed description thereof will be omitted. The in-vehicle device 100</b>A includes a camera (imaging unit) 101 , a vehicle sensor (driving condition acquisition unit) 102</b>A, a GPS receiving unit 103 , an ECU 104</b>A, and a transmitting unit 105 .

The ECU 104A is an electronic control unit similar to the ECU 104 in the first embodiment. The ECU 104</b>A functionally includes an image extractor 111 , an extraction period setter (condition setter) 112</b>A, and a trajectory recognizer 113 . That is, the vehicle-mounted device 100A includes a vehicle sensor 102A and an extraction cycle setting unit 112A instead of the vehicle sensor 102 and the extraction cycle setting unit 112 of the vehicle-mounted device 100 in the first embodiment.

The vehicle sensor 102A is a sensor that acquires the running condition of the vehicle V. FIG. In this embodiment, the vehicle sensor 102A detects the speed and acceleration of the vehicle V as driving conditions. That is, the vehicle sensor 102A includes a vehicle speed sensor that detects the speed of the vehicle V and an acceleration sensor that detects the acceleration of the vehicle. The vehicle sensor 102A inputs the detected speed and acceleration of the vehicle V to the ECU 104A.

The extraction cycle setting unit 112A sets extraction conditions based on the speed and acceleration of the vehicle V detected by the vehicle sensor 102A. This extraction condition includes the time at which the image extraction unit 111 extracts the camera image A next time. Specifically, when the image extracting unit 111 extracts the extracted camera image A1, the extraction cycle setting unit 112A sets the extracted camera image based on the speed and acceleration of the vehicle V at the time when the extracted camera image A1 is extracted. The overlap rate between the road surface image D generated based on the image A1 and the road surface image D when it is assumed that the road surface image D is generated based on the camera image A captured at a time later than the road surface image D is less than the overlap threshold. Calculate the time when The extraction cycle setting unit 112 sets the time at which the overlap rate is less than the overlap threshold as the extraction condition as the time at which the image extraction unit 111 should next extract the extracted camera image A1.

The image extraction unit 111 extracts an extraction camera image A1 based on the extraction conditions set by the extraction cycle setting unit 112. FIG. Specifically, the image extracting unit 111 refers to the extraction conditions, and if the current time is the time to extract the next extracted camera image A1 set in the extraction conditions, the camera imaged by the camera 101 The image A is extracted as an extracted camera image A1, and a road cut-out image C is generated from this extracted camera image A1.

(Flow of image transmission processing)
Next, the flow of image transmission processing performed in the in-vehicle device 100A will be described with reference to FIG. The camera 101 performs imaging at a predetermined imaging cycle. The processing shown in FIG. 7 is started at each imaging cycle (at each imaging time) when the camera 101 performs imaging.

As shown in FIG. 7, the trajectory recognition unit 113 recognizes the position and orientation of the vehicle V at the current time (S201). The image extraction unit 111 determines whether or not the current time is the time to extract the next extracted camera image A1 set in the extraction condition (S202). If the current time is not the time to extract the extracted camera image A1 next time (S202: NO), the in-vehicle device 100A ends the current process, and restarts the process shown in FIG. 7 after a predetermined imaging cycle has elapsed. do. Note that the case where the current time is not the time at which the next extracted camera image A1 is extracted means that the current time is before the time at which the next extracted camera image A1 is extracted.

On the other hand, if the current time is the time to extract the extracted camera image A1 next time (S202: YES), the image extraction unit 111 selects the camera image A captured by the camera 101 at the current time as the extracted camera image A1. A road cut-out image C is generated from the extracted extracted camera image A1 (S203). The vehicle sensor 102A detects the speed and acceleration of the vehicle V at the current time (S204).

Based on the speed and acceleration of the vehicle V, the extraction cycle setting unit 112A calculates the next extraction time of the extraction camera image A1. Specifically, as described above, the extraction cycle setting unit 112A calculates the time at which the overlap rate of the road surface image D is less than the overlap threshold, and sets it as an extraction condition (S205). The transmission unit 105 transmits the road surface cut-out image C generated in S203 and the position and orientation of the vehicle V at the current time recognized in S201 to the map generation device 200 by communication (S206).

As described above, in the map generation system 1A of the present embodiment, as in the first embodiment, the image captured by the camera 101 (road cut-out image C) is efficiently transmitted from the in-vehicle device 100A to the map generation device 200. can. In addition, after the camera image A captured by the camera 101 is extracted as the extracted camera image A1, the in-vehicle device 100A can grasp the timing of extracting the camera image A as the extracted camera image A1 next time. Therefore, the in-vehicle device 100A can use the camera 101 for other purposes until the next extraction of the extracted camera image A1.

For example, as shown in FIG. 8, the camera 101 captures four images with different exposure patterns every 0.1 seconds. Here, the camera 101 captures images of exposure patterns A to D, respectively. The exposure pattern A is an exposure pattern for capturing an image used to generate the road surface orthoimage E. As shown in FIG. In this case, if the image extracting unit 111 extracts the camera image A next time after 0.3 seconds, it is not necessary to take an image with the exposure pattern A during that time. Therefore, for example, the camera 101 can capture an image of the exposure pattern C at intervals of 0.05 seconds by capturing an image using the exposure pattern C for capturing an image used for object recognition instead of the exposure pattern A. . Thereby, for example, the cycle of object recognition can be shortened.

(Third embodiment)
A third embodiment of the map generation system will be described. In the in-vehicle device according to the third embodiment, by changing the range to be cut out from the camera image A as the road surface cut-out image C, the overlap rate when the road surface image D is pasted is set to a desired overlap rate. Note that in the present embodiment, instead of extracting the extracted camera image A1 from the plurality of camera images A and generating the road surface clipped image C as in the first embodiment, all the camera images captured by the camera 101 are A road cut-out image C is generated from A.

Specifically, as shown in FIG. 9, a map generation system 1B in the third embodiment includes an in-vehicle device 100B and a map generation device 200. FIG. The in-vehicle device 100</b>B includes a camera (imaging unit) 101 , a vehicle sensor (driving condition acquisition unit) 102 , a GPS receiving unit 103 , an ECU 104</b>B, and a transmitting unit 105 .

The ECU 104B is an electronic control unit similar to the ECU 104 in the first embodiment. The ECU 104</b>B functionally includes an image extractor 111</b>B, an image clipping range setter (condition setter) 112</b>B, and a trajectory recognizer 113 . That is, the vehicle-mounted device 100B includes an image extraction unit 111B and an image clipping range setting unit 112B instead of the image extraction unit 111 and the extraction cycle setting unit 112 of the vehicle-mounted device 100 in the first embodiment.

Based on the speed of the vehicle V detected by the vehicle sensor 102, the image cutout range setting unit 112B calculates a range (cutout range) to be cut out as the road surface cutout image C from the camera image A. FIG. Specifically, for example, it is assumed that the imaging cycle of the camera 101 is 0.1 second and the current speed of the vehicle V is 40 km/h. In this case, it is expected that the vehicle V will advance about 1 m until the next imaging timing of the camera 101 . Therefore, the image cropping range setting unit 112B sets the road surface image D generated based on the camera image A at the next imaging time and the road surface image D generated based on the camera image A at the current time. A range to be clipped from the camera image A as the road surface clipped image C is calculated so that the overlapping rate of is about 50%, for example. In this case, the range to be cut out from the camera image A as the road surface cut-out image C is a range of 8 m to 10 m in front of the vehicle V from the installation position of the camera 101 (that is, a range of 2 m in depth) and a range of ±3.5 m on the left and right ( That is, the image range corresponds to the area of 7 m width).

The image cut-out range setting unit 112B sets the range to be cut out as the calculated road surface cut-out image C as an extraction condition. The image cropping range setting unit 112B sets extraction conditions for each imaging cycle of the camera 101 . Image clipping range setting section 112B outputs the set extraction conditions to image extraction section 111B.

The image extracting unit 111B cuts out an image of a portion corresponding to the range set as the extraction condition from the camera image A to generate a road cut-out image C. FIG. Note that the image extraction unit 111B cuts out the road surface cutout image C based on the extraction conditions input from the image cutout range setting unit 112B each time the camera image A is captured by the camera 101 . The image extraction unit 111B outputs the cut road surface cut image C to the transmission unit 105 .

(Flow of image transmission processing)
Next, the flow of image transmission processing performed in the in-vehicle device 100B will be described with reference to FIG. The camera 101 performs imaging at a predetermined imaging cycle. The processing shown in FIG. 10 is started at each imaging cycle (at each imaging time) when the camera 101 performs imaging.

As shown in FIG. 9, the trajectory recognition unit 113 recognizes the position and attitude of the vehicle V at the current time (S301). The vehicle sensor 102 detects the speed of the vehicle V at the current time (S302). The image cutout range setting unit 112B calculates a range to be cut out as the road surface cutout image C from the camera image A based on the speed of the vehicle V (S303). The image extraction unit 111B cuts out the camera image A to generate a road surface cutout image C (S304). The transmission unit 105 transmits the generated road cut-out image C and the position and orientation of the vehicle V at the current time recognized in S301 to the map generation device 200 by communication (S305).

As described above, in the map generation system 1B of the present embodiment, the image extraction unit 111B extracts the camera image A so that the overlap rate of the road surface image D when generating the road surface orthoimage E is the desired overlap rate. A cut road surface image C is generated by cutting. As a result, the map generation system 1B suppresses redundant portions of the road surface cutout image C for generating the road surface orthorectified image E, and transmits the image captured by the camera 101 (road surface cutout image C) from the in-vehicle device 100B. It can be efficiently transmitted to the map generation device 200 .

Note that when the vehicle V is traveling at a constant speed, even if the overlapping rate of the road surface image D used when calculating the range to be cut out from the camera image A as the road surface cut-out image C is set low, the road surface image D It is unlikely that the image will be missing when pasted. However, during acceleration and deceleration, if the overlap rate is set low, there is a possibility that the image will be missing when the road surface image D is pasted. Therefore, the image cutout range setting unit 112B may set the cutout range of the road surface cutout image C in consideration of not only the speed of the vehicle V but also the acceleration of the vehicle V. FIG. In this case, the map generation system 1B can transmit the cut-out road surface image C and the like from the in-vehicle device 100B to the map generation device 200 more efficiently.

(Fourth embodiment)
A fourth embodiment of the map generation system will be described. As shown in FIG. 11, a map generation system 1C in the fourth embodiment includes an in-vehicle device 100C and a map generation device 200C.

Here, an overview of the map generation process performed by the map generation system 1C will be described. Most of the road surface of the road on which the vehicle V travels consists only of lanes (white lines), and only a part of the road surface has characteristic road markings or the like. For this reason, the map generation system 1C duplicates and pastes a road surface image of only lanes generated from a camera image captured at a certain time for a portion composed only of lanes. The map generation system 1C generates and pastes a road surface image including road markings and the like generated from the captured camera image for characteristic portions having road markings and the like. As a result, the map generation system 1C can reduce the amount of image data communication between the in-vehicle device 100C and the map generation device 200C.

As a specific example of this map generation processing, a case where a vehicle V is traveling from the left side to the right side in the figure on the traveling path L shown in the upper part of FIG. 12 will be described. First, it is assumed that the vehicle V is traveling at the leftmost position of the traveling path L shown in the upper part of FIG. The in-vehicle device 100C performs recognition processing of the road surface of the travel path L, determines that there is no road marking M ahead (immediately) of the vehicle V, and crosses only the lane (white line L1) where the road marking M does not exist. An output image C is cut out from the camera image A. Then, the in-vehicle device 100C transmits the cut road surface cut-out image C and the position and orientation of the vehicle V at that time to the map generation device 200C. The map generation device 200C generates a road surface image DA from the received road surface cutout image C, and pastes the road surface image DA on the coordinate system of the map based on the position and attitude of the vehicle V (see the lower part of FIG. 12).

After that, the vehicle V runs only on the lane for a while, so the in-vehicle device 100C does not transmit the cut-out road surface image C or the like. When the vehicle V reaches the position of the road marking M and the road marking M is recognized by the road surface recognition processing, the in-vehicle device 100C cuts out the road surface clipped image C from the camera image A including the road marking M. Then, the in-vehicle device 100C transmits the cut-out road surface image C including the road markings M, and the travel path of the vehicle V from the transmission of the cut-out road surface image C last time to the present (history of changes in the position of the vehicle V). and the attitude at each position on the travel locus to the map generating device 200C. The map generating device 200C generates a road surface image DB from the cut road surface image C including the received road marking M. FIG. As a result of the recognition processing of the road surface of the traveling road in the in-vehicle device 100C, the road surface with only the lane continues like the road surface image DA between the road surface image DA and the road surface image DB. Therefore, the map generation device 200C generates the road surface image Da by duplicating the road surface image DA generated based on the previously received road surface cutout image C of only the lanes. The map generation device 200C sequentially pastes the road surface image Da to the section from the road surface image DA to the road surface image DB based on the travel locus and direction of the vehicle V. FIG. After that, the map generation device 200C pastes the road surface image DB including the road marking M. FIG. In the lower part of FIG. 12, the road surface image Da generated by duplication is indicated by a dashed line in order to facilitate the distinction between the road surface image D and the road surface image Da. A road surface image Dc, which will be described later, is likewise indicated by a dashed line.

When the vehicle V travels further, the in-vehicle device 100C cannot recognize the road marking M by road surface recognition processing. When the road marking M cannot be recognized in this manner, the in-vehicle device 100C cuts out from the camera image A again the road surface cut-out image C of only the lane where the road marking M does not exist. Then, the in-vehicle device 100C transmits to the map generation device 200C the cut-out road surface cut-out image C, and the traveling locus and the posture at each position of the vehicle V from the transmission of the road cut-out image C last time to the present. do. The map generating device 200C generates a road surface image DC from the received road surface cutout image C, and pastes the road surface image DC on the coordinate system of the map based on the position and orientation of the vehicle V. FIG.

After that, the vehicle V runs only on the lane for a while, so the in-vehicle device 100C does not transmit the cut-out road surface image C or the like. Then, when the vehicle V reaches the end point of the area for which the map is generated, the in-vehicle device 100C displays the traveling locus of the vehicle V from the previous transmission of the cut-out road surface image C up to the present time and the respective positions on the traveling locus. and the attitude are transmitted to the map generation device 200C. As a result of the recognition processing of the road surface of the traveling road in the in-vehicle device 100C, the road surface with only lanes continues like the road surface image DC from the road surface image DC to the end point of the area for which the map is generated. Therefore, the map generation device 200C generates the road surface image Dc by duplicating the road surface image DC generated based on the previously received road surface cutout image C of only the lanes. Based on the travel locus and direction of the vehicle V, the map generation device 200C sequentially pastes the road surface image Dc in the section from the road surface image DC to the end point of the area for which the map is to be generated.

In this way, the map generation device 200C receives images of several points in the area for which the map is to be generated, along with the travel locus and direction. The map generation device 200C duplicates and pastes the image received last time for the positions for which no image has been received. In this manner, the road surface orthoimage E as a map is generated by pasting the road surface image by the map generation device 200C.

(Configuration of in-vehicle device)
The in-vehicle device 100C includes a camera (imaging unit) 101, a GPS receiving unit 103, an ECU 104C, and a transmitting unit 105C. The ECU 104C is an electronic control unit similar to the ECU 104 in the first embodiment. The ECU 104</b>C functionally includes a trajectory recognition section 113 , a travel path recognition section (driving situation acquisition section) 114 , an extraction condition setting section (condition setting section) 115 , and an image extraction section 116 .

The traveling road recognition unit 114 acquires the traveling situation of the vehicle V. FIG. The driving condition here refers to the condition of the road on which the vehicle V travels. The traveling road recognition unit 114 recognizes the presence or absence of road markings M attached on the road surface as the state of the traveling road. Specifically, the traveling road recognition unit 114 performs recognition processing of the road surface of the traveling road based on the camera image A of the camera 101 . The traveling road recognition unit 114 performs road recognition processing each time the camera image A is captured by the camera 101 . Here, the traveling road recognition unit 114 recognizes the presence or absence of the road marking M attached on the road surface and the type of the white line L1 (broken line, solid line, double line, etc.). For example, the travel path recognition unit 114 can recognize the presence or absence of road markings M based on a well-known image processing technique. When the road marking M exists on the road surface, the traveling road recognition unit 114 sets the road marking flag to ON. When the road marking M does not exist, the traveling road recognition unit 114 sets the road marking flag to OFF.

The extraction condition setting unit 115 sets extraction conditions based on the running condition of the vehicle V. FIG. Specifically, the extraction condition setting unit 115 sets the extraction condition based on the presence or absence of road markings M attached to the road surface of the traveling road as the driving situation. This extracting condition is a condition for the image extracting unit 116 to extract an extracting camera image A1 for extracting a road surface clipped image C from a plurality of camera images A captured by the camera 101 . More specifically, the extraction condition setting unit 115 sets, as extraction conditions, the camera 101 when the road surface flag is set to ON by the traveling road recognition unit 114 and when the road surface flag changes from ON to OFF. It is set that the captured camera image A is to be extracted. That is, the extraction condition setting unit 115 sets the time when the image extracting unit 116 extracts the extracted camera image A1 based on the recognition result of the traveling road recognized by the traveling road recognizing unit 114 .

The image extracting unit 116 extracts the camera images A satisfying the extraction conditions set by the extraction cycle setting unit 112 as extracted camera images A1 from among the plurality of camera images A captured by the camera 101 at a predetermined image capturing cycle. In addition, the image extracting unit 116 sets a range (clipping range) for cutting out the road surface clipped image C from the extracted camera image A1. The image extraction unit 116 uses, for example, the image range of the portion corresponding to the road surface equivalent area B described in the first embodiment as the basic range of the range to be cut out as the road surface cutout image C from the extracted camera image A1. This basic range is preset. The range to be cut out from the extracted camera image A1 as the road surface cut-out image C may be variable within the basic range based on the result of the recognition processing of the road surface of the traveling road.

For example, if the center line or boundary line of the road is a dashed line, the area to be extracted can be appropriately set based on the result of the road surface recognition processing, taking into consideration the period between the white line portion and the road surface portion of the dashed line. When the generation device 200C duplicates and sequentially pastes the road surface image including the dashed center line, etc., the white line portion and the road surface portion of the dashed line can be pasted together without any contradiction in the period.

The image extracting unit 116 cuts out the extracted camera image A1 based on the set range to generate a road cut-out image C. FIG. The image extraction unit 116 outputs the generated road surface clipped image C to the transmission unit 105C. Note that, at the start of the map generation process, the image extraction unit 116 extracts the camera image A captured by the camera 101 as the extracted camera image A1, and extracts the road crossing from the extracted camera image A1 even if the road surface flag is OFF. Generate an output image C.

When the road surface cut-out image C is input, the transmission unit 105C transmits the input road surface cut-out image C, the traveling locus of the vehicle V up to the present time since the road surface cut-out image C was transmitted last time, and each position. and the attitude of the map generation device 200C. At the start of the map generation process, the transmission unit 105C transmits the cut-out road surface image C generated by the image extraction unit 116 at the start of the generation process and the position and orientation of the vehicle V at that time to the map generation device 200C. Send. In addition, when the vehicle V reaches the end point of the area for which the map is generated, the transmission unit 105C transmits the traveling locus and the posture at each position of the vehicle V from the previous transmission of the cut-out road surface image C to the present. It is transmitted to the map generation device 200C.

(Details of the map generation device)
200 C of map generation apparatuses are provided with the receiving part 201 and 202 C of map generation parts. 202 C of map generation parts are electronic control units similar to ECU104 in 1st Embodiment. As described above, when the road surface cut-out image C is received, the map generation unit 202C converts the road surface cut-out image C into the road surface image D, and converts the road surface image D into the map coordinate system based on the travel locus and orientation. paste. Further, the map generation unit 202C duplicates the road surface image D generated based on the road surface cut-out image C previously transmitted (transmitted in the past) for the section in which the road surface cut-out image C has not been transmitted. , the duplicated road surface image D is pasted on the coordinate system of the map based on the travel locus and direction. As a result, a road surface orthoimage E is generated as a map.

(Flow of image transmission processing)
Next, the flow of image transmission processing performed by the in-vehicle device 100C will be described with reference to FIG. The camera 101 performs imaging at a predetermined imaging cycle. The processing shown in FIG. 13 is started at each imaging cycle (at each imaging time) when the camera 101 performs imaging.

The trajectory recognition unit 113 determines whether or not the map generation process in the map generation system 1C has just started (S401). If the map generation process has just started (S401: YES), the trajectory recognition unit 113 recognizes the position and orientation of the vehicle V at the current time (S402). The traveling road recognition unit 114 acquires the camera image A captured by the camera 101 at the current time (S403). Based on the camera image A, the traveling road recognition unit 114 recognizes the presence or absence of the road marking M attached on the road surface and the type of the white line L1. The traveling road recognition unit 114 sets the road marking flag to ON when the road marking M exists on the road surface, and sets the road marking flag to OFF when the road marking M does not exist (S404).

The image extraction unit 116 sets a basic range as a range for cutting out the road surface cut-out image C (S405). The image extracting unit 116 extracts the camera image A captured by the camera 101 at the current time as an extracted camera image A1, and cuts out a road surface cropped image C from the extracted camera image A1 based on the set basic range (S406). ). The transmission unit 105C transmits the road surface cut-out image C cut out by the image extraction unit 116 and the position and orientation of the vehicle V at the current time to the map generation device 200C by communication (S407). After transmitting the cut-out image C of the road surface, etc., the in-vehicle device 100C ends the current processing, and restarts the processing shown in FIG. 13 after a predetermined imaging cycle has elapsed.

On the other hand, if it is not immediately after the start of the map generation process (S401: NO), the trajectory recognition unit 113 recognizes the position and attitude of the vehicle V at the current time (S408). The traveling road recognition unit 114 acquires the camera image A captured by the camera 101 at the current time (S409). Based on the camera image A, the traveling road recognition unit 114 recognizes the presence or absence of the road marking M attached on the road surface and the type of the white line L1. The traveling road recognition unit 114 sets the road marking flag to ON when the road marking M exists on the road surface, and sets the road marking flag to OFF when the road marking M does not exist (S410).

The extraction condition setting unit 115 sets extraction conditions based on ON and OFF of the road surface flag set by the traveling road recognition unit 114 (S411). The image extraction unit 116 determines whether or not the camera image A captured at the current time satisfies the extraction conditions (S412). If the extraction condition is not satisfied (S412: NO), the in-vehicle device 100C terminates the current process, and restarts the process shown in FIG. 13 after a predetermined imaging cycle has elapsed.

If the extraction condition is satisfied (S412: YES), the image extraction unit 116 sets a range for cutting out the road surface cutout image C (S413). Here, the image extracting unit 116 may set a basic range as the range for cutting out the road surface cut-out image C, or may set a size range that does not exceed the basic range. The image extracting unit 116 extracts the camera image A captured by the camera 101 at the current time as an extracted camera image A1, and cuts out a road surface cropped image C from the extracted camera image A1 based on the set range (S414). . The transmission unit 105C transmits the road surface cut-out image C cut out by the image extraction unit 116, and the traveling locus of the vehicle V up to the present time since the road surface cut-out image C and the like were transmitted in the processing in S407 or S415 last time, and each The attitude at the position is transmitted to the map generating device 200C by communication (S415). After transmitting the cut-out image C of the road surface, etc., the in-vehicle device 100C ends the current processing, and restarts the processing shown in FIG. 13 after a predetermined imaging cycle has elapsed.

In this way, immediately after starting the map generation process, the in-vehicle device 100C performs the processes of S402 to S407 and transmits the cut-out road surface image C and the like to the map generation device 200C. If it is not immediately after the start of the map generation process, the in-vehicle device 100C performs the processes of S408 to S415, and transmits the road cut-out image C and the like to the map generation device 200C according to the extraction condition (whether or not there is a road marking M).

As described above, in the map generation system 1C of the present embodiment, the in-vehicle device 100C transmits the road cut-out image C and the like to the map generation device 200C according to the presence or absence of the road marking M. The map generation device 200C generates the road surface orthoimage E using the road surface cut-out image C transmitted last time for the section for which the road surface cut-out image C has not been transmitted. As a result, the map generation system 1C can reduce the amount of image data communication between the in-vehicle device 100C and the map generation device 200C, and can efficiently transmit the image data.

Although the embodiments of the present invention have been described above, the present invention is not limited to the above embodiments. For example, although the camera 101 images the front of the vehicle V, the camera 101 may image the rear of the vehicle V as long as it is around the vehicle V, for example. Even in this case, the map generation system 1 or the like can generate a map in the same manner as when the camera image in front of the vehicle V is used. Also, the imaging cycle of the camera 101 may be constant or variable.

1, 1A, 1B, 1C... map generation system, 100, 100A, 100B, 100C... in-vehicle device, 101... camera (imaging unit), 102, 102A... vehicle sensor (driving condition acquisition unit), 105, 105C... transmitting unit , 111, 111B, 116... Image extraction unit 112, 112A... Extraction period setting unit (condition setting unit) 112B... Image clipping range setting unit (condition setting unit) 113... Trajectory recognition unit 114... Travel path recognition Section (driving condition acquisition section), 115... Extraction condition setting section (condition setting section), 200, 200C... Map generating device, 201... Receiving section, 202C... Map generating section, V... Vehicle.

Claims (2)

A map generation system comprising: an in-vehicle device that is mounted on a vehicle and captures an image of the surroundings of the vehicle; and a map generation device that generates a map based on the image captured by the in-vehicle device,
The in-vehicle device
an imaging unit that images the surroundings of the vehicle at a predetermined imaging cycle;
a driving condition acquisition unit that acquires the driving condition of the vehicle;
a trajectory recognition unit that recognizes the travel trajectory of the vehicle;
a condition setting unit that sets conditions for extracting the image captured by the imaging unit based on the driving situation;
an image extraction unit that extracts the image that satisfies the extraction condition from among the images captured by the imaging unit;
a transmission unit that transmits the travel locus and the extracted image to the map generation device by communication;
with
The map generation device is
a receiving unit that receives, through communication, the travel locus and the image transmitted from the transmitting unit;
a map generation unit that generates a map based on the received travel locus and the image;
with
The traveling situation acquisition unit acquires a recognition result of a traveling road as the traveling situation,
The condition setting unit sets, as the extraction condition, a time at which the image extraction unit extracts the image as an extraction target based on the recognition result of the traveling road,
The map generating system, wherein when extracting the image, the image extracting unit sets a clipping range of the image based on the recognition result of the travel path, and extracts the image within the set clipping range. 2. The map generation system according to claim 1 , wherein said map generation unit generates said map based on said image transmitted in the past for a section in which said image has not been transmitted from said in-vehicle device.

Images