JP7203514B2 - Driving area determination device, driving area determination method, and road surface image machine learning model generation method - Google Patents

Description

The disclosed embodiments relate to a driving area determination device, a driving area determination method, and a road surface image machine learning model generation method.

Conventionally, obstacles that may collide with the vehicle are detected by comparing an image of the surroundings of the vehicle with pattern matching data created based on the assumption of the shape of the obstacle. There is a detection device (see Patent Document 1, for example). This obstacle detection result is used as information for an obstacle (unsuitable travel area), an appropriate travel area that does not hinder travel, and information for automatic driving control.

JP 2009-140023 A

However, a device that detects an obstacle using pattern matching data that is created by assuming the shape of an obstacle in advance cannot detect an unexpected obstacle. It may not be possible to accurately determine the areas that may be affected.

One aspect of the embodiments has been made in view of the above, and includes a device and method for accurately determining an unsuitable or suitable travel region on a road on which a vehicle travels. It is an object of the present invention to provide a travel area determination device and a travel area determination method capable of accurately determining an area in which there is a possibility that a

A travel area determination device according to an aspect of an embodiment includes an acquisition unit and a determination unit. The obtaining unit obtains an image of the surroundings of the vehicle from the imaging device. The determination unit determines a specific region in the image acquired by the acquisition unit by using a machine learning model obtained by performing machine learning on a plurality of images known to be road surfaces.

A driving area determination device and a driving area determination method according to an aspect of an embodiment can accurately determine an unsuitable driving area or a suitable driving area on a road on which a vehicle travels.

FIG. 1 is an explanatory diagram showing an overview of the travel area determination method according to the embodiment. FIG. 2 is a block diagram showing an example of the configuration of the travel area determination device according to the embodiment. FIG. 3 is an explanatory diagram showing an example of a road surface image collection according to the embodiment. FIG. 4 is an explanatory diagram showing an example of the operation of the travel area determination device according to the embodiment. FIG. 5 is a flowchart showing an example of processing executed by the control unit of the travel area determination device according to the embodiment. FIG. 6 is a block diagram showing an example of the configuration of a travel area determination device according to a modification of the embodiment.

Embodiments of a driving area determination device, a driving area determination method, and a road surface image machine learning model generation method will be described in detail below with reference to the accompanying drawings. In addition, this invention is not limited by embodiment shown below. FIG. 1 is an explanatory diagram showing an overview of the travel area determination method according to the embodiment.

A driving area determination device according to an embodiment determines a specific area in an image by analyzing an image around the vehicle captured by an imaging device mounted on the vehicle. Then, the travel area determination device determines whether or not the specific area is a suitable travel area suitable for vehicle travel.

The driving area determination device stores a road surface image machine learning model obtained by performing machine learning on a plurality of images in which it is known in advance that the subject is the road surface. The road surface image machine learning model is a program that, when an image is input, outputs a determination result that determines that an area in the image that has features similar to those of the road surface previously machine-learned is a road surface.

For example, as shown in FIG. 1A, when an image 100 with a box-shaped falling object 101 on the path of the vehicle is input from the imaging device, the driving area determination device converts the image into a road surface image machine learning model. to enter.

As shown in FIG. 1B, the road surface image machine learning model identifies an area A1 having features similar to those of the road surface machine-learned in advance in the lane in which the vehicle is traveling (including lanes in the same direction) in the image 100. It is determined to be a road surface (an example of a suitable travel area suitable for vehicle travel).

At this time, the road surface image machine learning model cannot determine what the object is in the area where the falling object 101 exists. At least it can be determined that it is not a road surface.

Therefore, as shown in FIG. 1(c), the driving area determination device detects the possibility that an obstacle exists in an area A2 other than the area A1 determined as the road surface by the road surface image machine learning model in the driving lane of the vehicle. is determined as an area (an example of an unsuitable travel area unsuitable for vehicle travel).

Area A shown in FIGS. 1(a), (b), and (c) is the road area (the road on which the vehicle is traveling (excluding the oncoming lane)) image machine learning model determines the driving lane of the vehicle in the image 100. It is an area outside the road (including lanes in the same direction), and this area is a travel-impossible area.

In addition, this area becomes a non-target area for determination by the road surface image machine learning model. In addition, the driving area determination device also sets the area above the horizon in the image 100 as a non-target area for determination by the road surface image machine learning model.

In addition, by learning and generating the road surface image machine learning model with image data that indicates the road area on which travel is possible (such as image data that does not regard the oncoming lane area as the road surface), it is possible to change to the above-described processing related to the road area. .

In this case, it is conceivable to generate a road surface image machine learning model by learning processing with image data that considers the road on the side without the inaccessible (central) divider to be the road surface. When performing control and the like, additional processing such as judging an oncoming vehicle as an obstacle, including its speed, is required.

As a result, when there is an obstacle that should not exist on the road surface, the traveling area determination device can detect the possibility of the presence of the obstacle rather than the device that detects the obstacle using the pattern matching data. It is possible to accurately determine the area A2 where

Specifically, a device that uses pattern matching data to detect obstacles detects obstacles such as vehicles and pedestrians that are expected to exist on the road surface. It is possible to determine the area where an object is.

However, sometimes there are things on the road that should not be on the road. For example, there are cases where objects such as household goods, building materials, etc., which normally should not be on the road, are dropped for some reason.

Such falling objects have a wide variety of shapes, unlike vehicles, pedestrians, and the like. Therefore, it is difficult for a device that detects an obstacle using pattern matching data to detect such a falling object as an obstacle.

It is possible to detect falling objects as obstacles by preparing pattern matching data for all items such as household goods and building materials. It is not realistic because it is necessary to prepare the data for

On the other hand, in the driving area determination method according to the embodiment, instead of specifying an obstacle, an area A2 other than the area A1 determined as the road surface by the road surface image machine learning model in the driving lane of the vehicle is identified as an obstacle. Determined as an area where an object may exist.

Therefore, according to the driving area determination method according to the embodiment, regardless of the type or shape of the obstacle, even if an object that should not normally be on the road is dropped on the road, the object can accurately determine regions where

In the above description, an area that was not judged as a road surface was judged as an obstacle. It is also possible to use it for control.

Next, with reference to FIG. 2, an example of the configuration of the travel area determination device 1 according to the embodiment will be described. FIG. 2 is a block diagram showing an example of the configuration of the travel area determination device 1 according to the embodiment. As shown in FIG. 2 , the travel area determination device 1 is connected to an imaging device 41 , a warning device 42 , an image display device 43 and a vehicle control device 44 .

Note that the road surface image collection 110 is an image collection including a plurality of images showing only various road surfaces with no obstacles. Each image included in the road surface image collection 110 is provided with information indicating that the subject is the road surface. The road surface image collection 110 is used by the driving area determination device 1 as teaching materials for supervised machine learning of road surface images. An example of the road surface image collection 110 will be described later with reference to FIG.

The imaging device 41 is, for example, an in-vehicle camera that is installed in the front part of the vehicle in which the travel area determination device 1 is mounted and captures an image of the front of the vehicle. Note that the imaging device 41 may be an in-vehicle camera that is provided at the side or rear of the vehicle and captures an image of the surroundings of the vehicle. The imaging device 41 outputs the captured image to the travel area determination device 1 .

The warning device 42 is a device that warns the user of the vehicle to that effect when the travel region determination device 1 determines a region where an obstacle may exist. It is an indicator lamp that notifies the presence of obstacles. The image display device 43 is, for example, a liquid crystal display that displays images captured by the imaging device 41 .

The vehicle control device 44 is, for example, an ECU (Electronic Control Unit) that controls the entire vehicle. For example, when the travel area determination device 1 determines an area in which an obstacle may exist, the vehicle control device 44 performs speed control, steering control, and braking control of the vehicle to detect the obstacle in the vehicle. avoid.

A travel area determination device 1 includes a control unit 2 and a storage unit 3 . The storage unit 3 is, for example, an information storage device such as data flash, and stores a road surface image machine learning model 31 . The control unit 2 includes a microcomputer having a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and various circuits.

The control unit 2 includes a learning unit 21, an acquiring unit 22, a dividing unit 23, a road surface determining unit 24, a synthesizing unit 25, a learning unit 21, an acquiring unit 22, a dividing unit 23, a road surface determining unit 24, a synthesizing unit 25, and a program stored in the ROM. An obstacle determination unit 26 , a display control unit 27 and a setting unit 28 are provided.

Note that the control unit 2 may be configured by hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). Each of the plurality of processing units provided in the control unit 2 implements or executes the action of information processing described below. Note that the internal configuration of the control unit 2 is not limited to the configuration shown in FIG. 2, and may be another configuration as long as it performs the information processing described later.

The learning unit 21 acquires a plurality of (for example, several thousand) road surface images from the road surface image collection 110, uses the road surface images as teaching materials for supervised machine learning, and generates and stores a road surface image machine learning model 31. Store in part 3.

Here, an example of the road surface image collection 110 according to the embodiment will be described with reference to FIG. FIG. 3 is an explanatory diagram showing an example of the road surface image collection 110 according to the embodiment. As shown in FIG. 3, the road surface image collection 110 includes, for example, a road surface image 111 in which a solid center line 111a is provided on the road surface, a road surface image 112 in which a broken center line 112a is provided on the road surface, and A road surface image 113 with safety zones 113a is included.

Further, the road surface image collection 110 includes, for example, a road surface image 114 in which characters 114a indicating the upper limit of the traveling speed are written on the road surface, a road surface image 115 in which a mark 115a indicating that there is a pedestrian crossing ahead, and A road surface image 116 with a pedestrian crossing 116a is included.

Further, the road surface image collection 110 includes, for example, a road surface image 117 with a manhole 117a on the road surface, a road surface image 118 with a road joint 118a on the road surface, and a road surface image 119 with a groove lid 119a on the road surface. . In this manner, the road surface image collection 110 includes road surface images 111 to 119 including various road components.

The learning unit 21 performs supervised machine learning on the features such as the color of the road surface and the color, shape, and arrangement of the road components for each of the road surface images 111 to 119 whose subject is known to be the road surface. Generate a machine learning model 31 .

Returning to FIG. 2, the description of the processing units other than the learning unit 21 included in the control unit 2 will proceed. Here, description will be made with reference to FIG. 4 as well. FIG. 4 is an explanatory diagram showing an example of the operation of the travel area determination device 1 according to the embodiment.

The acquisition unit 22 acquires an image of the surroundings of the vehicle from the imaging device 41 . Here, as shown in FIG. 4A, the acquisition unit 22 acquires an image 100 in which a box-shaped fallen object 101 is present on the path of the vehicle.

The acquisition unit 22 outputs the image 100 acquired from the imaging device 41 to the division unit 23 . As shown in FIG. 4A, the dividing unit 23 divides the image 100 input from the obtaining unit 22 into a plurality of divided regions, and outputs the image of each divided region to the road surface determination unit 24 . The road surface determination unit 24 reads the road surface image machine learning model 31 from the storage unit 3 and inputs the image of each divided area input from the dividing unit 23 to the road surface image machine learning model 31 .

The road surface image machine learning model 31 conceptually has three processing layers, an input layer 31a, an intermediate layer 31b, and an output layer 31c, as shown in FIG. 4(b). In addition, although FIG. 4A shows the case where the intermediate layer 31b is one layer, the intermediate layer 31b may be provided in plural layers.

The input layer 31a, the intermediate layer 31b, and the output layer 31c are provided with a plurality of nodes 31z for parallelly performing road surface determination of the image of each divided area. Each node 31z of the input layer 31a is connected to all the nodes 31z of the intermediate layer 31b. Each node 31z of the intermediate layer 31b is connected to all the nodes 31z of the output layer 31c. Thus, the road surface image machine learning model 31 has a neural network structure in which the nodes 31z are connected to each other.

Each node 31z of the input layer 31a determines whether or not the image of each divided region sequentially input from the dividing unit 23 is a road surface. Output to node 31z.

Similarly, each node 31z of the intermediate layer 31b determines whether or not the image of each divided area is a road surface, and outputs a determination result weighted higher to the node 31z of the output layer 31c if the image is likely to be a road surface.

Each node 31z of the output layer 31c determines whether or not the image of each divided area is a road surface, and outputs a determination result 31d. In addition, FIG. 4B shows the determination result 31d of the divided area in the second row from the bottom in the image 100. As shown in FIG.

In the example shown in FIG. 4(b), the road surface image machine learning model 31 determines that the second divided area from the bottom of the image 100 except for the images of the second and third divided areas from the left is the road surface. there is

The road surface determination unit 24 assigns the determination result 31 d by the road surface image machine learning model 31 to the image of each divided area, and outputs the image to the synthesis unit 25 . As shown in FIG. 4( c ), the synthesizing unit 25 synthesizes the images of the divided regions input from the road surface determining unit 24 , restores the image 100 before division, and outputs the image 100 to the obstacle determining unit 26 .

The obstacle determination unit 26 determines whether an obstacle exists in each divided area of the image 100 based on the determination result indicating whether or not it is a road surface given to each divided area of the combined image 100 input from the synthesizing unit 25 . Determine whether there is an area that may exist.

The obstacle determination unit 26 determines whether or not there is an area in which there is no possibility of an obstacle existing in each divided area of the image 100 based on the determination result indicating whether or not the divided area of the image 100 is a road surface. It is also determined whether In this manner, the obstacle determination unit 26 can accurately determine a region in which an obstacle may exist by performing determination for each divided region of the image 100 .

In addition, when there is an area where an obstacle may exist, the obstacle determination unit 26 determines whether the position and size of the area in the driving lane where an obstacle may exist interferes with the running of the vehicle. , the warning device 42 warns the driver of the vehicle. Then, the obstacle determination section 26 outputs the image 100 input from the synthesizing section 25 and information indicating the position of the area in the image 100 in which an obstacle may exist to the display control section 27 .

The display control unit 27 causes the image display device 43 to display the image 100 input from the obstacle determination unit 26 . Furthermore, as shown in FIG. 4(c), the display control unit 27 displays the vehicle driving lane in the image 100 other than the area A2 where the obstacle determination unit 26 determines that an obstacle may exist. The image display device 43 highlights the area A1 as a travelable area.

Accordingly, the display control unit 27 can clearly notify the user of the vehicle travelable area. After that, the display control unit 27 outputs to the setting unit 28 information indicating the position of the area in the image 100 where an obstacle may exist.

The setting unit 28 derives a travel route L1 (a route connecting travelable regions) that avoids the position of an area in which an obstacle may exist in the image 100, and transmits information indicating the derived travel route L1 to vehicle control. The information is output to the device 44 to set the travel route. Accordingly, the vehicle control device 44 can avoid the falling object 101 by controlling the vehicle to travel along the set travel route L1.

Next, an example of processing executed by the control unit 2 of the travel area determination device 1 according to the embodiment will be described with reference to FIG. 5 . FIG. 5 is a flowchart showing an example of processing executed by the control unit 2 of the travel area determination device 1 according to the embodiment. The control unit 2 repeatedly executes the processing of the flowchart shown in FIG. 5 during the operation of the driving area determination device 1 (while the vehicle is being driven, the engine is operating, etc.).

As shown in FIG. 5, the control unit 2 first acquires an image of the surroundings of the vehicle from the imaging device 41 (step S101). After that, the control unit 2 divides the acquired image into a plurality of divided areas (step S102).

Subsequently, the control unit 2 determines whether or not the image in the divided area is a road surface (step S103), and gives the determination result to each divided area (step S104). After that, the control unit 2 determines whether or not there is an undetermined divided area in the image of one frame (step S105).

If the control unit 2 determines that there is an undetermined divided area (step S105, Yes), it repeats the processing of steps S103 to S105 until there is no undetermined divided area.

If the control unit 2 determines that there is no undetermined divided area (step S105, No), it synthesizes the images of the divided areas (step S106) and returns the image to the image before division. After that, the control unit 2 determines for each divided area whether or not there is an obstacle area in which an obstacle may exist based on the determination result given to each divided area (step S107).

Then, when the controller 2 determines that there is no obstacle area (step S107, No), the process ends. Further, when the controller 2 determines that there is an obstacle area (step S107, Yes), it determines whether or not a warning is necessary (step S108).

The control unit 2 determines that a warning is not necessary when the position of the obstacle is on the road shoulder or the size (area) of the obstacle area is equal to or less than a threshold value and does not hinder the running of the vehicle. do. Then, when the controller 2 determines that the warning is not necessary (step S108, No), the process ends. If the controller 2 determines that a warning is necessary (step S108, Yes), the warning device 42 issues a warning (step S109).

Subsequently, the control unit 2 highlights the drivable area other than the obstacle area in the lane in which the vehicle is traveling in the image around the vehicle displayed by the image display device 43 (step S110). At this time, the control unit 2 can also highlight the obstacle area. At this time, the control unit 2 can also superimpose a travel route avoiding the obstacle area on the display image.

After that, the control unit 2 outputs information indicating a travel route that avoids the obstacle area to the vehicle control device 44, sets the travel route (step S111), and ends the process. The control unit 2 executes the processing of steps S101 to S110 each time an image is acquired from the imaging device 41. FIG.

Note that the processing executed by the control unit 2 is not limited to the processing shown in FIG. For example, after the process of step S101 described above, the control unit 2 determines an area outside the road (area A shown in FIG. 1 or an area above the horizon in the image 100) to limit the processing range of the road surface determination process. Alternatively, it can be used to display an area outside the road.

In the above-described embodiment, the driving area determination device 1 uses the road surface image machine learning model 31 to determine whether or not the image is the road surface. You can judge. Next, with reference to FIG. 6, the travel area determination device 1a that determines whether or not an image is a road surface by pattern matching processing will be described.

FIG. 6 is a block diagram showing an example of the configuration of a travel area determination device 1a according to a modification of the embodiment. Here, among the constituent elements shown in FIG. 6, the constituent elements that are the same as the constituent elements shown in FIG. 2 are given the same reference numerals as the constituent elements shown in FIG.

As shown in FIG. 6, the driving area determination device 1a differs from the driving area determination device 1 shown in FIG. 2 in the configuration of the control unit 2a and the information stored in the storage unit 3a. The storage unit 3 a stores the road formation image 32 .

The road structure image 32 includes, for example, the road shown in FIG. An image such as the gutter lid 119a is included.

The control unit 2a also includes an acquisition unit 22, a road surface determination unit 24a, an obstacle determination unit 26, a display control unit 27, and a setting unit . Of these plurality of processing units included in the control unit 2a, the processing units other than the road surface determination unit 24a perform the same processing as the acquisition unit 22, the obstacle determination unit 26, the display control unit 27, and the setting unit 28 shown in FIG. conduct.

The road surface determination unit 24a performs pattern matching processing using images of a plurality of types of road formations included in the road formations image 32 on the image of the surroundings of the vehicle input from the acquisition unit 22. determines the area of the road surface in the image.

Then, the road surface determination unit 24a outputs to the obstacle determination unit 26 information indicating the position of the area determined to be the road surface in the lane in which the vehicle is traveling. As a result, the obstacle determination unit 26 can determine an area other than the area determined to be the road surface by the road surface determination unit 24a in the lane in which the vehicle is traveling, as an obstacle area in which an obstacle may exist. .

As described above, the traveling area determination device 1a can determine whether or not an image is a road surface by pattern matching processing. It can be determined as an obstacle area that may exist.

In the above-described embodiment, the case where the travel area determination device 1 includes the learning unit 21 has been described, but the travel area determination device according to the embodiment does not necessarily include the learning unit 21 . The driving area determination device stores the road surface image machine learning model 31 generated by machine learning using a plurality of images known by another machine learning device that the subject is the road surface. Object judgment processing can be performed.

In this case, for example, the information processing device on the center side, which is wirelessly communicably connected to the traveling area determination device 1 mounted on the vehicle, acquires a plurality of images in which it is known that the subject is the road surface.
Then, the information processing device performs machine learning on the plurality of acquired images to generate a road surface image machine learning model (a machine learning model of a travel area in which the vehicle can safely travel), and distributes it to the travel area determination device 1. .

As a result, the travel area determination device 1 can perform the above-described obstacle determination processing by using the road surface image machine learning model distributed from the information processing device on the center side. In addition to the driving area determination device 1, the information processing device on the center side can distribute the road surface image machine learning model to any road driving support device that supports the road driving of the vehicle, so that it can be used for road driving support. .

Further effects and modifications can be easily derived by those skilled in the art. Therefore, the broader aspects of the invention are not limited to the specific details and representative embodiments so shown and described. Accordingly, various changes may be made without departing from the spirit or scope of the general inventive concept defined by the appended claims and equivalents thereof.

Reference Signs List 1, 1a running area determination device 2, 2a control unit 21 learning unit 22 acquisition unit 23 division unit 24, 24a road surface determination unit 25 synthesis unit 26 obstacle determination unit 27 display control unit 28 setting unit 3, 3a storage unit 31 road surface image Machine learning model 32 Road formation image 41 Imaging device 42 Warning device 43 Image display device 44 Vehicle control device

Claims (8)

an acquisition unit that acquires an image of the surroundings of the vehicle from the imaging device;
A determination unit that determines that a specific area in the image acquired by the acquisition unit is a road surface by a road surface machine learning model machine-learned using only a plurality of known road surface images,
The travel area determination device, wherein the determination unit determines an area that is not determined to be a road surface as an unsuitable travel area for the vehicle to travel. The driving region determination device according to claim 1, further comprising: a dividing unit that divides the image acquired by the acquiring unit into a plurality of pieces to form the specific regions. The determination unit is
The travel area determination device according to claim 1 or 2, wherein the specific area determined as a road surface is determined as a suitable travel area. The determination unit is
4. The driving area determination device according to claim 1, wherein the specific area is determined for an area within the driving lane of the vehicle in the image acquired by the acquisition unit. 4. A display control unit that causes a display device to highlight an area determined by the determining unit to be a suitable driving area in the driving lane of the vehicle in the image acquired by the acquiring unit, further comprising a display control unit. 2. The traveling area determination device according to 1. 6. The travel area determination device according to claim 3, further comprising a setting unit that sets the suitable travel area in the image acquired by the acquisition unit as a travel route of the vehicle. an acquisition step of acquiring an image in which the surroundings of the vehicle are captured from the imaging device;
A determination step of determining that a specific area in the image acquired by the acquisition step is a road surface by a road surface machine learning model machine -learned using only a plurality of known road surface images,
The driving area determining method, wherein the determining step determines an area not determined to be a road surface as an unsuitable driving area unsuitable for driving the vehicle. A method for generating a road surface image machine learning model for detecting a road surface obstacle that is an unsuitable driving area for a vehicle used by a road driving support device that performs road driving support,
A method of generating a road surface image machine learning model, comprising: generating a road surface image machine learning model by machine learning only the road surface from a plurality of images known to be road surfaces as objects.

Images