JP6795379B2 - Operation control device, operation control method and operation control program - Google Patents

Description

The present disclosure relates to a driving control device, a driving control method, and a driving control program that control the driving of a vehicle by either an automatic driving mode in which the vehicle is automatically driven or a manual driving mode in which the vehicle is manually driven by the driver. is there.

In recent years, the development of technology for automatically driving a vehicle has been promoted. In an environment where it is difficult to drive a vehicle automatically, it is preferable to drive the vehicle in a manual driving mode in which the vehicle is manually driven by the driver, instead of driving in the automatic driving mode in which the vehicle is automatically driven.

For example, in Patent Document 1, it is determined whether or not the control computer can be safely controlled, and when it is determined that the control computer can be safely controlled, the control computer is automatically operated and the control computer is operated. It is manually operated when it is determined that it cannot be controlled safely.

U.S. Pat. No. 8352110

However, in the above-mentioned conventional technique, further improvement is required.

The driving control device according to one aspect of the present disclosure is a driving control device that controls the driving of the vehicle by either an automatic driving mode in which the vehicle is automatically driven or a manual driving mode in which the vehicle is manually driven by the driver. A map storage unit that stores an object appearance map that defines an appearance area in which an object may appear, and an object detection unit that detects the object included in a captured image of the traveling direction of the vehicle. A switching unit that switches from the automatic driving mode to the manual driving mode based on the detection result of the object and the object appearance map, and a driving control unit that controls the driving of the vehicle by the switched manual driving mode. To be equipped.

According to the present disclosure, it is possible to switch from the automatic driving mode in which the vehicle is automatically driven to the manual driving mode in which the vehicle is manually driven by the driver, depending on the detection accuracy of the object.

It is a block diagram which shows the structure of the vehicle control system which concerns on Embodiment 1. It is a flowchart for demonstrating the operation of the operation control apparatus in Embodiment 1. It is a schematic diagram for demonstrating the process of switching from the automatic operation mode to the manual operation mode in Embodiment 1. It is a figure which shows an example of the photographed image and the appearance likelihood map image in the modification of Embodiment 1 of this embodiment. FIG. 5 is a schematic diagram for explaining an example in which two objects of the same type that are not detected at the same time are detected in the appearance region in the first embodiment. FIG. 5 is a schematic diagram for explaining an example in which two different types of objects that are not detected at the same time are detected in the appearance region in the first embodiment. It is a block diagram which shows the structure of the vehicle control system which concerns on Embodiment 1. It is a flowchart for demonstrating operation of the operation control apparatus in Embodiment 2 of this invention. It is a block diagram which shows the structure of the vehicle control system which concerns on embodiment 3. It is a flowchart for demonstrating operation of the operation control apparatus in Embodiment 3 of this invention. It is a schematic diagram for demonstrating the process of switching from the automatic operation mode to the manual operation mode in Embodiment 3.

(Knowledge on which this disclosure was based)
As described above, in Patent Document 1, it is determined whether or not the control computer can be safely controlled, and when it is determined that the control computer can be safely controlled, the automatic operation is performed. It is manually operated when the control computer determines that it cannot be safely controlled. For example, the control computer of Patent Document 1 determines that it cannot be safely controlled if there is no history of traveling in the past. In addition, the control computer determines that it cannot be safely controlled when a plurality of obstacles approaching the vehicle are detected.

In Patent Document 1, the condition for determining that the state of the external environment cannot be safely controlled is defined by human knowledge. However, since the judgment criteria differ between the recognition system that recognizes obstacles from the external environment and humans, there is an external environment that is difficult for the recognition system to recognize even if it is an external environment that can be easily recognized by humans. Therefore, in automatic operation, it is necessary to consider the behavior and performance of the recognition system with respect to the state of the external environment.

In order to solve the above problems, the driving control device according to one aspect of the present disclosure is the vehicle in either an automatic driving mode in which the vehicle is automatically driven or a manual driving mode in which the vehicle is manually driven by the driver. A driving control device that controls the driving of the vehicle, the map storage unit that stores an object appearance map that defines an appearance area in which an object may appear, and the photographed image that captures the traveling direction of the vehicle. An object detection unit that detects an object, a switching unit that switches from the automatic driving mode to the manual driving mode based on the detection result of the object and the object appearance map, and the switched manual driving mode of the vehicle. It is provided with an operation control unit that controls operation.

According to this configuration, an object appearance map that defines an appearance area in which an object may appear is stored in the map storage unit. Objects included in the captured image of the traveling direction of the vehicle are detected. Based on the object detection result and the object appearance map, the automatic operation mode can be switched to the manual operation mode. The driving of the vehicle is controlled by the switched manual driving mode.

Therefore, based on the detection result of the object and the object appearance map that defines the appearance area where the object may appear, it is determined whether or not to switch from the automatic operation mode to the manual operation mode. Depending on the accuracy, it is possible to switch from the automatic driving mode in which the vehicle is automatically driven to the manual driving mode in which the vehicle is manually driven by the driver.

Further, in the above-mentioned operation control device, the object appearance map may be managed in association with the map information.

According to this configuration, since the object appearance map is managed in association with the map information, it is possible to define the appearance area where the object on the map may appear.

Further, in the above-mentioned operation control device, the object appearance map is combined with the photographed image according to the traveling environment acquisition unit that acquires the traveling environment around the vehicle and the traveling environment acquired by the traveling environment acquisition unit. The switching unit further includes a generation unit that generates an object appearance map after the viewpoint conversion viewed from the same viewpoint, and the switching unit superimposes the object appearance map after the viewpoint conversion on the captured image and includes the captured image. When the position where the object is detected is outside the appearance area included in the object appearance map after the viewpoint conversion, the automatic operation mode may be switched to the manual operation mode.

According to this configuration, the traveling environment around the vehicle is acquired. According to the acquired driving environment, the object appearance map after the viewpoint conversion is generated when the object appearance map is viewed from the same viewpoint as the captured image. When the object appearance map after viewpoint conversion is superimposed on the captured image and the position where the object included in the captured image is detected is outside the appearance area included in the object appearance map after viewpoint conversion, manual operation is performed from the automatic operation mode. You can switch to the operation mode.

Therefore, if the position where the object included in the captured image is detected is outside the appearance area included in the object appearance map after the viewpoint conversion, it can be determined that the object is not accurately detected. In this case, By switching from the automatic driving mode to the manual driving mode, the driver can drive the vehicle.

Further, in the above-mentioned operation control device, the map includes a three-dimensional map represented by a three-dimensional coordinate system, and the appearance area is an appearance space in which the object may appear on the three-dimensional map. Including, the generation unit may convert the object appearance map in the three-dimensional coordinate system into an object appearance map after the viewpoint conversion in the two-dimensional coordinate system viewed from the same viewpoint as the captured image.

According to this configuration, the map includes a three-dimensional map represented by a three-dimensional coordinate system, and the appearance area includes an appearance space in which an object on the three-dimensional map may appear. The object appearance map of the three-dimensional coordinate system is converted into the object appearance map after the viewpoint conversion of the two-dimensional coordinate system viewed from the same viewpoint as the captured image.

Therefore, since the object appearance map defines the appearance space in which the object may appear on the three-dimensional map, the range of the appearance space in which the object may appear can be defined more accurately. The accuracy of switching from the automatic operation mode to the manual operation mode can be improved.

Further, in the above-mentioned operation control device, the map storage unit is represented by a plurality of two-dimensional coordinate systems having the same viewpoint as the captured image, in which the appearance areas are defined according to the shapes of the plurality of roads. The driving environment acquisition unit that stores the object appearance map of the vehicle and acquires the shape of the road included in the photographed image of the traveling direction of the vehicle, and the object acquisition map acquired from the plurality of object appearance maps. It may further include a determination unit that determines one object appearance map according to the shape of the road.

According to this configuration, the map storage unit is a plurality of object appearance maps represented by a two-dimensional coordinate system having the same viewpoint as the captured image, in which the appearance areas are defined according to the shapes of the plurality of roads. I remember. The shape of the road included in the photographed image of the traveling direction of the vehicle is acquired. From the plurality of object appearance maps, one object appearance map according to the acquired road shape is determined.

Therefore, instead of the map data, a plurality of object appearance maps represented by a two-dimensional coordinate system, which is the same viewpoint as the captured image and whose appearance areas are defined according to the shapes of a plurality of roads, are stored in the memory. Therefore, the amount of data stored in the memory can be reduced.

Further, in the above-mentioned operation control device, the map includes a three-dimensional map represented by a three-dimensional coordinate system, and the appearance area is an appearance space in which the object may appear on the three-dimensional map. Including, the map storage unit stores a plurality of object appearance maps each represented by a three-dimensional coordinate system in which the appearance space is defined according to the shape of the plurality of roads, and the traveling direction of the vehicle. A determination to determine one object appearance map according to the acquired shape of the road from the traveling environment acquisition unit that acquires the shape of the road included in the photographed image and the plurality of object appearance maps. When the unit and the determination unit determine the object appearance map of 1, the coordinate system that converts the object appearance map of 1 in the 3D coordinate system into an object appearance map of the same 2D coordinate system as the captured image. A conversion unit may be further provided.

According to this configuration, the map includes a three-dimensional map represented by a three-dimensional coordinate system, and the appearance area includes an appearance space in which an object on the three-dimensional map may appear. The map storage unit stores a plurality of object appearance maps represented by a three-dimensional coordinate system in which appearance spaces are defined according to the shapes of the plurality of roads. The shape of the road included in the photographed image of the traveling direction of the vehicle is acquired. From the plurality of object appearance maps, one object appearance map according to the acquired road shape is determined. When the object appearance map of 1 is determined, the object appearance map of 1 in the 3D coordinate system is converted into the object appearance map of the same 2D coordinate system as the captured image.

Therefore, since each of the plurality of object appearance maps defines the appearance space in which the object may appear on the three-dimensional map, it is possible to more accurately define the range of the appearance space in which the object may appear. It is possible to improve the accuracy of switching from the automatic operation mode to the manual operation mode.

Further, in the above-mentioned operation control device, the map storage unit is represented by a plurality of two-dimensional coordinate systems having the same viewpoint as the captured image, in which the appearance areas are defined according to the shapes of the plurality of roads. The object appearance map of the above is stored, and one object appearance map is determined from the plurality of object appearance maps according to the identification number storage unit and the position where the object included in the captured image is detected. Further includes a determination unit for storing the determined identification number corresponding to the determined object appearance map in the identification number storage unit, and the switching unit stores the identification number in the identification number storage unit within a predetermined time. When the identification number is switched a predetermined number of times, the automatic operation mode may be switched to the manual operation mode.

According to this configuration, the map storage unit stores a plurality of object appearance maps represented by a two-dimensional coordinate system having the same viewpoint as the captured image, in which appearance areas are defined according to the shapes of the plurality of roads. doing. One object appearance map is determined from a plurality of object appearance maps according to the position where the object included in the captured image is detected, and the identification number corresponding to the determined one object appearance map is the identification number storage unit. Is remembered in. When the identification number stored in the identification number storage unit is switched a predetermined number of times within a predetermined time, the automatic operation mode is switched to the manual operation mode.

Therefore, if the object appearance map changes in a short time, it can be determined that the object cannot be detected accurately. In this case, the driver can drive the vehicle by switching from the automatic driving mode to the manual driving mode.

Further, in the above-mentioned operation control device, the object may be a person, and the appearance area may be an area through which the person passes.

According to this configuration, it is possible to switch from the automatic driving mode in which the vehicle is automatically driven to the manual driving mode in which the vehicle is manually driven by the driver, depending on the detection accuracy of the person.

The driving control method according to another aspect of the present disclosure is a driving control device that controls the driving of the vehicle by either an automatic driving mode in which the vehicle is automatically driven or a manual driving mode in which the vehicle is manually driven by the driver. The driving control method in the above, which detects the object included in the photographed image of the traveling direction of the vehicle, and defines the detection result of the object and the appearance area where the object may appear. Based on the above, the automatic driving mode is switched to the manual driving mode, and the driving of the vehicle is controlled by the switched manual driving mode.

According to this configuration, an object included in a captured image of the traveling direction of the vehicle is detected. The automatic operation mode can be switched to the manual operation mode based on the detection result of the object and the object appearance map that defines the appearance area where the object may appear. The driving of the vehicle is controlled by the switched manual driving mode.

Therefore, based on the detection result of the object and the object appearance map that defines the appearance area where the object may appear, it is determined whether or not to switch from the automatic operation mode to the manual operation mode. Depending on the accuracy, it is possible to switch from the automatic driving mode in which the vehicle is automatically driven to the manual driving mode in which the vehicle is manually driven by the driver.

The driving control program according to another aspect of the present disclosure is a driving control program that controls the driving of the vehicle by either an automatic driving mode in which the vehicle is automatically driven or a manual driving mode in which the vehicle is manually driven by the driver. A map storage unit that stores an object appearance map that defines an appearance area in which an object may appear, and an object detection unit that detects the object included in a captured image of the traveling direction of the vehicle. As a switching unit that switches from the automatic driving mode to the manual driving mode based on the detection result of the object and the object appearance map, and as a driving control unit that controls the driving of the vehicle by the switched manual driving mode. Make your computer work.

According to this configuration, an object appearance map that defines an appearance area in which an object may appear is stored in the map storage unit. Objects included in the captured image of the traveling direction of the vehicle are detected. Based on the object detection result and the object appearance map, the automatic operation mode can be switched to the manual operation mode. The driving of the vehicle is controlled by the switched manual driving mode.

Therefore, based on the detection result of the object and the object appearance map that defines the appearance area where the object may appear, it is determined whether or not to switch from the automatic operation mode to the manual operation mode. Depending on the accuracy, it is possible to switch from the automatic driving mode in which the vehicle is automatically driven to the manual driving mode in which the vehicle is manually driven by the driver.

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. The following embodiments are examples that embody the present disclosure, and do not limit the technical scope of the present disclosure.

(Embodiment 1)
FIG. 1 is a block diagram showing a configuration of a vehicle control system according to the first embodiment. The vehicle control system shown in FIG. 1 includes a driving control device 1 and a camera 2. The driving control device 1 controls the driving of the vehicle by either an automatic driving mode in which the vehicle is automatically driven or a manual driving mode in which the vehicle is manually driven by the driver. The vehicle is an automobile, but the present disclosure is not particularly limited to this, and the vehicle may be various vehicles such as motorcycles, trucks, buses, trains, and flying objects.

The camera 2 is arranged near the rear-view mirror of the vehicle and takes an image of the front of the vehicle. The camera 2 may capture an image of the traveling direction of the vehicle such as the rear of the vehicle as well as the front of the vehicle.

The driving control device 1 is arranged in the vehicle. The operation control device 1 includes a memory 3 and a processor 4.

The memory 3 is a computer-readable recording medium, for example, a hard disk drive, a ROM (Read Only Memory), a RAM (Random Access Memory), an optical disk, a semiconductor memory, and the like. The memory 3 stores an operation control program executed by the processor 4. Further, the memory 3 includes an appearance likelihood map storage unit 13.

The appearance likelihood map storage unit 13 is an example of a map storage unit, and stores an appearance likelihood map (object appearance map) that defines an appearance area in which an object may appear. The appearance likelihood map is managed in association with the map information. The appearance likelihood map in the first embodiment defines an appearance area in which an object may appear on the two-dimensional map. The object is, for example, a person or a vehicle, and the appearance likelihood map individually shows an appearance area through which a person passes and an appearance area through which a vehicle passes.

The appearance likelihood map is stored in advance in the appearance likelihood map storage unit 13, but the present disclosure is not particularly limited to this, and the appearance likelihood map is acquired from a server via a communication unit (not shown). It may be stored in the storage unit 13.

In addition, the appearance area of the appearance likelihood map may be statistically calculated from the detection results of past persons or vehicles. Further, the appearance area of the appearance likelihood map may be assigned by the operator on the two-dimensional map, or may be assigned to the area where a person or a vehicle can pass according to a predetermined rule such as the Road Traffic Act. ..

The processor 4 is, for example, a CPU (Central Processing Unit) and executes an operation control program stored in the memory 3. The processor 4 includes an object detection unit 11, a driving environment acquisition unit 12, an appearance likelihood map generation unit 14, an operation mode switching unit 15, and a vehicle control unit 16.

The object detection unit 11 detects an object included in a captured image obtained by photographing the traveling direction of the vehicle by the camera 2. The object detection unit 11 detects a person or a vehicle included in a captured image obtained by photographing the traveling direction of the vehicle by the camera 2. The object detection unit 11 detects a person or a vehicle included in the captured image, for example, by inputting the captured image into the machine-learned neural network. The detection target by the object detection unit 11 is not limited to a person or a vehicle.

The traveling environment acquisition unit 12 acquires the traveling environment around the vehicle. The traveling environment acquisition unit 12 recognizes the lane drawn on the road from the photographed image of the traveling direction of the vehicle by the camera 2, and acquires the shape of the traveling road. In addition, the traveling environment acquisition unit 12 recognizes the sign and the traffic light from the captured image, and acquires the sign information and the traffic light information. In addition, the traveling environment acquisition unit 12 acquires the traveling direction of the vehicle, the speed of the vehicle, and the current position of the vehicle. In addition, the driving environment acquisition unit 12 may estimate the position and orientation of the own vehicle on the three-dimensional map by using a sensor other than the camera (for example, LIDAR (Laser Imaging Detection and Ranking)). The current position can be acquired from a GPS (Global Positioning System) (not shown). In addition, the traveling direction can be calculated by acquiring the positions of two points by GPS.

The appearance likelihood map generation unit 14 is an example of the generation unit, and the appearance likelihood map image obtained by viewing the appearance likelihood map from the same viewpoint as the captured image is obtained according to the driving environment acquired by the driving environment acquisition unit 12. Generate. In the present embodiment, the appearance likelihood map generation unit 14 generates an appearance likelihood map image, but the present disclosure is not particularly limited to this, and the appearance likelihood is a two-dimensional matrix corresponding to the viewpoint. It may represent a degree map. The appearance likelihood map generation unit 14 specifies which position on the appearance likelihood map and in which direction the vehicle is traveling from the current position of the vehicle acquired by the driving environment acquisition unit 12 and the traveling direction of the vehicle. can do. Further, the appearance likelihood map generation unit 14 specifies the position of the viewpoint of the camera 2 and the shooting angle from the position of the lane acquired by the traveling environment acquisition unit 12. The traveling environment acquisition unit 12 may acquire the position and shooting angle of the viewpoint of the camera 2 stored in advance. Then, the appearance likelihood map generation unit 14 generates an appearance likelihood map image viewed from the same viewpoint as the captured image in the traveling direction from the current position of the vehicle on the appearance likelihood map.

The operation mode switching unit 15 is an example of the switching unit, and switches from the automatic operation mode to the manual operation mode based on the detection result of the object and the appearance likelihood map. The operation mode switching unit 15 superimposes the appearance likelihood map image on the captured image, and automatically operates when the position where the object included in the captured image is detected is outside the appearance region included in the appearance likelihood map image. Switch from mode to manual operation mode.

The vehicle control unit 16 controls the operation of the vehicle by either an automatic driving mode in which the vehicle is automatically driven or a manual driving mode in which the vehicle is manually driven by the driver. The vehicle control unit 16 controls the operation of the vehicle by the manual operation mode switched by the operation mode switching unit 15.

Subsequently, the operation of the operation control device according to the first embodiment will be described.

FIG. 2 is a flowchart for explaining the operation of the operation control device according to the first embodiment.

First, in step S1, the object detection unit 11 detects a person included in the captured image obtained by capturing the traveling direction of the vehicle by the camera 2. In the following description, an example of detecting a person will be described. The object detection unit 11 detects a person included in the captured image by image recognition. The object detection unit 11 may detect not only a pedestrian but also a person riding a bicycle.

Next, in step S2, the traveling environment acquisition unit 12 acquires the traveling environment around the vehicle. As described above, the traveling environment acquisition unit 12 recognizes the lane drawn on the road from the photographed image taken by the camera 2 and acquires the shape of the traveling road. In addition, the traveling environment acquisition unit 12 acquires the traveling direction of the vehicle and the current position of the vehicle.

Next, in step S3, the appearance likelihood map generation unit 14 obtains an appearance likelihood map image obtained by viewing the appearance likelihood map from the same viewpoint as the captured image according to the driving environment acquired by the driving environment acquisition unit 12. Generate.

Next, in step S4, the operation mode switching unit 15 superimposes the appearance likelihood map image on the captured image, and the position where the person included in the captured image is detected is the appearance region included in the appearance likelihood map image. Determine if it is outside. Here, when it is determined that the position where the person is detected is outside the appearance area (YES in step S4), in step S5, the operation mode switching unit 15 switches from the automatic operation mode to the manual operation mode.

On the other hand, when it is determined that the position where the person is detected is not outside the appearance area (NO in step S4), in step S6, the operation mode switching unit 15 maintains the current automatic operation mode.

In the manual operation mode, if it is determined that the position where the person is detected is not outside the appearance area, the operation mode switching unit 15 may switch from the manual operation mode to the automatic operation mode.

Further, when switching from the automatic operation mode to the manual operation mode, the operation mode switching unit 15 may notify the driver to switch from the automatic operation mode to the manual operation mode. In this case, the operation mode switching unit 15 may notify by voice or an image, for example. Further, the operation mode switching unit 15 may switch from the automatic operation mode to the manual operation mode after notifying the driver that the automatic operation mode is to be switched to the manual operation mode.

In addition, although FIG. 2 describes an example of determining whether or not to switch from the automatic operation mode to the manual operation mode for each frame, the manual operation is performed from the automatic operation mode based on the determination results of a plurality of frames. It may be determined whether or not to switch to the mode.

FIG. 3 is a schematic diagram for explaining the process of switching from the automatic operation mode to the manual operation mode in the first embodiment.

The operation mode switching unit 15 superimposes the appearance likelihood map image 102 on the captured image 101. The appearance likelihood map image 102 includes an appearance area 103 in which a person is more likely to appear than the other areas. In the example shown in FIG. 3, the appearance region 103 is formed on a pedestrian road outside the roadway. In FIG. 3, the appearance region 103 is represented by an elliptical shape, but the present disclosure is not particularly limited to this, and any shape may be used.

In FIG. 3, the object detection unit 11 detects a person and erroneously detects a telephone pole as a person. At this time, the person detection positions 111 exist in the appearance area 103, but the utility pole detection positions 112 and 113 exist outside the appearance area 103. Therefore, the operation mode switching unit 15 determines that the detection positions 112 and 113 in which the person included in the captured image 101 is detected are outside the appearance area 103 included in the appearance likelihood map image 102, and starts from the automatic operation mode. Switch to manual operation mode.

In FIG. 3, two detection positions detected as a person by the object detection unit 11 exist outside the appearance area 103, but when at least one detection position exists outside the appearance area 103, the detection position is automatic. You can switch from the operation mode to the manual operation mode.

In this way, when a person is detected outside the appearance area 103 where a person may appear, there is a high possibility that the object detection unit 11 does not correctly detect the person. Therefore, the vehicle is operated in the automatic operation mode as it is. Judging that it is difficult, switch from the automatic operation mode to the manual operation mode. As a result, it is possible to switch from the automatic driving mode in which the vehicle is automatically driven according to the detection accuracy of the object to the manual driving mode in which the vehicle is manually driven by the driver, instead of the automatic driving mode in which the detection accuracy of the object is low. , You can let the driver drive the vehicle.

The appearance probability map described above defines an appearance area in which an object may appear on a two-dimensional map, but the present disclosure is not particularly limited to this, and the appearance probability map is 3 You may define an appearance space in which an object may appear on a dimensional map. In this case, the appearance likelihood map generation unit 14 views the appearance likelihood map of the three-dimensional coordinate system from the same viewpoint as the captured image according to the driving environment acquired by the driving environment acquisition unit 12. Converts to an appearance likelihood map image of.

In this way, the appearance likelihood map defines the appearance space in which the object may appear on the three-dimensional map, so that the range of the appearance space in which the object may appear should be defined more accurately. It is possible to improve the accuracy of switching from the automatic operation mode to the manual operation mode.

FIG. 4 is a diagram showing an example of a captured image and an appearance likelihood map image in the modified example of the first embodiment.

As shown in FIG. 4, the appearance likelihood map storage unit 13 has a plurality of appearance regions represented by a two-dimensional coordinate system having the same viewpoint as the captured image, in which appearance regions are defined according to the shapes of the plurality of roads. The appearance likelihood map image may be stored. For example, in the appearance likelihood map image 102a, appearance regions 103 are set on both sides of the roadway, and in the appearance likelihood map image 102b, appearance regions 103 are set on the left side of the roadway, and the appearance likelihood map image 102c. In, the appearance area 103 is set at the end of the T-shaped road.

In the modified example of the first embodiment, the operation control device 1 includes an appearance likelihood map determination unit instead of the appearance likelihood map generation unit 14.

The traveling environment acquisition unit 12 recognizes the lane drawn on the road from the photographed image taken by the camera 2, and acquires the shape of the road included in the photographed image of the traveling direction of the vehicle. The appearance likelihood map determination unit has one appearance likelihood according to the shape of the road acquired by the driving environment acquisition unit 12 from among a plurality of appearance likelihood map images stored in the appearance likelihood map storage unit 13. Determine the degree map image. That is, in FIG. 4, the appearance likelihood map image 102a that matches the road shape of the captured image 101a is selected, the appearance likelihood map image 102b that matches the road shape of the captured image 101b is selected, and the road shape of the captured image 101c. The appearance likelihood map image 102c that matches is selected.

In this way, instead of map data, a plurality of appearance likelihood map images represented by a two-dimensional coordinate system having the same viewpoint as the captured image, in which appearance areas are defined according to the shapes of a plurality of roads, are stored in memory. By storing in 3, the amount of data stored in memory 3 can be reduced.

In the first embodiment, when it is determined that the position where the person included in the captured image 101 is detected is outside the appearance area 103 included in the appearance likelihood map image 102, the automatic operation mode is manually performed. Although switching to the operation mode, the present disclosure is not particularly limited to this, and when two objects of the same type that are not detected at the same time are detected in the appearance area, the automatic operation mode is changed to the manual operation mode. You may switch to.

FIG. 5 is a schematic diagram for explaining an example in which two objects of the same type that are not detected at the same time are detected in the appearance region in the first embodiment.

In FIG. 5, the appearance likelihood map image 102 is superimposed on the captured image 101, and the object detection unit 11 detects the vehicle. The appearance likelihood map image 102 shows the appearance area 104a formed on the roadway 131 in the direction of travel of the vehicle and the appearance area 104b formed on the roadway 132 extending to the left from the intersection of the roadway 131 in the direction of travel of the vehicle. And the appearance region 104c formed on the roadway 133 extending to the right from the intersection of the roadway 131 in the traveling direction of the vehicle. Appearance areas 104a, 104b, 104c are areas where vehicles may appear.

The detection positions 121 and 122 by the object detection unit 11 correctly detect the vehicle, respectively, and the detection position 123 erroneously detects a part on the roadway 132 as a vehicle instead of the vehicle.

Here, the traveling environment acquisition unit 12 recognizes the color of the traffic light 135 from the captured image 101 captured by the camera 2. As a result, the traveling environment acquisition unit 12 acquires the traffic light information that the color of the traffic light 135 is blue. Originally, the detection positions 121, 122, and 123 by the object detection unit 11 are all within the appearance areas 104a and 104b, so that the operation mode switching unit 15 does not switch from the automatic operation mode to the manual operation mode, and is currently Maintain the automatic operation mode of.

However, since the traffic light information acquired by the traveling environment acquisition unit 12 is blue, it is considered that there is no vehicle entering the intersection from the roadway 132. Therefore, when the driving mode switching unit 15 determines that the vehicle detection positions 121, 122, 123 included in the captured image 101 are within the two appearance regions 104a, 104b included in the appearance likelihood map image 102, It is determined whether or not the traffic light information is blue, and if it is determined that the traffic light information is blue, the automatic operation mode is switched to the manual operation mode.

In this way, when two vehicles of the same type that are not detected at the same time are detected in the appearance area, the automatic driving mode can be switched to the manual driving mode.

Further, in the first embodiment, when two different types of objects that are not detected at the same time are detected in the appearance region, the automatic operation mode may be switched to the manual operation mode.

FIG. 6 is a schematic diagram for explaining an example in which two different types of objects that are not detected at the same time are detected in the appearance region in the first embodiment.

In FIG. 6, the appearance likelihood map image 102 is superimposed on the captured image 101, and the object detection unit 11 detects a person and a vehicle which are two different types of objects. The appearance likelihood map image 102 shows the appearance area 104a formed on the roadway 131 in the direction of travel of the vehicle and the appearance area 104b formed on the roadway 132 extending to the left from the intersection of the roadway 131 in the direction of travel of the vehicle. And the appearance region 104c formed on the roadway 133 extending to the right from the intersection of the roadway 131 in the traveling direction of the vehicle. Appearance areas 104a, 104b, 104c are areas where vehicles may appear.

Further, the appearance likelihood map image 102 includes appearance regions 103 formed on both sides of the roadway 131, 132, 133. The appearance area 103 is an area where a person may appear.

The detection positions 121 and 122 by the object detection unit 11 correctly detect the vehicle, respectively, and the detection position 111 erroneously detects a part on the roadway 131 as a person instead of a person.

Here, the appearance area 104a is an area where a vehicle may appear, not an area where a person may appear. Therefore, in the appearance area 104a, the vehicle and the person are not detected at the same time. Therefore, the driving mode switching unit 15 determines that the vehicle detection positions 121 and 122 included in the captured image 101 and the person detection positions 111 are within the vehicle appearance region 104a included in the appearance likelihood map image 102. If it is determined, the automatic operation mode is switched to the manual operation mode.

In this way, when two different types of vehicles that are not detected at the same time are detected in the appearance area, the automatic driving mode can be switched to the manual driving mode.

(Embodiment 2)
In the first embodiment described above, the appearance likelihood map storage unit 13 stores an appearance probability map that defines an appearance area in which an object on a two-dimensional map may appear. In the second form, the appearance likelihood map storage unit 13 stores an appearance probability map that defines an appearance space in which an object on a three-dimensional map may appear.

FIG. 7 is a block diagram showing a configuration of a vehicle control system according to the first embodiment. The vehicle control system shown in FIG. 7 includes a driving control device 1 and a camera 2. The operation control device 1 includes a memory 3 and a processor 4. The memory 3 includes an appearance likelihood map storage unit 17.

The appearance likelihood map storage unit 17 is an example of a map storage unit, and is a plurality of appearance likelihood maps (models) represented by a three-dimensional coordinate system in which appearance regions are defined according to the shapes of a plurality of roads. ) Is memorized. The plurality of appearance likelihood maps in the second embodiment define an appearance space in which an object on a three-dimensional map may appear. The object is, for example, a person or a vehicle, and the appearance likelihood map individually shows an appearance area through which a person passes and an appearance area through which a vehicle passes. Further, the appearance space is represented by a three-dimensional space instead of a two-dimensional space.

Although a plurality of appearance likelihood maps are stored in advance in the appearance likelihood map storage unit 17, the present disclosure is not particularly limited to this, and the appearance likelihood maps are acquired from a server via a communication unit (not shown) and appear likelihood. It may be stored in the degree map storage unit 17.

Further, the appearance space of the plurality of appearance likelihood maps may be statistically calculated from the detection results of past persons or vehicles. Further, the appearance space of the plurality of appearance likelihood maps may be assigned by the operator on the three-dimensional map, and is assigned to the space where a person or a vehicle can pass according to a predetermined rule such as the Road Traffic Act. May be good.

The processor 4 is, for example, a CPU, and executes an operation control program stored in the memory 3. The processor 4 includes an object detection unit 11, a traveling environment acquisition unit 12, an appearance likelihood map determination unit 22, a driving mode switching unit 15, a vehicle control unit 16, and a coordinate system conversion unit 18. In the second embodiment, the same components as those in the first embodiment are designated by the same reference numerals, and the description thereof will be omitted.

The appearance likelihood map determination unit 22 has one appearance likelihood according to the shape of the road acquired by the driving environment acquisition unit 12 from among a plurality of appearance likelihood maps stored in the appearance likelihood map storage unit 17. Determine the degree map.

When the appearance probability map of 1 is determined by the appearance probability map determination unit 22, the coordinate system conversion unit 18 uses the appearance probability map of 1 in the three-dimensional coordinate system as the appearance of the same two-dimensional coordinate system as the captured image. Convert to a likelihood map image.

Subsequently, the operation of the operation control device according to the second embodiment will be described.

FIG. 8 is a flowchart for explaining the operation of the operation control device according to the second embodiment.

Since the processing of steps S11 to S12 is the same as the processing of steps S1 to S2 shown in FIG. 2, the description thereof will be omitted.

Next, in step S13, the appearance likelihood map determination unit 22 has the shape of the road acquired by the traveling environment acquisition unit 12 from among the plurality of appearance likelihood maps stored in the appearance likelihood map storage unit 17. The appearance likelihood map of 1 is determined according to.

Next, in step S14, the coordinate system conversion unit 18 uses the appearance probability map of 1 in the three-dimensional coordinate system determined by the appearance probability map determination unit 22 as the appearance probability of the same two-dimensional coordinate system as the captured image. Convert to a map image. That is, the appearance likelihood map of 1 determined by the appearance likelihood map determination unit 22 is represented by a three-dimensional coordinate system. The coordinate system conversion unit 18 coordinates the appearance likelihood map of 1 represented by the three-dimensional coordinate system into an appearance likelihood map image represented by the two-dimensional viewpoint coordinate system based on the viewpoint of the camera 2. ..

Since the processes of steps S15 to S17 are the same as the processes of steps S4 to S6 shown in FIG. 2, the description thereof will be omitted.

In this way, the appearance space in which an object may appear is represented by a three-dimensional space instead of a two-dimensional space, so that the range of the appearance space can be defined more accurately, and the range of the appearance space can be defined manually from the automatic operation mode. The accuracy of switching to the operation mode can be improved.

(Embodiment 3)
In the second embodiment described above, the appearance likelihood map image is determined according to the driving environment, but in the third embodiment, the appearance likelihood map image is determined according to the position where the object is detected and appears. When the likelihood map image is switched a predetermined number of times within a predetermined time, the automatic operation mode is switched to the manual operation mode.

FIG. 9 is a block diagram showing the configuration of the vehicle control system according to the third embodiment. The vehicle control system shown in FIG. 9 includes a driving control device 1 and a camera 2. The operation control device 1 includes a memory 3 and a processor 4. The memory 3 includes an appearance likelihood map storage unit 13 and an identification number storage unit 19. In the third embodiment, the same components as those in the first embodiment are designated by the same reference numerals, and the description thereof will be omitted.

The appearance likelihood map storage unit 13 stores a plurality of appearance likelihood map images represented by a two-dimensional coordinate system having the same viewpoint as the captured image, in which appearance areas are defined according to the shapes of the plurality of roads. ing. An identification number is assigned to each of the plurality of appearance likelihood map images.

The identification number storage unit 19 stores an identification number for identifying the appearance likelihood map image determined by the optimum appearance likelihood map determination unit 20.

The processor 4 is, for example, a CPU, and executes an operation control program stored in the memory 3. The processor 4 includes an object detection unit 11, a driving environment acquisition unit 12, a vehicle control unit 16, an optimum appearance likelihood map determination unit 20, and an operation mode switching unit 21.

The optimum appearance likelihood map determination unit 20 determines one appearance likelihood map image from the plurality of appearance likelihood map images according to the position where the object included in the captured image is detected, and determines the appearance likelihood map image of 1. The identification number corresponding to the appearance likelihood map image is stored in the identification number storage unit 19.

When the identification number stored in the identification number storage unit 19 is switched a predetermined number of times within a predetermined time, the operation mode switching unit 21 switches from the automatic operation mode to the manual operation mode.

Subsequently, the operation of the operation control device according to the third embodiment will be described.

FIG. 10 is a flowchart for explaining the operation of the operation control device according to the third embodiment.

Since the processing of steps S21 to S22 is the same as the processing of steps S1 to S2 shown in FIG. 2, the description thereof will be omitted.

Next, in step S23, the optimum appearance likelihood map determination unit 20 selects one appearance likelihood map image from the plurality of appearance likelihood map images according to the position where the object included in the captured image is detected. decide. The optimum appearance likelihood map determination unit 20 determines the optimum appearance likelihood map image having an appearance region corresponding to the position where the object is detected from the plurality of appearance likelihood map images.

Next, in step S24, the optimum appearance likelihood map determination unit 20 stores the identification number of the determined optimum appearance likelihood map image in the identification number storage unit 19.

Next, in step S25, the operation mode switching unit 21 determines whether or not the time from the first storage of the identification number of the optimum appearance likelihood map image in the identification number storage unit 19 is timed. Here, when it is determined that the time is not measured (NO in step S25), in step S26, the operation mode switching unit 21 first stores the identification number of the optimum appearance likelihood map image in the identification number storage unit 19. Start timing the time since then. After starting the timekeeping, the process returns to the process of step S21.

On the other hand, when it is determined that the time is being measured (YES in step S25), in step S27, the operation mode switching unit 21 determines whether or not a predetermined time has elapsed. Here, if it is determined that the predetermined time has not elapsed (NO in step S27), the process returns to the process of step S21.

On the other hand, when it is determined that the predetermined time has elapsed (YES in step S27), in step S28, the operation mode switching unit 21 changes the identification number stored in the identification number storage unit 19 a predetermined number of times during the predetermined time. Determine if you did. The identification number storage unit 19 accumulates and stores the identification numbers of the optimum appearance likelihood map images determined by the optimum appearance likelihood map determination unit 20 within a predetermined time. Therefore, it is possible to calculate how many times the identification number has changed by counting the number of identification numbers stored in the identification number storage unit 19 when the predetermined time has elapsed.

Here, when it is determined that the identification number has changed a predetermined number of times (YES in step S28), in step S29, the operation mode switching unit 21 switches from the automatic operation mode to the manual operation mode.

On the other hand, when it is determined that the identification number has not changed a predetermined number of times (NO in step S28), in step S30, the operation mode switching unit 21 maintains the current automatic operation mode.

FIG. 11 is a schematic diagram for explaining the process of switching from the automatic operation mode to the manual operation mode in the third embodiment. In FIG. 11, the horizontal axis represents time t.

As shown in FIG. 11, at time t0, a person included in the captured image is detected, an appearance likelihood map image is determined according to the position where the object included in the captured image is detected, and the determined optimum appearance likelihood is determined. The identification number (ID: 3) of the degree map image is stored in the identification number storage unit 19. Then, the time counting is started from the time t0.

Next, it is determined whether or not the predetermined time T has elapsed. The predetermined time T is, for example, 1 second. Here, if it is determined that the predetermined time T has not elapsed, the person included in the captured image is detected again. On the other hand, when it is determined that the predetermined time T has elapsed, it is determined whether or not the identification number has changed a predetermined number of times. The predetermined number of times is, for example, two times. That is, when the appearance likelihood map image changes twice per second, the automatic operation mode is switched to the manual operation mode.

In FIG. 11, after the identification number (ID: 3) is first stored, the identification number (ID: 102) is stored at time t1, and the identification number (ID: 401) is stored at time t2. Therefore, the appearance likelihood map image is changed twice by the time T elapses. It is determined that the identification number has changed a predetermined number of times before the predetermined time T elapses, and the automatic operation mode is switched to the manual operation mode.

If the appearance likelihood map image changes in a short time, it can be judged that the object cannot be detected accurately. In this case, by switching from the automatic driving mode to the manual driving mode, the driver can drive the vehicle instead of the automatic driving mode in which the detection accuracy of the object is low.

In the present disclosure, all or part of a unit, device, member or part, or all or part of a functional block in the block diagram shown in the figure, is a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (Large Scale Integration). ) May be executed by one or more electronic circuits including. The LSI or IC may be integrated on one chip, or may be configured by combining a plurality of chips. For example, functional blocks other than the storage element may be integrated on one chip. Here, they are called LSIs and ICs, but the names change depending on the degree of integration, and they may be called system LSIs, VLSIs (Very Large Scale Integration), or ULSIs (Ultra Large Scale Integration). A Field Programmable Gate Array (FPGA), which is programmed after the LSI is manufactured, or a Reconfigurable Logical Device that can reconfigure the junction relationship inside the LSI or set up the circuit partition inside the LSI can also be used for the same purpose.

Further, all or part of the function or operation of a unit, device, member or part can be performed by software processing. In this case, the software is recorded on a non-temporary recording medium such as one or more ROMs, optical discs, hard disk drives, and is identified by the software when it is executed by a processor. Functions are performed by the processor and peripherals. The system or device may include one or more non-temporary recording media on which the software is recorded, a processor, and the required hardware device, such as an interface.

The driving control device, the driving control method, and the driving control program according to the present disclosure can switch from the automatic driving mode in which the vehicle is automatically driven to the manual driving mode in which the driver manually drives the vehicle according to the detection accuracy of the object. It can be used as a driving control device, a driving control method, and a driving control program that control the driving of a vehicle by either an automatic driving mode or a manual driving mode.

1 Operation control device 2 Camera 3 Memory 4 Processor 11 Object detection unit 12 Driving environment acquisition unit 13, 17 Appearance likelihood map storage unit 14 Appearance likelihood map generation unit 15, 21 Operation mode switching unit 16 Vehicle control unit 18 Coordinate system conversion Unit 19 Identification number storage unit 20 Optimal appearance likelihood map determination unit 22 Appearance likelihood map determination unit

Claims (10)

An operation control device that controls the operation of the vehicle by either an automatic driving mode in which the vehicle is automatically driven or a manual driving mode in which the vehicle is manually driven by the driver.
A map storage unit that stores an object appearance map that defines the appearance area where an object may appear,
An object detection unit that detects the object included in the captured image of the traveling direction of the vehicle, and
When the position of the object detected by the object detection unit is outside the appearance area included in the object appearance map , a switching unit for switching from the automatic operation mode to the manual operation mode, and
A driving control unit that controls the driving of the vehicle by the switched manual driving mode,
An operation control device equipped with. The object appearance map is managed in association with map information.
The operation control device according to claim 1. A driving environment acquisition unit that acquires the driving environment around the vehicle,
A generation unit that generates an object appearance map after viewpoint conversion when the object appearance map is viewed from the same viewpoint as the captured image according to the driving environment acquired by the driving environment acquisition unit.
With more
The switching unit superimposes the object appearance map after the viewpoint conversion on the captured image, and the position where the object included in the captured image is detected is included in the object appearance map after the viewpoint conversion. When it is out of the area, the automatic operation mode is switched to the manual operation mode.
The operation control device according to claim 1. The map includes a three-dimensional map represented by a three-dimensional coordinate system.
The appearance area includes an appearance space in which the object may appear on the three-dimensional map.
The generation unit converts the object appearance map in the three-dimensional coordinate system into an object appearance map after the viewpoint conversion in the two-dimensional coordinate system viewed from the same viewpoint as the captured image.
The operation control device according to claim 3. The map storage unit stores a plurality of object appearance maps represented by a two-dimensional coordinate system having the same viewpoint as the captured image, in which the appearance areas are defined according to the shapes of the plurality of roads. ,
A driving environment acquisition unit that acquires the shape of the road included in a photographed image of the traveling direction of the vehicle, and
A determination unit that determines one object appearance map according to the acquired shape of the road from the plurality of object appearance maps.
Further prepare
The operation control device according to claim 1. The map includes a three-dimensional map represented by a three-dimensional coordinate system.
The appearance area includes an appearance space in which the object may appear on the three-dimensional map.
The map storage unit stores a plurality of object appearance maps represented by a three-dimensional coordinate system in which the appearance space is defined according to the shape of the plurality of roads.
A driving environment acquisition unit that acquires the shape of the road included in a photographed image of the traveling direction of the vehicle, and
A determination unit that determines one object appearance map according to the acquired shape of the road from the plurality of object appearance maps.
When the object appearance map of 1 is determined by the determination unit, a coordinate system conversion unit that converts the object appearance map of 1 in the 3D coordinate system into an object appearance map of the same 2D coordinate system as the captured image. ,
Further prepare
The operation control device according to claim 1. The map storage unit stores a plurality of object appearance maps represented by a two-dimensional coordinate system having the same viewpoint as the captured image, in which the appearance areas are defined according to the shapes of the plurality of roads. ,
Identification number storage unit and
One object appearance map is determined from the plurality of object appearance maps according to the position where the object included in the captured image is detected, and the identification number corresponding to the determined object appearance map is assigned. Further equipped with a determination unit to be stored in the identification number storage unit,
When the identification number stored in the identification number storage unit is switched a predetermined number of times within a predetermined time, the switching unit switches from the automatic operation mode to the manual operation mode.
The operation control device according to claim 1. The object is a person
The appearance area is an area through which the person passes.
The operation control device according to any one of claims 1 to 7. A driving control method in a driving control device that controls the driving of the vehicle by either an automatic driving mode in which the vehicle is automatically driven or a manual driving mode in which the vehicle is manually driven by the driver.
The object included in the captured image obtained by photographing the traveling direction of the vehicle is detected.
When the position of the detected object is outside the appearance area where the object defined by the object appearance map may appear, the automatic operation mode is switched to the manual operation mode.
The driving of the vehicle is controlled by the switched manual driving mode.
Operation control method. A driving control program that controls the driving of the vehicle by either an automatic driving mode in which the vehicle is automatically driven or a manual driving mode in which the vehicle is manually driven by the driver.
A map storage unit that stores an object appearance map that defines the appearance area where an object may appear,
An object detection unit that detects the object included in the captured image of the traveling direction of the vehicle, and
When the position of the object detected by the object detection unit is outside the appearance area included in the object appearance map , a switching unit for switching from the automatic operation mode to the manual operation mode, and
The computer functions as a driving control unit that controls the driving of the vehicle by the switched manual driving mode.
Operation control program.

Images