JP7350500B2 - Display method of images for remote control and remote control device - Google Patents

Description

The present invention relates to a method for displaying images used for remote control of a vehicle, and a remote control device that displays images for remote control.

From the vehicle to be remotely controlled, images of the front, left and right sides of the vehicle are acquired via wireless communication, and these images are displayed on a display device for the operator to see. A remote control device that allows a vehicle to be remotely controlled while viewing an image is known (Patent Document 1).

Japanese Patent Application Publication No. 2018-180771

In conventional remote control devices, one operator performs remote control while viewing a plurality of types of images, which may result in a heavy load required for image confirmation.

The problem to be solved by the present invention is to provide a remote control image display method and a remote control device that can reduce the burden of image confirmation by an operator.

The present invention acquires an image of the surroundings of the vehicle from a vehicle equipped with an autonomous driving control function, displays the acquired surrounding image as an image for remote control of the vehicle, and remotely controls the vehicle according to the state of the vehicle. When highlighting a predetermined area of the operation image, including multiple types of direction images taken from the vehicle in multiple directions around the vehicle, highlighting the entire direction image with high importance depending on the state of the vehicle, When the vehicle is traveling within the lane, the front image, left side image, and right side image of the vehicle are highlighted as direction images, and when the vehicle moves from the lane in which it is traveling to the right lane, the vehicle's direction image is highlighted. The front image, right side image, and right rear image are highlighted, and when the vehicle moves from the lane in which it is traveling to the left lane, the above image is highlighted by highlighting the front image, left side image, and left rear image of the vehicle. Solve problems.

According to the present invention, the operator of the remote control device only needs to focus on a highlighted predetermined area of the remote control image when remotely controlling the vehicle, thereby reducing the burden of image confirmation by the operator. I can do it.

1 is a block diagram showing an example of the configuration of an unmanned driving system to which a remote control image display method and remote control device according to the present invention are applied. FIG. 2 is a block diagram showing the functional configuration of a vehicle and a remote control device. FIG. 2 is a top view of the vehicle showing the positions of a camera, a radar device, and a lidar device provided in the vehicle. 3 is a diagram showing an operation screen displayed on a display unit by the display control unit of FIG. 2. FIG. This is an example of an image displayed on the bird's-eye view screen PV in FIG. 4, and is a satellite image of a one-lane road (two lanes up and down) with left-hand traffic. 6 is a diagram showing a left screen LV, a front screen FV, a right screen RV, a left rear screen LBV, a rear screen BV, and a right rear screen RBV of the vehicle V0 shown in FIG. 5, respectively. 7 is an overhead view screen PV showing the avoidance route of the vehicle V0 shown in FIGS. 5 and 6. FIG. FIG. 6 is a diagram showing an operation screen displayed on the display section of the remote control device when the vehicle is traveling normally. A situation in which a vehicle V0 is traveling on a road with one lane in each direction and stops behind the stopped vehicle V1 because a stopped vehicle V1 is parked in front of it, and then resumes driving by bypassing the stopped vehicle V1 by remote control. FIG. 10 is a diagram showing an example of a left side image, a front image, a right side image, a left rear image, a rear image, and a right rear image displayed on the display unit of the remote control device by the image enhancement unit in the situation shown in FIG. 9. FIG. 10 is a diagram showing an example of a left side image, a front image, a right side image, a left rear image, a rear image, and a right rear image displayed on the display unit of the remote control device by the object emphasis unit in the situation shown in FIG. 9. FIG. 10 is a diagram showing an example of a left side image, a front image, a right side image, a left rear image, a rear image, and a right rear image displayed on the display unit of the remote control device by the analysis information emphasis unit in the situation shown in FIG. 9. FIG. It is a flowchart which shows the control procedure of an unmanned driving system. 14 is a flowchart showing the subroutine processing of step S7 in FIG. 13.

Embodiments of the present invention will be described below based on the drawings. FIG. 1 is a block diagram showing an example of the configuration of an unmanned driving system 1 according to the present embodiment. The unmanned driving system 1 according to the present embodiment is also an embodiment for implementing a remote control image display method and a remote control device according to the present invention.

The unmanned driving system 1 according to the present embodiment has an autonomous driving control function and operates a vehicle V0, which is equipped with an autonomous driving control function and runs under autonomous driving control or under autonomous driving control based on remote control from the outside, as a taxi, a passenger car, or other commercial vehicle. It is a system to be operated. The unmanned driving system 1 according to the present embodiment includes a plurality of vehicles V0, a plurality of remote control devices 3, and a network NW1 that connects the vehicle V0 and the remote control devices 3 so as to be able to communicate with each other.

The remote control device 3 remotely controls the vehicle V0 when the vehicle V0 is no longer able to travel under autonomous driving control. The plurality of remote control devices 3 are installed in a vehicle management center 4 equipped with communication equipment. In the vehicle management center 4, a management server 5 connected to a local network NW2 such as a LAN together with a remote control device 3 is installed. The management server 5 connects to the plurality of vehicles V0 through the network NW1, connects to the plurality of remote control devices 3 through the local network NW2, and manages the plurality of vehicles V0 and the remote control devices 3.

For example, the Internet is used as the network NW1. Further, to connect each vehicle V0 to the network NW1, a mobile phone communication network or the like that is widespread in a wide area and can provide a relatively high communication speed is used. In addition, a plurality of access points for wireless communication are installed in a predetermined area where the unmanned driving system 1 is operated, and a wireless communication network dedicated to the unmanned driving system is formed. 3 may be connected.

In the unmanned driving system 1 according to the present embodiment, for example, a plurality of vehicles V0 are arranged in advance at predetermined boarding start positions. For example, the passenger P operates a dedicated application installed on a mobile terminal T such as a smartphone, enters information such as customer information, ride start time, destination, etc., and makes a reservation for the ride. The information input into the dedicated application of the mobile terminal T is transmitted to the management server 5 as reservation information.

Based on the received reservation information, the management server 5 sets a vehicle V0 (hereinafter also referred to as a reserved vehicle) that carries the passenger P, a boarding start time, and a route to the destination. In addition, the management server 5 transmits the vehicle number of the set reserved vehicle V0, boarding start time, destination, and travel route as reservation confirmation information to the passenger P's mobile terminal T. Furthermore, the management server 5 transmits the customer information of the passenger P, the boarding start time, the destination, and the route of travel as operation information to the reserved vehicle V0. The reserved vehicle V0 sets a travel route for driving on the road within the driving route without straying from the lane, and uses autonomous driving control to autonomously travel along the travel route while observing traffic regulations. to the destination.

In addition, while the vehicle V0 is autonomously traveling along the travel route set along the driving route, the autonomous driving control is prevented from moving out of the lane due to obstacles on the road such as parked vehicles or construction sites in the direction of travel. If the vehicle is unable to do so, the vehicle temporarily stops and sends a support request signal to the management server 5. The management server 5, which has received the support request signal from the vehicle V0, determines the remote control device 3 that remotely controls the vehicle V0 from among the plurality of remote control devices 3, requests support from the determined remote control device 3, and The remote control device 3 that has received the support request and the vehicle V0 are communicably connected. The remote control device 3 sets an avoidance route, which is a travel route for avoiding obstacles on the road, and transmits it to the vehicle V0 so that the vehicle V0 can perform autonomous driving control again. The vehicle V0 travels on an avoidance route under autonomous driving control to avoid obstacles on the road, and then travels again on a travel route set along the driving route under autonomous driving control.

FIG. 2 is a block diagram showing the functional configuration of the vehicle V0 and the remote control device 3. As shown in FIG. Note that in FIG. 2, illustration of the network NW1 is omitted to avoid complication of the drawing. The vehicle V0 according to the present embodiment includes a communication section 21, a sensor 22, an obstacle recognition section 23, a route setting section 24, a travel control section 25, and a vehicle information acquisition section 26.

Among the parts that make up the vehicle V0, the obstacle recognition part 23, the route setting part 24, the travel control part 25, and the vehicle information acquisition part 26 are operated by one or more computers and software installed on the computers. It is configured. A computer is comprised of a ROM that stores programs for making the above-mentioned parts function, a CPU that executes the programs stored in the ROM, and a RAM that functions as an accessible storage device. Note that an MPU, DSP, ASIC, FPGA, etc. can be used in place of or in addition to the CPU. Further, the computers constituting each section are connected to the communication section 21 and the sensor 22 by, for example, a CAN (Controller Area Network) or other in-vehicle LAN in order to mutually transmit and receive information.

The communication unit 21 communicates with the vehicle management center 4 via the network NW1. Specifically, the communication unit 21 receives the customer information of the passenger P, the boarding start time, the destination, and the travel route transmitted from the management server 5 as operation information. The communication unit 21 also transmits the current position information (for example, latitude, longitude, etc.) of the vehicle V0 and the direction in which the front of the vehicle V0 is facing (hereinafter also referred to as vehicle direction information) at predetermined time intervals or predetermined travel distances. ) and speed information to the management server 5. The management server 5 manages the running status of each vehicle V0 based on the received current position information, vehicle direction information, and speed information. Furthermore, the communication unit 21 transmits an assistance request signal to the management server 5 when the vehicle V0 is no longer able to travel under autonomous driving control. Further, when the remote control device 3 that remotely controls the vehicle V0 is determined, the communication unit 21 transmits various vehicle information necessary for remotely controlling the vehicle V0 to the remote control device 3.

Sensor 22 detects the surrounding situation of vehicle V0. The sensor 22 includes, for example, a camera and an obstacle sensor. The camera is an imaging device that captures an image of the surroundings of the vehicle V0 using an image sensor such as a CCD or CMOS, and generates image data of the surroundings image. The camera constituting the sensor 22 is composed of a plurality of cameras, and for example, as shown in FIG. 3 which is a top view of the vehicle V0, a front camera 221F that images the front of the vehicle V0, and a rear camera 221F that images the rear of the vehicle V0. It includes a camera 221B, a left camera 221L and a right camera 221R that take images of the left and right sides of the vehicle V0, respectively, and a left rear camera 221LB and a right rear camera 221RB that take images of the left and right rear of the vehicle V0, respectively.

The obstacle sensor forming the sensor 22 detects obstacles around the vehicle V0. As the obstacle sensor, various sensors such as radar devices and lidar devices are used alone or in appropriate combinations. As shown in FIG. 3, the radar device 222a is installed, for example, at the front of the vehicle V0, and radiates millimeter waves or ultrasonic waves to the front of the vehicle V0 and receives reflected waves. Detects the distance and relative speed to surrounding obstacles (objects). The lidar devices 222b and 222c are installed, for example, at the front and rear of the vehicle V0, emit laser light around the vehicle V0, receive reflected light, and three-dimensionally map the received reflected light. As a result, the distance to objects existing around the vehicle V0, the shape of the objects, etc. are detected.

The obstacle recognition unit 23 analyzes the image data captured by the camera of the sensor 22 and the obstacle sensor information that is the detection result of the obstacle sensor, and identifies objects existing around the vehicle V0 that may affect autonomous driving. Recognize the objects that affect you. Objects recognized by the obstacle recognition unit 23 include, for example, moving objects such as pedestrians, bicycles, motorcycles, and automobiles, stationary objects such as stopped vehicles, construction sites, fallen objects on the road, and structures on the road. Roadside structures include guardrails, road shoulders, median strips, traffic lights, etc. Note that the image data from the front camera 221F is also used to detect lane marks such as center lines and outside lines of the roadway, and road markings such as stop lines and crosswalks.

The route setting unit 24 sets a travel route along which the vehicle V0 actually travels along the travel route received from the vehicle management center 4. Although not shown in detail, the route setting section 24 includes a position detection section and a map database.

The position detection section includes a GPS unit, a gyro sensor, a vehicle speed sensor, and the like. The position detection unit detects radio waves transmitted from multiple GPS satellites by the GPS unit, periodically acquires position information of the vehicle V0, and detects the acquired position information of the vehicle V0 and the angle change acquired from the gyro sensor. The current position information of the vehicle V0 is generated based on the information and the speed information acquired from the vehicle speed sensor.

The map database is similar to the map information used in car navigation devices and the like, and records information on roads as well as location information on various facilities and specific points. Road-related information includes road type, road width, number of lanes, location of intersections and traffic lights, existence and number of right-turn/left-turn lanes, speed limits, merging points, branching points, toll booths, and number of lanes. Location information such as reduced location, service area (SA)/parking area (PA), etc. are recorded.

The route setting unit 24 determines the road of the driving route on which the vehicle V0 will travel, based on the current position information of the vehicle V0 generated by the position detection unit, the map database, and the recognition results of obstacles by the obstacle recognition unit 23. By specifying the width of the road and setting the travel route of the vehicle V0 in the center of the width of the road, a travel route that allows the vehicle V0 to travel without protruding from the lane of the route is set. The route setting unit 24 corresponds to the route setting unit of the present invention.

The driving control unit 25 is a functional unit that controls the autonomous driving control function of the vehicle V0, and controls each part of the vehicle V0 to cause the vehicle V0 to travel along the driving route set by the route setting unit 24. More specifically, the traveling control unit 25 controls, for example, a drive mechanism such as an engine or a motor, a transmission mechanism, a braking mechanism, etc. to adjust the speed of the vehicle V0, and also adjusts the speed of the vehicle V0 along the longitudinal direction of the vehicle V0. The steering mechanism is controlled so that the vehicle center line VL (see FIG. 3) moved along the travel route. Further, when the vehicle V0 turns right or left along the travel route, the travel control unit 25 controls the drive mechanism, transmission mechanism, and braking mechanism to adjust the speed, and also controls the steering mechanism, direction indicator, etc. to make the vehicle V0 turn right or left.

Further, the travel control unit 25 determines whether there is a temporary stop position, an intersection with a red light, etc. in the traveling direction of the vehicle V0 based on the current position information, the map database, and the recognition result of the obstacle recognition unit 23. If this is determined, the drive mechanism, transmission mechanism, braking mechanism, etc. are controlled to reduce the speed of the vehicle V0 and bring it to a stop. In addition, when the obstacle recognition unit 23 determines that there is an obstacle such as a stopped vehicle, a construction site, or a pedestrian in the direction of movement of the vehicle V0, the driving control unit 25 detects the presence of an obstacle such as a stopped vehicle, a construction site, a pedestrian, etc. that interferes with the autonomous driving control. The control unit controls the drive mechanism, transmission mechanism, braking mechanism, etc. to reduce the speed of the vehicle V0 and, if necessary, stops the vehicle V0. In addition, in this embodiment, the autonomous driving control function provided in the driving control unit 25 is not particularly limited, and the technology at the time of filing of the present invention can be used as appropriate.

Furthermore, when the vehicle V0 is stopped because it is no longer able to travel under autonomous travel control, the travel control unit 25 transmits an assistance request signal to the management server 5 of the vehicle management center 4 through the communication unit 21. Note that the support request signal may be a signal that simply requests support, or may be a signal that includes the content of the support request. If the support request signal includes the content of the support request and is transmitted, the management server 5 can use the content of the support request to determine the remote control device 3 that remotely controls the vehicle V0. Further, when the support request signal is transmitted from a plurality of vehicles V0, it can be used to determine the priority of the vehicle V0 to start support, etc., depending on the content of the support request in the support request signal.

The vehicle information acquisition unit 26 acquires various vehicle information to be transmitted to the remote control device 3 that remotely controls the vehicle V0. The vehicle information acquired by the vehicle information acquisition unit 26 includes, for example, surrounding images captured by the front camera 221F, rear camera 221B, left camera 221L, right camera 221R, left rear camera 221LB, and right rear camera 221RB of the sensor 22. image data, obstacle sensor information regarding obstacles detected by the radar device 222a and lidar devices 222b, 222c, current position information of the vehicle V0, vehicle direction information, etc. are included.

Next, the remote control device 3 will be explained. As shown in FIG. 2, the remote control device 3 includes a communication section 31, an input section 32, a display section 33, a display control section 34, and a remote control section 35. The display control unit 34 and remote control unit 35 of the remote control device 3 are configured by one or more computers and software installed on the computers. A computer is comprised of a ROM that stores programs for making the above-mentioned parts function, a CPU that executes the programs stored in the ROM, and a RAM that functions as an accessible storage device. Note that an MPU, DSP, ASIC, FPGA, etc. can be used in place of or in addition to the CPU. Further, the computers constituting each section are connected to a communication section 31, an input section 32, and a display section 33 so as to mutually transmit and receive information.

Communication unit 31 communicates with vehicle V0 via network NW1. Specifically, when performing remote control of the vehicle V0 requested by the management server 5, the communication unit 31 sends the front camera 221F, rear camera 221B, Image data of surrounding images captured by the left camera 221L, right camera 221R, left rear camera 221LB, and right rear camera 221RB, and obstacle sensor information regarding obstacles detected by the radar device 222a and lidar devices 222b and 222c, Current position information and vehicle direction information of vehicle V0 are received. Further, the communication unit 31 transmits remote control information such as an avoidance route to the vehicle V0, for example, in order to enable the vehicle V0 to avoid obstacles on the road and continue autonomous driving control. That is, the communication section 31 corresponds to the video acquisition means of the present invention.

The input unit 32 is a device operated by the operator of the remote control device 3, and is used to perform various operations such as generating an avoidance route for the vehicle V0. As the input unit 32, for example, a touch panel arranged on a display, a keyboard, a device for detecting an operator's manual operation such as a mouse or a joystick, a voice input device equipped with a microphone and a voice recognition unit, etc. are used.

The display unit 33 is a display device such as a liquid crystal display, and displays an operation screen for remotely controlling the vehicle V0. The display control unit 34 causes the display unit 33 to display an operation screen. The display section 33 and the display control section 34 correspond to display means and display control means of the present invention.

FIG. 4 shows an operation screen displayed on the display unit 33 by the display control unit 34. This operation screen includes an overhead screen PV, a left side screen LV, a front screen FV, a right side screen RV, a left rear screen LBV, a rear screen BV, and a right rear screen RBV. The images displayed on the bird's-eye view screen PV, left side screen LV, front screen FV, right side screen RV, left rear screen LBV, rear screen BV, and right rear screen RBV are images from the vehicle V0 to the surrounding area. This corresponds to multiple types of directional images captured in multiple directions.

The bird's-eye view screen PV includes, for example, a map image, a satellite image, or an aerial image acquired based on the current position information of the vehicle V0, a top view image of the vehicle V0, and obstacles detected by the radar device 222a and lidar devices 222b and 222c. The sensor information is displayed in an overlapping manner. The left screen LV, front screen FV, right screen RV, left rear screen LBV, rear screen BV, and right rear screen RBV include the left camera 221L, front camera 221F, right camera 221R, left rear camera 221LB, which are received from the vehicle V0. Image data of the rear camera 221B and the right rear camera 221RB are respectively displayed.

For example, the bird's-eye view screen PV shown in FIG. 5 is a satellite image of a one-lane road (two lanes up and down) with left-hand traffic. Vehicle shelter areas L1s and L2s are provided in the up lane L1 of this road, which goes from the bottom to the top in the figure, and the down lane L2, which goes from the top to the bottom in the figure, respectively. Further, a zebra zone Z is provided in a part between the up lane L1 and the down lane L2. Furthermore, road shoulders U1 and U2 and sidewalks J1 and J2 are arranged on the outside of the up lane L1 and the down lane L2, respectively. Further, in the up lane L1, a top view of a vehicle V0 autonomously traveling along a travel route under autonomous travel control is displayed. In addition, in front of the vehicle V0, there is obstacle sensor information FI that is the detection result of the lidar devices 222b and 222c of the vehicle V0 detecting a stopped vehicle V1 (see front screen FV in FIG. 6) existing in the up lane L1. Displayed. Only the outline of the rear part of the stopped vehicle V1 is displayed in the obstacle sensor information FI.

FIG. 6 shows the left screen LV, front screen FV, right screen RV, left rear screen LBV, rear screen BV, and right rear screen RBV of the vehicle V0 shown in FIG. The rear part of the stopped vehicle V1 imaged by the front camera 221F is displayed on the front screen FV. Note that in the left screen LV, front screen FV, right screen RV, left rear screen LBV, rear screen BV, and right rear screen RBV, the portion hatched with dots indicates the body of the vehicle V0.

The remote control unit 35 is configured to control the vehicle V0 on the road when the vehicle V0, which is traveling under autonomous driving control on a travel route set along the driving route, is unable to autonomously travel due to an obstacle on the road such as a stopped vehicle or a construction site. Set an avoidance route to avoid obstacles using autonomous driving control. That is, the remote control unit 35 corresponds to the route setting unit of the present invention, together with the route setting unit 24 of the vehicle V0. To set the avoidance route, operate the input unit 32 to set the avoidance route from the front of the vehicle V0 to avoid the obstacle and return to the original travel route on the overhead screen PV displayed on the display unit 33. This is done by inputting.

FIG. 7 is an overhead screen PV showing the avoidance route of the vehicle V0 shown in FIGS. 5 and 6. The operator of the remote control device 3 sets a plurality of route points Pt, which are planned positions for the vehicle V0 to pass, from the front of the vehicle V0 so as to avoid the stopped vehicle V1. The remote control unit 35 connects the plurality of set route points Pt to generate an avoidance route Ra. On the bird's-eye view screen PV, the plurality of set route points Pt, the generated avoidance route Ra, and the movement trajectory Mt of the vehicle V0 traveling along the avoidance route Ra are displayed on the satellite image. The set avoidance route Ra is transmitted to the vehicle V0 by the communication unit 31.

The display control unit 34 includes an image display unit 341, an image enhancement unit 342, an object detection unit 343, an object enhancement unit 344, an object prediction unit 345, and an analysis information enhancement unit 346. The image display unit 341 displays the image data of the left camera 221L, front camera 221F, right camera 221R, left rear camera 221LB, rear camera 221B, and right rear camera 221RB received from the vehicle V0 on the left screen LV of the display unit 33, It is displayed on the front screen FV, right side screen RV, left rear screen LBV, rear screen BV, and right rear screen RBV, respectively. The image display unit 341 also displays images based on the image data of the left camera 221L, front camera 221F, right camera 221R, left rear camera 221LB, rear camera 221B, and right rear camera 221RB, which are regularly received from the vehicle V0. The left screen LV, front screen FV, right screen RV, left rear screen LBV, rear screen BV, and right rear screen RBV of the section 33 are updated.

The image enhancement unit 342 highlights a predetermined area of the remote control image according to the state of the vehicle V0. Specifically, a predetermined area of the remote control image is highlighted depending on the traveling direction of the vehicle V0. The traveling direction is defined as the direction in which the vehicle V0 is traveling or the direction of the traveling route along which the vehicle V0 is driven under autonomous driving control, and a predetermined area of the remote control image is highlighted.

For example, when the vehicle V0 is traveling on a travel route set by autonomous driving control, that is, when it is traveling normally, the left screen LV of the display unit 33, the front The outer frames of the screen FV and the right screen RV are highlighted with, for example, thick red lines.

Further, as shown in FIG. 9(A), when the vehicle V0 is traveling on a road with one lane on each side, a stopped vehicle V1 is stopped in front, and the vehicle V0 is stopped behind the stopped vehicle V1. In this case, the vehicle V0 needs to detour around the stopped vehicle V1 and resume traveling by remote control, as shown in FIG. 2(B). In such a case, the direction of the avoidance route generated by the remote control device 3 is specified as the traveling direction, and as shown in FIG. The outer frame of the screen RBV is highlighted with, for example, a thick red line.

The object detection unit 343 analyzes the image data of the left camera 221L, front camera 221F, right camera 221R, left rear camera 221LB, rear camera 221B, and right rear camera 221RB received from the vehicle V0, and uses the analysis results and the vehicle Based on the radar device 222a and the obstacle sensor information of the lidar devices 222b and 222c received from V0, detect objects that affect the running of the vehicle V0, objects that exist on a route that intersects the running route of the vehicle V0, etc. do. Objects recognized by the object detection unit 343 include, for example, moving objects such as pedestrians, bicycles, motorcycles, and automobiles, stationary objects such as stopped vehicles, construction sites, fallen objects on the road, and structures on the road. Roadside structures include guardrails, road shoulders, median strips, traffic lights, etc.

The object prediction unit 345 identifies the type of object detected by the object detection unit 343, and determines the possibility of approaching the vehicle V0 according to the type of the object. For example, pedestrians, bicycles, cars, etc. are detected as moving objects, and the approachability is determined based on the position, moving speed, moving direction, and position of the own vehicle V0 of each of these objects. .

The object enhancement unit 344 emphasizes a predetermined area of the remote control image when the object detection unit 343 detects an object that affects the running of the vehicle V0 or an object that has a high possibility of approaching the vehicle V0. indicate. Specifically, an object that affects the running of the vehicle V0 and a predetermined area of the remote control image that includes this object are highlighted.

For example, in the example shown in FIG. 11, the object detection unit 343 detects the stopped vehicle V1 displayed on the front screen FV, so the object emphasis unit 344 blinks the outer frame of the front screen FV. Highlight. Furthermore, based on the detection result of the object detection unit 343, the object enhancement unit 344 displays an image showing the stopped vehicle V1 at the position where the stopped vehicle V1 is stopped on the bird's-eye view screen PV in a color tone such as red that tends to attract attention. Highlight it by displaying it with or blinking it. In this way, the detected object is displayed not only on the screen where the object is actually displayed, but also on the bird's-eye view screen PV, making it easier to understand the positional relationship with the own vehicle V0. The bird's-eye view screen PV is a screen that is always checked by the operator during remote control of the vehicle V0, so the outer frame etc. is not highlighted, but it is highlighted when displaying detected objects. You can also do this.

When an object is detected by the object detection unit 343, the analysis information emphasis unit 346 specifies object information including at least one of the type of the object, the distance to the object, the speed of the object, or the relative speed between the object and the vehicle. Then, the object information is displayed superimposed on the remote control image along with object detection information indicating that an object has been detected. Additionally, information indicating the approach of the object to the vehicle is displayed as object detection information.

For example, in the example shown in FIG. 12, the object detection unit 343 detects a stopped vehicle V1 displayed on the front screen FV and an overtaking vehicle V2 displayed on the right screen RV. The overtaking vehicle V2 is a vehicle that is attempting to overtake the vehicle V0 and the stopped vehicle V1 from the opposite lane. The object emphasizing unit 344 highlights the outer frames of the front screen FV and right screen RV by blinking them. Furthermore, based on the detection result of the object detection unit 343, the object enhancement unit 344 highlights an image showing the stopped vehicle V1 at the position where the stopped vehicle V1 is stopped on the bird's-eye view screen PV, and displays an image showing the stopped vehicle V1 in a highlighted manner at the position where the stopped vehicle V1 is stopped, and the overtaking vehicle V2 is stopped. An image showing the overtaking vehicle V2 is highlighted at the location where the overtaking vehicle V2 is located. The analysis information emphasizing unit 346 displays object detection information indicating “there is a stopped vehicle” on the front screen FV. In addition, the analysis information emphasizing unit 346 displays object detection information such as "vehicle approaching" and object information "approaching at XX km" on the right screen RV. Furthermore, the analysis information emphasizing unit 346 displays object detection information such as "stopped vehicle present" and "approaching vehicle present" in the vicinity of the stopped vehicle V1 and in the vicinity of the overtaking vehicle V2 on the bird's-eye view screen PV, respectively.

Next, the operation of the unmanned driving system 1 of this embodiment will be explained with reference to the flowcharts shown in FIGS. 13 and 14. When the management server 5 receives the reservation information from the passenger P's mobile terminal T, it sets the vehicle V0 carrying the passenger P, the boarding start time, the route to the destination, etc. (step S1), and Reservation confirmation information is transmitted to terminal T, and operation information is transmitted to reserved vehicle V0 (step S2). The management server 5 receives the current position information, vehicle direction information, and speed information transmitted from the vehicle V0 every predetermined time or every predetermined travel distance, and manages and monitors the autonomous running status of the vehicle V0 (step S3 ).

The vehicle V0 receives the operation information transmitted from the management server 5 through the communication unit 21, and the route setting unit 24 causes the vehicle V0 to travel without straying from the lane on the travel route based on the travel route in the operation information. Set a driving route for the vehicle.

When the boarding start time arrives, the vehicle V0 allows the passenger P to board the vehicle and starts autonomous driving control for transporting the passenger P to the destination. In this autonomous driving control, the vehicle V0 controls the driving mechanism, transmission mechanism, braking mechanism, etc. by the driving control unit 25 to adjust the speed, and also controls the steering mechanism so that the vehicle center line VL moves on the driving route. Control. In addition, when the vehicle V0 makes a right or left turn along the travel route, the travel control unit 25 controls the drive mechanism, transmission mechanism, and braking mechanism to adjust the speed, and turns the steering mechanism, direction indicator, etc. Make a controlled right or left turn. Furthermore, if it is determined that there is a temporary stop position, an intersection with a red light, etc. in the direction of travel, based on the current position information, the map database, and the recognition result of the obstacle recognition unit 23, the vehicle V0 The drive control unit 25 controls the drive mechanism, transmission mechanism, braking mechanism, etc. to reduce the speed of the vehicle V0 and bring it to a stop.

Vehicle V0 running under autonomous driving control captures images of the surroundings of vehicle V0 by the front camera 221F, rear camera 221B, left camera 221L, right camera 221R, left rear camera 221LB, and right rear camera 221RB of the sensor 22. Furthermore, the autonomously running vehicle V0 detects objects around the vehicle V0 using the radar device 222a and lidar devices 222b and 222c of the sensor 22. The obstacle recognition unit 23 of the vehicle V0 analyzes the image data captured by each camera of the sensor 22 and the obstacle sensor information that is the detection result of the radar device 222a and lidar devices 222b, 222c, and identifies the object of the vehicle V0. Recognize objects around you that affect autonomous driving.

If there is an obstacle on the road such as a stopped vehicle or a construction site in the direction of travel of the vehicle V0, and autonomous driving control cannot be performed without the vehicle V0 protruding from the lane, the driving control unit 25 performs autonomous driving control. The vehicle V0 is temporarily stopped and a support request signal is transmitted to the management server 5.

The management server 5 that has received the support request signal from the vehicle V0 (YES in step S4) determines the remote control device 3 that remotely controls the vehicle V0 from among the plurality of remote control devices 3, and selects the determined remote control device 3. A support request is made to the vehicle V0, and the remote control device 3 that received the support request is communicably connected to the vehicle V0 (step S5).

The remote control device 3 receives various driving information such as image data acquired by the vehicle information acquisition unit 26 from the vehicle V0 that has received the support request. This various traveling information includes image data of surrounding images captured by the front camera 221F, rear camera 221B, left camera 221L, right camera 221R, left rear camera 221LB, and right rear camera 221RB, and the radar device 222a and lidar device 222b. , 222c, current position information of the vehicle V0, vehicle direction information, etc.

The remote control device 3 starts an avoidance process for causing the vehicle V0 to avoid obstacles and run based on various data such as image data received from the vehicle V0. Specifically, the image display unit 341 of the display control unit 34 displays the vehicle V0 and obstacle sensor information FI on the overhead screen PV displayed on the satellite image and the image captured by the left camera 221L on the display unit 33. The left screen LV, the front screen FV taken by the front camera 221F, the right screen RV taken by the right camera 221R, the left rear screen LBV taken by the left rear camera 221LB, and the left rear screen LBV taken by the rear camera 221B. The rear screen BV and the right rear screen RBV captured by the right rear camera 221RB are displayed (step S6).

Next, the remote control device 3 executes a region-of-interest determination process for determining a region of interest in the remote-control image (step S7). FIG. 14 is a flowchart showing the subroutine of the attention area determination process in step S7 of FIG. As shown in FIG. 14, in the attention area determination process, the image enhancement unit 342 specifies the traveling direction of the vehicle V0 or the direction of the traveling route as the traveling direction (step S71), and emphasizes the traveling direction based on the identified traveling direction. The screen of the direction image to be displayed is determined (step S72). For example, when the vehicle V0 is normally traveling on a travel route set by autonomous driving control, the outer frames of the left screen LV, front screen FV, and right screen RV of the display unit 33 are However, three regions of interest are determined, for example, the left screen LV, the front screen FV, and the right screen RV, as highlighted by a thick red line or the like.

Alternatively, as shown in FIGS. 9(A) and (B), while the vehicle V0 is traveling on a road with one lane on each side, a stopped vehicle V1 is stopped in front, and the vehicle V0 is stopped behind the stopped vehicle V1. In such a case, it is necessary for the vehicle V0 to detour around the stopped vehicle V1 and resume traveling by remote control. In such a case, the direction of the avoidance route generated by the remote control device 3 is specified as the traveling direction, and as shown in FIG. Three regions of interest are determined: the front screen FV, the right screen RV, and the right rear screen RBV so that the outer frame of the screen RBV is highlighted, for example, with a thick red line.

Next, an additional region of interest is determined (step S73). Here, the object detection unit 343 analyzes images in each direction and detects objects that affect the travel of the vehicle V0, objects that exist on a route that intersects the travel route of the vehicle V0, and the like. Further, the object prediction unit 345 identifies the type of object detected by the object detection unit 343, and determines the possibility of approaching the vehicle V0 according to the type of the object. The object highlighting unit 344 determines an object to be highlighted when an object is detected or when an object with predicted approachability is detected. For example, in the example shown in FIG. 11, the object detection unit 343 detects the stopped vehicle V1 displayed on the front screen FV, so the additional attention area is set in front so as to highlight the outer frame of the front screen FV. Set to screen FV. Furthermore, the object enhancement unit 344 adds an additional attention area so as to highlight an image showing the stopped vehicle V1 at the position where the stopped vehicle V1 is stopped on the bird's-eye view screen PV based on the detection result of the object detection unit 343. is determined as the bird's-eye view PV.

In the next step S74, the object detection unit 343 determines whether or not an object has been detected. If an object is detected, the process proceeds to step S75; if no object is detected, the process proceeds to step S8 of FIG. 13. When an object is detected by the object detection unit 343, in step S75, the analysis information emphasis unit 346 detects at least one of the type of the detected object, the distance to the object, the speed of the object, or the relative speed between the object and the vehicle. Identify object information that includes one. Then, in the subsequent step S76, this object information and object detection information indicating that an object has been detected are generated. Note that information indicating the approach status of the object to the vehicle is generated as the object detection information. For example, in the example shown in FIG. 12, the object detection unit 343 detects a stopped vehicle V1 displayed on the front screen FV and an overtaking vehicle V2 displayed on the right screen RV. Therefore, the analysis information emphasizing unit 346 specifies that the vehicle V1 is a stopped vehicle, and specifies that the overtaking vehicle V2 is a vehicle that is about to overtake the vehicle V0 and the stopped vehicle V1 from the opposite lane.

Returning to step S8 in FIG. 13, in step S8, the image enhancement unit 342 executes a process of highlighting the screen determined in step S72. Regarding the example shown in FIG. 8, since the left screen LV, front screen FV, and right screen RV of the display unit 33 are the screens determined in step S72, these left screen LV, front screen FV, and right screen RV Each outer frame is highlighted using, for example, a thick red line.

Also, in step S8, when an object is detected, or when an object with predicted approachability is detected, the object highlighting unit 344 executes a process of highlighting the object. Regarding the example shown in FIG. 11, since the additional attention area determined in step S73 is the front screen FV, the outer frame of the front screen FV is highlighted. Moreover, since the bird's-eye view screen PV is also determined as an additional attention area in step S73, an image showing the stopped vehicle V1 is displayed in an emphasized manner at the position where the stopped vehicle V1 is parked on the bird's-eye view screen PV.

Furthermore, in step S8, the analysis information emphasizing unit 346 highlights the object information generated in step S76 on the corresponding screen. For example, in the example shown in FIG. 12, the object detection unit 343 detects a stopped vehicle V1 displayed on the front screen FV and an overtaking vehicle V2 displayed on the right screen RV. Since the overtaking vehicle V2 is identified as a vehicle that is attempting to overtake the vehicle V0 and the stopped vehicle V1 from the opposite lane, the analysis information emphasizing unit 346 The text data "Stopped Vehicle" is highlighted, and the text data "Approaching Vehicle" is highlighted next to vehicle V2. In addition, the analysis information emphasizing unit 346 highlights the text data "There is a stopped vehicle" next to the vehicle V1 on the front screen FV, and "There is an approaching vehicle, approaching at XX km" next to the vehicle V2 on the right screen RV. Highlight the text data.

In step S9, since the remote control device 3 is requested to perform remote control in step S5, the operator of the remote control device 3 uses the remote control device 3 based on various data such as image data received from the vehicle V0. 3 to set an avoidance route for the vehicle V0 to avoid obstacles on the road through autonomous driving control. Specifically, the display control unit 34 of the remote control device 3 causes the display unit 33 to display an overhead screen PV in which the vehicle V0 and obstacle sensor information FI are displayed on a satellite image, and an image captured by the left camera 221L. The left screen LV, the front screen FV taken by the front camera 221F, the right screen RV taken by the right camera 221R, the left rear screen LBV taken by the left rear camera 221LB, and the left rear screen LBV taken by the rear camera 221B. The rear screen BV and the right rear screen RBV captured by the right rear camera 221RB are displayed.

As shown in FIG. 7, the operator of the remote control device 3 can view the bird's-eye view screen PV, the left screen LV, the front screen FV, the right screen RV, the left rear screen LBV, and the rear screen displayed on the display unit 33, as shown in FIG. The situation of the vehicle V0 is grasped based on the BV and the right rear screen RBV, and the input unit 32 is operated to set a plurality of route points Pt to avoid obstacles on the bird's-eye view screen PV. The remote control device 3 generates an avoidance route Ra by connecting the plurality of set route points Pt. On the bird's-eye view screen PV, the plurality of route points Pt set by the display control unit 34, the generated avoidance route Ra, and the movement trajectory Mt of the vehicle V0 traveling along the avoidance route Ra are displayed, and the operator confirmed by. When setting this avoidance route, the operator operates while paying attention to the highlighted screen, so the operator's operational load can be reduced.

In step S10, the avoidance route Ra set in step S9 is transmitted to the vehicle V0 by the communication unit 31, and the vehicle V0 that receives this starts autonomous travel for avoidance according to the avoidance route. When the vehicle V0 returns to the original driving route after avoiding the obstacle (YES in step S11), the driving control unit 25 of the vehicle V0 resumes autonomous driving control of the original driving route. Then, in step S11, the process of steps S6 to S10 is repeated until the avoidance drive is completed, and when the avoidance drive is completed, the process advances to step S12.

In step S12, it is determined whether the vehicle V0 running autonomously has arrived at the destination, and the processes of steps S3 to S12 are repeated until the vehicle V0 reaches the destination, and once the vehicle V0 has arrived at the destination, the process proceeds to step S13. Then, the monitoring of the autonomous running of the vehicle V0 is ended.

As explained above, the unmanned driving system 1 of the present embodiment, when the vehicle V0 equipped with an autonomous driving control function is driven by autonomous driving control based on remote control from the outside of the vehicle, from the vehicle V0, A surrounding image of the surroundings of the vehicle is acquired, the acquired surrounding image is displayed as a remote control image of the vehicle V0, and a predetermined area of the remote control image is highlighted depending on the state of the vehicle V0. Thereby, the operator of the remote control device 3 can easily grasp the area to be focused on, and the burden on the operator required for image confirmation can be reduced.

In addition, the unmanned driving system 1 of this embodiment uses an image for remote control according to the traveling direction of the vehicle V0 (including the direction in which the vehicle is traveling or the direction of the traveling route on which the vehicle is driven by autonomous driving control). Since the predetermined area of the image is highlighted, the operator of the remote control device 3 can more easily grasp the area of interest, and the burden on the operator required for image confirmation can be further reduced.

Furthermore, the unmanned driving system 1 of the present embodiment detects an object that affects the running of the vehicle V0 from the remote control image, and when an object is detected, a predetermined area of the remote control image is changed according to the object. Highlight. At this time, the object and a predetermined area of the remote control image including the object may be highlighted. This makes it easier to grasp objects that may interfere with the vehicle V0 to be remotely controlled, and further reduces the burden on the operator required for image confirmation.

Furthermore, the unmanned driving system 1 of this embodiment detects an object existing on a route that intersects the travel route on which the vehicle V0 travels under autonomous driving control from the remote control image, and when an object is detected, Highlight. At this time, the type of object may be identified and the object may be highlighted depending on the approachability to the vehicle depending on the type of object. This makes it easier to grasp objects that may interfere with the vehicle V0 to be remotely controlled, and further reduces the burden on the operator required for image confirmation.

Furthermore, the unmanned driving system 1 of this embodiment detects an object that affects the running of the vehicle V0 from the remote control image, and when an object is detected, the type of the object, the distance to the object, the speed of the object, etc. Alternatively, object information including at least one of the relative speeds between the object and the vehicle V0 is specified, and the object information is displayed superimposed on the remote control image together with object detection information indicating that the object has been detected. At this time, information indicating how the object approaches the vehicle may be displayed as the object detection information. This makes it easier to grasp detailed information about objects that may interfere with the vehicle V0 to be remotely controlled, and further reduces the burden on the operator required for image confirmation.

Furthermore, in the unmanned driving system 1 of this embodiment, the remote control images include multiple types of direction images taken from the vehicle V0 in multiple directions around the vehicle, and depending on the state of the vehicle, the direction images with high importance are selected. Highlight the whole thing. For example, when a vehicle is traveling within a lane, the front image, left side image, and right side image of the vehicle are highlighted as direction images, and when the vehicle moves from the lane in which it is traveling to the right lane, The front image, right side image, and right rear image of the vehicle are highlighted, and when the vehicle moves from the lane in which it is traveling to the left lane, the front image, left side image, and left rear image of the vehicle are highlighted, and the vehicle is When a travel route is generated by autonomous travel control, the direction image corresponding to the direction of the travel route is highlighted. This makes it easier for the operator of the remote control device 3 to grasp the area of interest, and further reduces the burden on the operator required for image confirmation.

Furthermore, the unmanned driving system 1 of this embodiment detects an object that affects the running of the vehicle V0 from the remote control image, and when an object is detected, highlights it in a different color depending on the state of the object. . At this time, the remote control image includes multiple types of directional images taken from the vehicle V0 in multiple directions around the vehicle, and when an object is detected, it is possible to determine whether or not the detected object is displayed and the direction in which the object approaches. Accordingly, the outer frames of multiple types of direction images may be highlighted in different colors. At this time, images for remote control include multiple types of directional images taken from the vehicle in multiple directions around it, and a top view image of the vehicle superimposed on a map image, satellite image, or aerial image of the current location of the vehicle. Detects objects that affect the running of the vehicle from multiple types of directional images, including the displayed bird's-eye view image, and when an object is detected, displays an image showing the detected object overlaid on the bird's-eye view image. You may. This makes it easier for the operator of the remote control device 3 to grasp the area of interest, and further reduces the burden on the operator required for image confirmation.

1...Unmanned driving system V0...Vehicle 21...Communication unit 22...Sensor 221F...Front camera 221L...Left camera 221R...Right side camera 221B...Rear camera 221LB...Left rear camera 221RB...Right rear camera 222a...Radar device 222b, 222c...Rider Device 23...Obstacle recognition section 24...Route setting section 25...Travel control section 26...Vehicle information acquisition section 3...Remote control device 31...Communication section 32...Input section 33...Display section PV...Overhead view screen LV...Left side screen RV... Right side screen BV...Rear screen LBV...Left rear screen RBV...Right rear screen 34...Display control unit 35...Remote control unit 4...Vehicle management center 5...Management server Ra...Avoidance route

Claims (13)

When a vehicle equipped with an autonomous driving control function is driven by autonomous driving control based on remote control from the outside of the vehicle, acquiring a surrounding image of the surroundings of the vehicle from the vehicle,
Displaying the acquired surrounding image as a remote control image of the vehicle,
In a method for displaying a remote control image, the method includes highlighting a predetermined area of the remote control image according to a state of the vehicle,
The remote control image includes a plurality of types of directional images taken from the vehicle in a plurality of surrounding directions,
Highlighting the entire direction image with high importance according to the state of the vehicle,
When the vehicle is traveling within a lane, highlighting a front image, a left side image, and a right side image of the vehicle as the direction image,
When the vehicle moves from the lane in which it is traveling to the right lane, highlighting the front image, right side image, and right rear image of the vehicle,
A remote control image display method that highlights a front image, a left side image, and a left rear image of the vehicle when the vehicle moves from the lane in which it is traveling to the left lane. The method for displaying a remote control image according to claim 1, wherein a predetermined area of the remote control image is highlighted depending on the traveling direction of the vehicle. The remote control according to claim 2, wherein a predetermined area of the remote control image is highlighted, with the direction in which the vehicle is traveling or the direction of a travel route along which the vehicle is driven under autonomous driving control as the traveling direction. How to display images. Detecting an object that affects the running of the vehicle from the remote control image,
2. The method for displaying a remote control image according to claim 1, wherein when the object is detected, a predetermined area of the remote control image is highlighted depending on the object. The remote controller according to claim 4, wherein when an object that affects the running of the vehicle is detected from the remote control image, the object and a predetermined area of the remote control image including the object are highlighted. How to display images for operation. Detecting, from the remote control image, an object present on a route that intersects with a travel route on which the vehicle travels under autonomous driving control;
The method for displaying images for remote control according to claim 1, wherein when the object is detected, the object is highlighted. 7. The remote control device according to claim 6, wherein when the object is detected, the type of the object is specified, and the object is highlighted depending on the possibility of approaching the vehicle depending on the type of the object. How to display images. Detecting an object that affects the running of the vehicle from the remote control image,
When the object is detected, specifying object information including at least one of the type of the object, the distance to the object, the speed of the object, or the relative speed of the object and the vehicle;
2. The method for displaying a remote control image according to claim 1, wherein the object information is displayed superimposed on the remote control image together with object detection information indicating that the object has been detected. 9. The method for displaying images for remote control according to claim 8, wherein information indicating how the object approaches the vehicle is displayed as the object detection information. Detecting an object that affects the running of the vehicle from the remote control image,
2. The method for displaying images for remote control according to claim 1, wherein when the object is detected, the object is highlighted in a different color depending on the state of the object. The remote control image includes a plurality of types of directional images taken from the vehicle in a plurality of surrounding directions,
According to claim 10, when the object is detected, the outer frames of the plurality of types of directional images are highlighted in different colors depending on whether the detected object is displayed and the approaching direction of the object. How to display images for remote control. The remote control image is obtained by superimposing a top view image of the vehicle on a plurality of types of directional images taken from the vehicle in a plurality of surrounding directions, and a map image, satellite image, or aerial image of the current position of the vehicle. including the displayed bird's-eye view image;
Detecting objects that affect the running of the vehicle from the plurality of types of direction images,
12. The method for displaying images for remote control according to claim 11, wherein when the object is detected, an image showing the detected object is displayed superimposed on the bird's-eye view image. A remote control device that remotely controls a vehicle equipped with an autonomous driving control function from outside the vehicle and causes it to travel under autonomous driving control,
an image acquisition means for acquiring surrounding images captured from the vehicle in a plurality of directions around the vehicle;
Display means for displaying the acquired surrounding image as a remote control image of the vehicle, including a plurality of types of directional images taken from the vehicle in a plurality of surrounding directions;
display control means for highlighting a predetermined area of the remote control image according to the state of the vehicle;
Highlighting the entire direction image with high importance according to the state of the vehicle,
When the vehicle is traveling within a lane, highlighting a front image, a left side image, and a right side image of the vehicle as the direction image,
When the vehicle moves from the lane in which it is traveling to the right lane, highlighting the front image, right side image, and right rear image of the vehicle,
A remote control device comprising: an image enhancement unit that highlights a front image, a left side image, and a left rear image of the vehicle when the vehicle moves from the lane in which it is traveling to the left lane.

Images