JP2022153395A - Image display unit and image display method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To achieve, in a case of an automatic driving vehicle in which a driver is absent, communication with a person outside the vehicle and contribute to smooth and safe driving.

SOLUTION: When receiving standby information to the effect that walking of a pedestrian is prioritized from an automatic driving device 990, a detection unit 730 performs analysis processing on a photographic image transmitted from a photographing unit 910 and detects the attitude and line of sight of the pedestrian. The detection unit 730 then transmits a result of detection to a display control unit 750. When a driver is not in a vehicle CR, the display control unit 750 determines whether the pedestrian is in the attitude of a preparation posture for moving in front of the vehicle CR and the line of sight of the pedestrian is directed to the vehicle CR on the basis of the result of detection performed by the detection unit 730. When a result of the determination is affirmative, the display control unit 750 performs control of displaying, on a screen 955, an image imitating a gesture performed by the driver of the vehicle CR to the pedestrian, the gesture urging a movement in front of the vehicle CR.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2023,JPO&INPIT

Description

The present invention relates to an image display device, an image display method, an image display program, and a recording medium in which the image display program is recorded.

2. Description of the Related Art In recent years, efforts have been made to develop unmanned vehicles for practical use that do not require operation by a driver. As one of the technologies related to such unmanned vehicles, there is a technique for taking countermeasures against people outside the vehicle, and various techniques for taking such countermeasures have been proposed.

As one of such proposed technologies, when an automatically driving vehicle recognizes a person, there is a technology that notifies the person that the person is recognized by the automatically driving vehicle (see Patent Document 1: hereinafter, “Conventional (referred to as "Example 1"). In the technique of the prior art example 1, processing for detecting a person existing around an automatically driving vehicle is performed from a photographed image photographed by a camera. Then, when a person is detected by the detection process, the person is irradiated with a laser beam or a sound is output.

Moreover, although it is not a technology related to an unmanned vehicle, there is a technology that calls attention to pedestrians in an easy-to-understand manner (see Patent Document 2: hereinafter referred to as "conventional example 2"). In the technique of Conventional Example 2, a pedestrian is identified based on an image captured by a camera, and determination processing is performed to determine whether or not the pedestrian is present in a set area. Then, when a pedestrian exists in the set area, an image of a pedestrian crossing or the like is projected onto the road surface.

JP 2017-159881 A JP 2008-007079 A

By the way, in order to drive smoothly and safely on roads, it is necessary for the driver to give signals to pedestrians to encourage pedestrians to preferentially walk at crosswalks, and for the driver to operate other vehicles to encourage other vehicles to run ahead. Communication such as signaling to the driver is important, but there are times when the driver is not present when driving an autonomous vehicle. In the technique of the conventional example 1 described above, the person is notified that the person is recognized by the automatic driving vehicle, but since the notification prompting the preferential walking is not performed, the person who receives the notification It is common to stay in place and wait for the vehicle to pass. As a result, the technique of Conventional Example 1 is not sufficient as a notification for smooth and safe road travel.

On the other hand, if the technique of Conventional Example 2 is applied to the technique related to an unmanned vehicle, it becomes possible to communicate with a pedestrian to encourage pedestrians to preferentially walk in an autonomous vehicle. However, in the method of projecting an image onto the road, depending on the condition of the road such as unevenness, the image projected onto the road may be distorted and the content of the image may not be conveyed to pedestrians. As a result, in the technique of the conventional example 2, there is room for improvement in the notification for smooth and safe road driving.

For this reason, even in the case of an autonomous vehicle without a driver, there is a technology that can contribute to smooth and safe driving by realizing communication with people outside the vehicle in the same way as in the case of a manned vehicle. Desired. Meeting such demands is one of the problems to be solved by the present invention.

The invention according to claim 1 is an image display device arranged in a vehicle capable of autonomous travel,
Displaying an image simulating a gesture made by a driver in the vehicle to the person outside the vehicle, the image being visible to the person outside the vehicle when the driver is not present in the vehicle. The image display device is characterized by comprising a display control unit that causes the image to be displayed.

The invention according to claim 10 is an image display method used in an image display device that is arranged in a vehicle capable of autonomous driving and has a display control unit, wherein when the driver of the vehicle is not present, A display control step of causing the display control unit to display an image simulating a gesture made by a driver in the vehicle toward the person, which is visible to a person outside the vehicle. This image display method is characterized by:

According to an eleventh aspect of the invention, there is provided an image display program that causes a computer of an image display device to execute the image display method according to the tenth aspect.

According to a twelfth aspect of the invention, there is provided a recording medium in which the image display program according to the eleventh aspect is recorded so as to be readable by a computer of an image display device.

1 is a block diagram showing the configuration of an image display device according to one embodiment of the present invention; FIG. 1 is a block diagram schematically showing the configuration of an image display device according to one embodiment of the present invention; FIG. 3 is a diagram for explaining the arrangement relationship of a projection unit, a half mirror, and a screen in the display unit of FIG. 2; FIG. 3 is a diagram referred to when describing the operation of the image display device of FIG. 2; FIG. 3 is a flowchart for explaining display control processing of an image simulating a driver by the image display device of FIG. 2; FIG. 6 is a flowchart for explaining the first type image display control process of FIG. 5; FIG. FIG. 6 is a flowchart for explaining the second type image display control process of FIG. 5; FIG. FIG. 4 is a diagram for explaining how an image simulating a driver displayed on a screen is viewed from the outside of the vehicle;

An embodiment of the present invention will be described below with reference to FIG. In the following description and drawings, the same or equivalent elements are denoted by the same reference numerals, and overlapping descriptions are omitted.

[Constitution]
FIG. 1 shows a block diagram of the configuration of an image display device 700 according to one embodiment. The image display device 700 is arranged in a vehicle CR capable of autonomous travel. In this embodiment, in addition to the image display device 700 , the image display device 700 as well as the imaging unit 910 and the display unit 950 are arranged in the vehicle CR, and these are connected to the image display device 700 . An automatic driving device 990 is arranged in the vehicle CR, and the automatic driving device 990 is connected to the image display device 700 .

The photographing unit 910 described above is provided with a camera that photographs at least the front of the vehicle CR in the photographing field of view including the traveling direction of the vehicle CR. A photographed result (captured image) by the photographing unit 910 is sent to the image display device 700 . Although not shown, the photographing result of the photographing unit 910 is also sent to the automatic driving device 990, and is utilized in the automatic driving control by the automatic driving device 990.

The display unit 950 described above includes a projection unit 951 and a screen 955 in this embodiment.

The projection unit 951 is configured with a video projector. Projector 951 receives display image data sent from image display device 700 . Then, the projection unit 951 projects and displays an image corresponding to the display image data on the screen 955 .

The screen 955 is located in front of the backrest of the front seat in this embodiment. The screen 955 displays the image projected by the projection unit 951 so as to be visible to people outside the vehicle CR.

The automatic driving device 990 described above controls the starting, acceleration/deceleration, steering, stopping, etc. of the vehicle CR according to the driving plan and the driving environment. The automatic driving device 990 includes a storage unit (not shown) that stores map information. Then, the automatic driving device 990 uses driving information sent from a driving information detection unit (not shown) such as a vehicle speed detection unit and a position detection unit, and a photographed image sent from the imaging unit 910, etc., to obtain map information. It refers to it appropriately and performs automatic operation control.

As such automatic driving control, for example, when the automatic driving device 990 detects a pedestrian trying to walk on a crosswalk on a road without traffic lights, the vehicle CR is stopped until the pedestrian walks on the crosswalk. is to be performed. When performing the control, the automatic driving device 990 generates standby information indicating that pedestrian walking (crossing) is prioritized, and sends it to the image display device 700 .

In addition, as such automatic driving control, the automatic driving device 990 detects, for example, that another vehicle traveling in the oncoming lane is waiting for the passage of the vehicle CR for a right turn or the like, depending on the driving environment. In order to give precedence (priority) to the right turn of the other vehicle, the vehicle CR is controlled to decelerate or stop. Then, when performing the control, the automatic driving device 990 generates standby information indicating that the movement of the other vehicle precedes (prioritizes), and sends it to the image display device 700 . Here, other vehicles include automobiles, motorcycles, bicycles, and the like.

In addition, as such automatic driving control, the automatic driving device 990, for example, when it detects that another vehicle is waiting for the movement of the vehicle CR to park in the parking lot, according to the driving environment, the other vehicle In order to advance the parking in the parking lot, control is performed to stop the vehicle CR. Then, when performing the control, the automatic driving device 990 generates standby information indicating that the movement of the other vehicle precedes (prioritizes), and sends it to the image display device 700 .

<Configuration of image display device 700>
The configuration of the image display device 700 will be described.

The image display device 700 includes a storage section 710 and a detection section 730, as shown in FIG. The image display device 700 also includes a display control section 750 .

The storage unit 710 described above includes a nonvolatile storage element. The storage unit 710 stores image data simulating the driver of the vehicle CR, and camera information indicating the arrangement position and optical characteristics of the camera provided in the imaging unit 910 . A detection unit 730 and a display control unit 750 can access the storage unit 710 .

The image data simulating the driver of the vehicle CR simulates that the driver makes a gesture to the person in front of the vehicle CR when the person is looking at the vehicle CR. It is image data. An example of the content of the gesture is a gesture that prompts the pedestrian to move ahead of the vehicle CR. In other words, it is a gesture indicating an intention to give priority to walking (crossing) of a pedestrian who may enter the planned travel route of the vehicle CR.

Another example of the content of the gesture is a gesture prompting the operator of the other vehicle to move the other vehicle ahead of the vehicle CR. In other words, it is a gesture that indicates the operator of another vehicle, which may enter the planned travel route of the vehicle CR, to give priority to the travel of the other vehicle.

The detection unit 730 described above receives the captured image sent from the imaging unit 910 . Then, when receiving the standby information sent from the automatic driving device 990, the detection unit 730 analyzes the captured image, and obtains information about pedestrians who may enter the planned travel route of the vehicle CR. Alternatively, information about other vehicles that may enter the planned travel route of the vehicle CR and the operator of the other vehicle (this information is also referred to as "information about other moving bodies") is detected. That is, the detection unit 730 detects the presence or absence of a person (a pedestrian, an operator of another vehicle, etc.) or another vehicle that may enter the planned travel route of the vehicle CR.

Note that when another vehicle is detected, the detection unit 730 detects the distance between the vehicle CR and the other vehicle based on the analysis result of the captured image. A detection result by the detection unit 730 is sent to the display control unit 750 . Details of the processing by the detection unit 730 will be described later.

The display control unit 750 described above receives information about other moving objects sent from the detection unit 730 . Subsequently, when there is no driver in the vehicle CR and the standby information sent from the automatic driving device 990 is received, the display control unit 750 performs , when it is determined that an image simulating a driver should be displayed, the image data in the storage unit 710 is read to generate display image data. Then, the display control section 750 sends the display image data to the display section 950 . Details of the processing by the display control unit 750 will be described later.

In this embodiment, there may be a driver detection section that detects that a driver is present in the vehicle CR. The driver detection unit may detect that the driver is present in the vehicle CR from a signal from a seat sensor or the like installed in the seat of the vehicle CR.

Further, the driver detection unit may detect that the driver is present in the vehicle CR depending on the type of driving mode of automatic driving executed by the automatic driving device 990 . For example, when the automatic driving device 990 is executing a driving mode such as an "unmanned driving mode" that is executed when the driver is clearly not present in the vehicle CR, the driver detection unit detects that the driver does not exist in the vehicle CR.

[motion]
The operation of the image display device 700 configured as described above will be described mainly focusing on the processing by the detection unit 730 and the display control unit 750 .

As a premise, it is assumed that the imaging unit 910 has started its operation and is sequentially sending the imaging results (captured images) to the image display device 700 and the automatic driving device 990 . Further, the automatic driving device 990 has started its operation, and when it generates the standby information described above, it sends the standby information to the image display device 700 .

<Processing for detecting information on other moving bodies>
The process of detecting information on other moving bodies is executed when standby information sent from the automatic driving device 990 is received. When the standby information sent from the automatic driving device 990 is information to the effect that priority is given to walking by pedestrians, the detection unit 730 detects information sent from the photographing unit 910 during the process of detecting information about other moving bodies. The captured images are analyzed to detect pedestrians in the images.

The detection unit 730 may analyze the photographed image sent from the photographing unit 910 and further detect at least one of the detected posture, line of sight, and face direction of the pedestrian. Then, the detection unit 730 uses the detection result of the pedestrian, the detection result of at least one of the pedestrian's posture, line of sight, and face direction as information about the pedestrian as standby information indicating that priority is given to walking of the pedestrian. are sequentially sent to the display control unit 750 during the period of receiving the Further, the detection unit 730 may particularly detect the presence or absence of a person (pedestrian, operator of another vehicle, etc.) or another vehicle that may enter the planned travel route of the vehicle CR.

Further, when the standby information sent from the automatic driving device 990 is information to the effect that the movement of the other vehicle precedes the movement of the other vehicle in the process of detecting information about the other moving body, the detection unit 730 detects the information sent from the imaging unit 910. The captured image is analyzed to detect the other vehicle and the operator of the other vehicle in the image. Note that the detection unit 730 may analyze the captured image sent from the imaging unit 910 and further detect the detected motion of the other vehicle and the line of sight of the operator.

Further, the detection unit 730 detects the distance between the vehicle CR and the other vehicle based on the captured image sent from the imaging unit 910 and the camera information stored in the storage unit 710 . Then, the detection unit 730 detects the other vehicle and the operator of the other vehicle, the motion of the other vehicle, the detection result of the line of sight or the direction of the face of the operator of the other vehicle, and the vehicle CR and the other vehicle. The detection result of the distance to the other vehicle is sequentially sent to the display control unit 750 as information about the other vehicle and the operator during the period in which the standby information indicating that the other vehicle is to move ahead is received.

<Display control processing for an image simulating a driver>
In the display control process of the image simulating the driver, the display control unit 750 first determines whether an input indicating that the driver is present in the vehicle CR has been made to the input unit. By determining whether or not there is no driver in the vehicle CR, absence determination processing is performed. When the result of the non-existence determination process is negative, the display control unit 750 repeats the non-existence determination process.

When the result of the non-existence determination process is affirmative, the display control unit 750 performs the standby information acquisition determination process of whether or not the standby information sent from the automatic driving device 990 has been received. If the result of the standby information acquisition determination process is negative, the standby information acquisition determination process is ended, and the non-existence determination process described above is started again.

On the other hand, if the result of the standby information acquisition determination process is affirmative, the display control unit 750 determines whether the content of the standby information is standby information to the effect that "walking of pedestrians is prioritized" or "other Information determination processing is performed as to whether or not the information is standby information to the effect that the movement of the vehicle is to be preceded.

According to the information determination process, if the content of the standby information is information to the effect that "walking of pedestrians is prioritized", the display control unit 750 performs , determination processing (posture determination processing) is performed to determine whether or not the pedestrian is ready to move in front of the vehicle CR (attitude to walk on the crosswalk in front of the vehicle). Here, the ready state is a state in which the pedestrian is about to move (cross) in front of the vehicle CR. In other words, it is the state before the pedestrian starts to cross in front of the vehicle CR.

The display control unit 750 also performs determination processing (line-of-sight determination processing) as to whether or not the line of sight of the pedestrian is directed to the vehicle CR based on the information regarding the pedestrian sent from the detection unit 730 . In the present embodiment, the line-of-sight determination process can be rephrased as a process of determining whether or not the pedestrian recognizes the vehicle CR. That is, when the result of the line-of-sight determination process is affirmative, it is determined that the pedestrian recognizes the vehicle CR. When the result of the posture determination process or the line-of-sight determination process is affirmative, the display control unit 750 reads the image data in the storage unit 710 and generates display image data. Then, the display control section 750 sends the display image data to the display section 950 . As a result, the screen 955 displays an image that simulates a gesture that the driver makes to the pedestrian to encourage forward movement of the vehicle CR.

Note that the display control unit 750 may read the image data in the storage unit 710 and generate the display image data only when both the posture determination process and the line-of-sight determination process described above are affirmative.

In addition, the posture determination processing and line-of-sight determination processing can be omitted. That is, when it is determined that there is a pedestrian based on the information about the pedestrian sent from the detection unit 730, the display control unit 750 (even if the pedestrian's line of sight is not directed to the vehicle CR) The display image data may be generated by reading the image data in the storage unit 710 .

On the other hand, if the content of the standby information is "information to the effect that the movement of the other vehicle precedes" as a result of the information determination processing, the display control unit 750 detects the other moving object sent from the detection unit 730 and the operation information. Based on the information regarding the operator, whether or not the other vehicle is waiting for the vehicle CR to move, the line of sight of the operator is directed to the vehicle CR, and the distance between the vehicle CR and the other vehicle is within a predetermined distance. or determination processing is performed. Then, when the result of the determination process is affirmative, the display control unit 750 reads the image data in the storage unit 710 and generates display image data. Then, the display control section 750 sends the display image data to the display section 950 . As a result, an image imitating a gesture made by the driver to the operator of the other vehicle to prompt the other vehicle to move ahead of the vehicle CR is displayed on the screen 955 . Here, the predetermined distance is determined in advance based on experiments, simulations, experiences, etc., from the viewpoint of enabling the operator of the other vehicle to recognize an image simulating the driver of the vehicle CR.

It should be noted that in the determination process, it may not be determined whether or not the line of sight of the operator of the other vehicle is directed to the vehicle CR. That is, when it is determined that there is another moving body (another vehicle or an operator of the other vehicle) based on the information about the other moving body and the operator sent from the detection unit 730, (the line of sight of the operator is The display control unit 750 may read the image data in the storage unit 710 and generate the display image data (even if it is not directed toward the vehicle CR).

As described above, in the present embodiment, when the standby information indicating that pedestrian walking is prioritized is received from the automatic driving device 990, the detection unit 730 analyzes the photographed image sent from the photographing unit 910. to detect a pedestrian in the image, and detect at least one of the detected pedestrian's posture, line of sight, and face direction. Then, the detection unit 730 sends the detection result to the display control unit 750 as information about the pedestrian.

The display control unit 750 detects the pedestrian sent from the detection unit 730 when the vehicle CR is in a state in which there is no driver and when standby information indicating that priority is given to pedestrian walking is received from the automatic driving device 900. Based on this information, it is determined whether or not the pedestrian is ready to move in front of the vehicle CR and whether or not the pedestrian's line of sight is directed to the vehicle CR. Then, when the result of the determination process is affirmative, the display control unit 750 displays an image simulating a gesture made by the driver of the vehicle CR to the pedestrian to prompt the forward movement of the vehicle CR to be displayed on the screen. 955 is displayed.

Thereby, a pedestrian who wants to walk in a crosswalk can be encouraged to walk across the pedestrian crossing by a gesture. Therefore, communication with pedestrians can be realized as in the case of a manned vehicle.

In addition, in the present embodiment, when receiving standby information from the automatic driving device 990 to the effect that the movement of the other vehicle is preceded, the detection unit 730 performs analysis processing of the captured image sent from the imaging unit 910, The other vehicle and the operator of the other vehicle are detected in the image, and the motion of the detected other vehicle and the direction of the line of sight or face of the operator are detected. Then, the detection unit 730 sends the detection result to the display control unit 750 as information about the other moving body and the operator.

The display control unit 750 is in a state in which there is no driver in the vehicle CR, and when standby information to the effect that the movement of the other vehicle is to precede is received from the automatic driving device 990, the display control unit 750 displays the other movement sent from the detection unit 730. Based on the information about the body and the operator, the other vehicle is waiting for the preceding movement of the vehicle CR, the operator's line of sight is directed to the vehicle CR, and the distance between the vehicle CR and the other vehicle is within a predetermined distance. A determination process of whether or not is performed. Then, when the result of the determination process is affirmative, the display control unit 750 prompts the driver of the vehicle CR to the operator of the other vehicle to move the other vehicle ahead of the traveling of the vehicle CR. Control is performed to display an image simulating a gesture on the screen 955 .

As a result, the operator of another vehicle waiting for the vehicle CR to pass for a right turn or the like, or the operator of another vehicle waiting for the vehicle CR to move in order to park in a parking lot, can use gestures to Advance movement can be encouraged. Therefore, as in the case of manned vehicles, it is possible to communicate with operators of other vehicles.

Therefore, according to this embodiment, in the case of an automatically driven vehicle without a driver, it is possible to achieve communication with people outside the vehicle and contribute to smooth and safe driving, as in the case of a manned vehicle. can.

[Modification of Embodiment]
The present invention is not limited to the above embodiments, and various modifications are possible.

For example, in the above embodiment, an image imitating the gesture of the driver is displayed, but an image imitating the gesture of a passenger other than the driver may be displayed.

Further, in the above embodiment, an image simulating the driver of the vehicle is projected onto the screen arranged in the backrest of the front seat. Alternatively, a transparent screen may be placed on the glass (for example, the windshield) of the vehicle, and an image simulating the driver of the vehicle CR may be projected onto the screen.

Further, in the above embodiment, the video projector displays an image simulating the driver by projecting the image onto the screen. may In addition, by displaying the image on a hologram display or the like, the occupant of the vehicle may be viewed three-dimensionally by an outsider.

Also, the image (moving image) simulating the driver may be changed based on the information regarding the pedestrian or the operator of the other vehicle detected by the detection unit. For example, the image may be displayed so that the line of sight of the driver in the image faces the direction in which the pedestrian or the operator of the other vehicle exists. It is also possible to detect that a pedestrian or an operator of another vehicle has bowed, and display an image in which the driver also bows in the image accordingly. By doing so, the pedestrian or the operator of the other vehicle can recognize that the other party (vehicle) recognizes the presence of the other vehicle, so that the pedestrian or the operator of the other vehicle can feel secure. can give.

In addition, when an outside person visually recognizes the passengers of the vehicle, in addition to projecting the image on the screen as described above, so-called projection mapping technology can be used to display, for example, a three-dimensional human-shaped structure on the front seat. An image simulating a passenger may be projected onto the three-dimensional structure. In this case, it is desirable that the three-dimensional structure can be stored in the seat.

In addition, when displaying an image simulating a passenger on a display or the like, the image is transformed by affine transformation or the like according to the direction in which the outsider is present. A configuration may be adopted in which an image simulating a passenger can be visually recognized without a sense of incongruity when viewed from above. Note that when the affine transformation or the like is not performed, the image may be displayed only when an outsider is present in a direction in which the image simulating the passenger can be visually recognized without discomfort. For example, when an outsider is present in front of the vehicle, when the outsider can display only an image simulating a passenger passenger without a sense of discomfort, the outsider can be displayed in front of the vehicle. The image may be displayed only when the presence is detected.

Also, in the above embodiment, a person in the traveling direction of the vehicle is detected based on the camera image. On the other hand, a person in the traveling direction of the vehicle may be detected based on the known Lidar (Light Detection and Ranging) measurement technology.

Note that part or all of the detection unit and display control unit of the image display device in the above embodiment is configured as a computer as a calculation unit including a central processing unit (CPU), etc., and is prepared in advance. A part or all of the functions of these elements may be performed by executing the program on the computer. This program is recorded in a computer-readable recording medium such as a hard disk, CD-ROM, or DVD, and is loaded from the recording medium and executed by the computer. In addition, this program may be acquired in the form of being recorded on a portable recording medium such as a CD-ROM or DVD, or may be acquired in the form of distribution via a network such as the Internet. good too.

Further, in the above embodiment, the automatic driving device 990 and the image display device 700 are described separately in FIG. 1 for convenience of explanation. ) may be included in the automatic driving device 990 .

An embodiment of the present invention will now be described with reference to FIGS. 2 to 8. FIG. In the following description and drawings, the same or equivalent elements are denoted by the same reference numerals, and overlapping descriptions are omitted.

[Constitution]
FIG. 2 is a block diagram showing a schematic configuration of an image display device 100 according to an embodiment. The image display device 100 is one aspect of the image display device 700 (see FIG. 1) according to the embodiment described above.

The image display device 100 is arranged in a vehicle CR capable of autonomous travel. Also, in this embodiment, the vehicle CR is provided with a photographing unit 210 and a display unit 250 , which are connected to the image display device 100 . An automatic driving device 290 is arranged in the vehicle CR, and the automatic driving device 290 is connected to the image display device 100 .

In this embodiment, the photographing unit 210 includes a camera for photographing at least the front of the vehicle CR in a photographing field of view including the traveling direction of the vehicle CR. The photographing unit 210 may include a camera capable of all-around photographing, or may be configured to be capable of all-around photographing by combining a plurality of cameras. A photographed result (captured image) by the photographing unit 210 is sent to the image display device 100 . Although not shown, the photographing result of the photographing unit 210 is also sent to the automatic driving device 290, and is utilized in the automatic driving control by the automatic driving device 290. That is, the imaging unit 210 functions as the imaging section 910 of the above-described embodiment.

The display unit 250 described above includes a projection unit 251, a half mirror 253, and a screen 255 in this embodiment. Here, FIG. 3 shows the arrangement relationship of the projection unit 251, the half mirror 253 and the screen 255. As shown in FIG.

The projection unit 251 is configured with an image projector and is arranged near the footrest of the front seat. The projection unit 251 receives display image data sent from the image display device 100 . Then, the projection unit 251 irradiates the half mirror 253 with image light corresponding to the display image data.

A half mirror 253 is arranged above the front window. The half mirror 253 reflects the image light emitted from the video projector. The image light thus reflected travels toward the screen 255 .

The screen 255 is located in front of the backrest of the front seat in this embodiment. The screen 255 receives image light projected by the projection unit 251 and reflected by the half mirror 253, and displays an image visible to people outside the vehicle CR.

That is, the display unit 250 functions as the display section 950 of the embodiment described above.

The automatic driving device 290 described above controls the starting, acceleration/deceleration, steering, stopping, etc. of the vehicle CR according to the driving plan and the driving environment. The automatic driving device 290 includes a storage unit (not shown) that stores map information. Then, the automatic driving device 290 uses the travel information sent from a travel information detection unit (not shown) such as a vehicle speed detection unit and a position detection unit, and the photographed image sent from the photographing unit 210, etc., to obtain map information. is appropriately referenced to perform automatic operation control.

As such automatic driving control, when the automatic driving device 290 detects a pedestrian walking on a crosswalk on a road without traffic lights, the vehicle CR is stopped until the pedestrian walks on the crosswalk. It's like When performing the control, the automatic driving device 290 generates standby information to the effect that priority is given to pedestrian walking, and sends it to the image display device 100 . After transmission of the standby information, when the pedestrian walking on the pedestrian crossing is no longer detected, the automatic driving device 290 generates standby cancellation information and sends it to the image display device 100 .

In addition, as such automatic driving control, the automatic driving device 290 detects, for example, that another vehicle traveling in the oncoming lane is waiting for the passage of the vehicle CR for a right turn or the like, depending on the driving environment. Then, the vehicle CR is controlled to decelerate or stop so that the other vehicle can turn right. When performing the control, the automatic driving device 290 generates standby information to the effect that the movement of the other vehicle is preceded, and sends it to the image display device 100 . After the transmission of the standby information, when the other vehicle waiting for the passage of the vehicle CR is no longer detected, the automatic driving device 290 generates standby release information and sends it to the image display device 100 .

Further, as such automatic driving control, the automatic driving device 290, for example, when it detects that the other vehicle is waiting for the movement of the vehicle CR to park in the parking lot, according to the driving environment, the other vehicle In order to advance the parking in the parking lot, control is performed to stop the vehicle CR. When performing the control, the automatic driving device 290 generates standby information to the effect that the movement of the other vehicle is preceded, and sends it to the image display device 100 . After the transmission of the standby information, when the other vehicle waiting for the movement of the vehicle CR is no longer detected, the automatic driving device 290 generates standby release information and sends it to the image display device 100 .

<Configuration of image display device 100>
Next, the configuration of the image display device 100 will be described.

FIG. 2 is a block diagram showing a schematic configuration of an image display device 100 according to an embodiment. The image display device 100 includes a processing control unit 110 and a storage unit 120, as shown in FIG.

The processing control unit 110 described above controls the entire image display device 100 and executes various processing. The processing control unit 110 includes a central processing unit (CPU) as a computing means and its peripheral circuits. Various functions of the image display device 100 are realized by the processing control unit 110 executing various programs. These functions include the functions of the detection unit 730 and the display control unit 750 in the above-described embodiments. Details of the processing executed by the processing control unit 110 will be described later.

The program executed by the processing control unit 110 is recorded in a computer-readable recording medium such as a hard disk, CD-ROM, DVD, etc., and is loaded from the recording medium and executed. In addition, this program may be acquired in the form of being recorded on a portable recording medium such as a CD-ROM or DVD, or may be acquired in the form of distribution via a network such as the Internet. good too.

The storage unit 120 described above includes a non-volatile storage device such as a hard disk device, and stores various information data used in the image display device 100 . Such information data includes image data simulating the driver of the vehicle CR, camera information indicating the arrangement position and optical characteristics of the camera provided in the photographing unit 210, and the like. The storage unit 120 is accessible by the processing control unit 110 . That is, the storage unit 120 functions as the storage section 710 of the above-described embodiment.

The image data simulating the driver of the vehicle CR simulates that the driver makes a gesture to the person in front of the vehicle CR when the person is looking at the vehicle CR. It is the data of the image that was created. The content of the gesture is a gesture that prompts the pedestrian to move forward of the vehicle CR, and is a gesture that prompts the operator of the other vehicle to move the other vehicle ahead of the vehicle CR.

[motion]
The operation of the image display device 100 configured as described above will be described mainly focusing on the display control processing of the image simulating the driver by the processing control unit 110 . Also, the operation of the image display apparatus 100 of this embodiment will be described with reference to FIGS. 4A and 4B as appropriate. Here, FIG. 4A is a drawing showing the positional relationship between the vehicle CR and the pedestrian HA at the intersection. FIG. 4B is a drawing showing the positional relationship between the vehicle CR, the pedestrian HB, and the oncoming vehicle (another vehicle) T at the intersection. 4(A) and 4(B) indicate the planned travel route RT of the vehicle CR, and H schematically indicates the photographing range of the photographing unit 210. As shown in FIG. The photographing range H is an example. For example, when the photographing unit 210 is configured by a camera capable of photographing the entire surroundings, the photographing range H is a circular shape covering the entire periphery centered on the vehicle CR. can be represented schematically.

As a premise, it is assumed that the imaging unit 210 has started operation and is sequentially sending imaging results (captured images) to the image display device 100 . Further, it is assumed that the automatic driving device 290 has started its operation, and when it generates the standby information described above, it sends the standby information to the image display device 100 .

As shown in FIG. 5, in the display control process of the image simulating the driver, first, in step S11, the processing control unit 110 receives an input indicating that the driver is present in the vehicle CR. It is determined whether or not the driver is in the vehicle CR by determining whether or not the input unit shown in the figure is being operated. If the result of this determination is negative (step S11: N), the process of step S11 is repeated.

If the result of determination in step S11 is affirmative (step S11: Y), the process proceeds to step S12. In step S<b>12 , the processing control unit 110 determines whether or not it has received standby information from the automatic driving device 290 to the effect that “walking of pedestrians is prioritized”. If the result of this determination is affirmative (step S12: Y), the process proceeds to step S13. In step S13, the processing control unit 110 performs "first type image display control processing". Details of the processing of step S13 will be described later. Then, when the process of step S13 ends, the process returns to step S11.

If the result of determination in step S12 is negative (step S12: N), the process proceeds to step S14. In step S<b>14 , the processing control unit 110 determines whether or not it has received standby information from the automatic driving device 290 to the effect that “the movement of the other vehicle is preceded”. If the result of this determination is affirmative (step S14: Y), the process proceeds to step S15. In step S15, the processing control unit 110 performs "second type image display control processing". The details of the processing of step S15 will be described later. Then, when the process of step S15 ends, the process returns to step S11.

If the result of determination in step S14 is negative (step S14: N), the process returns to step S11. After that, the processing of steps S11 to S15 is repeated.

<Type 1 image display control processing>
The "first type image display control process" in step S13 described above will be described.

In this "type 1 image display control process", as shown in FIG. 6, first, in step S21, the processing control unit 110 analyzes the photographed image sent from the photographing unit 210, and walks in the image. A person is detected, and at least one of the detected pedestrian's posture, line of sight, and face orientation is detected.

Subsequently, in step S22, the processing control unit 110, based on the image analysis in step S21, determines that the pedestrian is ready to move in front of the vehicle CR (walking on the crosswalk), and that the pedestrian It is determined whether or not the first type image display condition that the line of sight is directed to the vehicle CR is satisfied. If the result of this determination is affirmative (step S22: Y), the process proceeds to step S23.

The processing control unit 110 analyzes the photographed image sent from the photographing unit 210, and particularly detects the presence or absence of a person in the image who may enter the planned travel route RT of the vehicle CR. may Here, the planned travel route is a travel route that the vehicle CR will travel in the future, such as a travel route set by a navigation device (not shown) installed in the vehicle CR. In this case, the processing control unit 110 detects whether or not a pedestrian exists in the photographed image, and if detected, the pedestrian is detected from the position, posture, etc. of the detected pedestrian. determines whether there is a possibility of entering the scheduled travel route RT. The processing control unit 110 detects whether or not a pedestrian exists in the photographed image, and if detected, determines that the pedestrian may enter the planned travel route RT of the vehicle CR. It is also possible to

In the case of FIG. 4A, the pedestrian HA is detected in the photographed image, and from the position, posture, etc. of the detected pedestrian HA, the possibility that the pedestrian will enter the planned travel route RT of the vehicle CR. determine whether there is In the case of FIG. 4B, the pedestrian HB is detected in the photographed image, and the pedestrian HB enters the planned travel route RT of the vehicle CR based on the detected position, posture, etc. of the pedestrian HB. Determine if it is possible.

In the case of FIGS. 4A and 4B, when it is detected that the pedestrian HA and the pedestrian HB are trying to cross the pedestrian crossing or that their line of sight is directed toward the vehicle CR, , it is determined that the pedestrians HA and HB may enter the planned travel route RT of the vehicle CR.

In step S23, the processing control unit 110 reads the image data in the storage unit 120 and generates display image data. The processing control unit 110 then sends the display image data to the projection unit 251 of the display unit 250 . As a result, the screen 255 displays an image simulating a gesture that the driver makes to the pedestrian to encourage forward movement of the vehicle CR (to encourage walking on the crosswalk). After that, the process proceeds to step S26, which will be described later.

On the other hand, if the result of determination in step S22 is negative (step S22: N), the process proceeds to step S24. In step S24, the processing control unit 110 determines whether or not an image simulating the driver of the vehicle CR is being displayed. If the result of this determination is negative (step S24: N), the process proceeds to step S26.

If the result of determination in step S24 is affirmative (step S24: Y), the process proceeds to step S25. In step S25, the processing control unit 110 performs control to erase the display of the image simulating the driver of the vehicle CR. Then, the process proceeds to step S26.

In step S<b>26 , the processing control unit 110 determines whether or not release information for releasing the standby for “prioritizing pedestrian walking” has been received from the automatic driving device 290 . If the result of this determination is negative (step S26: N), the process returns to step S21.

If the result of determination in step S26 is affirmative (step S26: Y), the process proceeds to step S27. In step S27, the process control unit 110 determines whether or not an image simulating the driver of the vehicle CR is being displayed, as in the process of step S24 described above. If the result of this determination is negative (step S27: N), the process of step S13 ends. Then, the process returns to step S11 in FIG. 5 described above.

If the result of determination in step S27 is affirmative (step S27: Y), the process proceeds to step S28. In step S28, the processing control unit 110 performs control to erase the display of the image simulating the driver of the vehicle CR in the same manner as in the processing of step S25 described above. When the control for erasing the display of the image is performed in this manner, the process of step S13 ends. Then, the process returns to step S11 in FIG. 5 described above.

<Second type image display control processing>
The "second type image display control process" in step S15 described above will be described.

As shown in FIG. 7, in this "second type image display control process", first, in step S31, the processing control unit 110 analyzes the photographed image sent from the photographing unit 210 and The operator of the vehicle and the other vehicle is detected, and the motion of the detected other vehicle and the line of sight of the operator are detected.

Subsequently, in step S32, the processing control unit 110 determines, based on the image analysis in step S31, that another vehicle is waiting for the vehicle CR to move, that the operator's line of sight is directed to the vehicle CR, and that the vehicle CR is not the other vehicle. It is determined whether or not the second type image display condition that the distance to the vehicle is within a predetermined distance is satisfied. If the result of this determination is affirmative (step S32: Y), the process proceeds to step S33.

The processing control unit 110 analyzes the photographed image sent from the photographing unit 210, and in particular, the other vehicle (the operator of the other vehicle) that may enter the planned travel route RT of the vehicle CR in the image. ) may be detected. In this case, the processing control unit 110 detects whether or not another vehicle (operator of the other vehicle) exists in the captured image, and if detected, the detected other vehicle (operator of the other vehicle) is detected. ), it is determined whether there is a possibility that the other vehicle (the operator of the other vehicle) will enter the planned travel route RT of the vehicle CR.

In the case of FIG. 4B, another vehicle T is detected in the photographed image, and from the position, behavior, etc. of the detected other vehicle T, there is a possibility that the pedestrian will enter the planned travel route RT of the vehicle CR. determine whether there is In the case of FIG. 4B, when the other vehicle T is traveling straight ahead, it blocks the planned travel route RT of the vehicle CR, so the other vehicle enters the planned travel route RT of the vehicle CR. judged to be possible.

In step S33, the processing control unit 110 reads the image data in the storage unit 120 to generate display image data. The processing control unit 110 then sends the display image data to the projection unit 251 of the display unit 250 . As a result, an image imitating a gesture made by the driver to the operator of the other vehicle to prompt the other vehicle to move ahead of the vehicle CR is displayed on the screen 255 . After that, the process proceeds to step S36, which will be described later.

On the other hand, if the result of determination in step S32 is negative (step S34: N), the process proceeds to step S34. In step S34, the processing control unit 110 determines whether or not an image simulating the driver of the vehicle CR is being displayed. If the result of this determination is negative (step S34: N), the process proceeds to step S36.

If the result of determination in step S34 is affirmative (step S34: Y), the process proceeds to step S35. In step S35, the processing control unit 110 performs control to erase the display of the image simulating the driver of the vehicle CR. Then, the process proceeds to step S36.

In step S<b>36 , the processing control unit 110 determines whether or not it has received release information from the automatic driving device 290 to release the standby for “preceding the movement of the other vehicle”. If the result of this determination is negative (step S36: N), the process returns to step S31.

If the result of determination in step S36 is affirmative (step S36: Y), the process proceeds to step S37. In step S37, the processing control unit 110 determines whether or not an image simulating the driver of the vehicle CR is being displayed, as in the processing of step S34 described above. If the result of this determination is negative (step S37: N), the process of step S15 ends. Then, the process returns to step S11 in FIG. 5 described above.

If the result of determination in step S37 is affirmative (step S37: Y), the process proceeds to step S38. In step S38, the processing control unit 110 performs control to erase the display of the image simulating the driver of the vehicle CR, similar to the processing in step S35 described above. When the control for erasing the display of the image is performed in this manner, the process of step S15 ends. Then, the process returns to step S11 in FIG. 5 described above.

Here, FIG. 8 shows an example of the appearance of the image simulating the driver (male) of the vehicle CR displayed on the screen 255 by the processing in steps S23 and S33 when viewed from the outside of the vehicle CR. It is

As described above, in this embodiment, when the standby information indicating that pedestrian walking is prioritized is received from the automatic driving device 290, the processing control unit 110 analyzes the photographed image sent from the photographing unit 210. Processing is performed to detect a pedestrian in the image and to detect at least one of the detected pedestrian's pose, line of sight, and face orientation. Then, when there is no driver in the vehicle CR, the processing control unit 110 determines that the pedestrian is ready to move in front of the vehicle CR based on the result of the analysis processing of the photographed image, and that the pedestrian is directed to the vehicle CR. Then, when the result of the determination process is affirmative, the processing control unit 110 displays an image simulating a gesture made by the driver of the vehicle CR to the pedestrian to encourage forward movement of the vehicle CR on the screen. 255 is displayed.

Thereby, a pedestrian who wants to walk in a crosswalk can be encouraged to walk across the pedestrian crossing by a gesture. Therefore, communication with pedestrians can be realized as in the case of a manned vehicle.

In addition, in this embodiment, when receiving standby information from the automatic driving device 290 to the effect that the movement of the other vehicle is preceded, the processing control unit 110 analyzes the captured image sent from the imaging unit 210. , detect the other vehicle and the operator of the other vehicle in the image, and detect the detected motion of the other vehicle and the direction of the line of sight or face of the operator. Then, when there is no driver in the vehicle CR, the processing control unit 110 determines that the other vehicle is waiting for the preceding movement of the vehicle CR based on the result of the analysis processing of the photographed image, and the line of sight of the operator is directed to the vehicle CR. and the distance between the vehicle CR and the other vehicle is within a predetermined distance. Then, when the result of the determination process is affirmative, the processing control unit 110 prompts the driver of the vehicle CR to the operator of the other vehicle to move the other vehicle ahead of the running of the vehicle CR. Control is performed to display an image simulating a gesture on the screen 255 .

As a result, the operator of the other vehicle waiting for the vehicle CR to pass for a right turn or the like, or the operator of the other vehicle waiting for the vehicle CR to move to park in the parking lot, can use gestures to Advance movement can be encouraged. Therefore, as in the case of manned vehicles, it is possible to communicate with operators of other vehicles.

Therefore, according to this embodiment, in the case of an automatically driven vehicle without a driver, it is possible to realize communication with people outside the vehicle and contribute to smooth and safe driving, as in the case of a manned vehicle. can.

[Modification of Embodiment]
The present invention is not limited to the above embodiments, and various modifications are possible.

For example, in the above embodiment, the driver displayed on the screen is male. On the other hand, the attributes of the driver (excluding children) displayed on the screen may be changed according to the attributes such as sex, age, and nationality of the pedestrian or the operator of the other vehicle. For example, when the pedestrian is a child, an image of a young woman may be displayed on the screen. This makes it possible to give the child a sense of security.

Also, a driver having the same attribute as the pedestrian or the driver of another vehicle (excluding children) may be displayed on the screen. This makes it possible to give a sense of security to pedestrians and operators of other vehicles.

Further, in the above embodiment, the present invention is applied to the vehicles shown in FIGS. 3 and 8, but the present invention may be applied to other vehicles such as trucks and heavy equipment vehicles. Of course.

Further, in the above embodiment, an image simulating the driver of the vehicle is projected onto the screen placed on the backrest of the front seat. Alternatively, a transparent screen may be placed on the glass of the vehicle (for example, the windshield or the side glass of the vehicle), and an image simulating the driver of the vehicle may be projected onto the screen.

In addition, the above examples can be appropriately modified in the same manner as the above-described embodiment.

DESCRIPTION OF SYMBOLS 100... Image display apparatus 110... Processing control unit (detection part, display control part)
700... Image display device 730... Detector 750... Display controller

Claims (1)

An image display device arranged in a vehicle capable of autonomous travel,
Displaying an image simulating a gesture made by a driver in the vehicle to the person outside the vehicle, the image being visible to the person outside the vehicle when the driver is not present in the vehicle. An image display device, comprising: a display control unit that causes the image to be displayed.

Images