JP6769187B2 - Display method of display device and display device - Google Patents

Description

The present invention relates to a display method and a display device of a display device that displays the surroundings of the own vehicle and visually assists an occupant.

Conventionally, the current position of the own vehicle is specified, and one of the images taken by a plurality of cameras provided in the own vehicle is selected and selected based on the surrounding road configuration information including the specified current position. A display method for displaying the captured image on a display device is known (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2005-309812

In the display method of the conventional display device, one of the images taken by a plurality of cameras is displayed according to the traveling condition of the own vehicle, and the viewpoint is not changed even in the same image. There wasn't. Therefore, when the relative positional relationship between the object existing around the own vehicle and the own vehicle changes, the relative positional relationship between the own vehicle and the object is displayed in the image displayed on the display device. Sometimes it was difficult to figure out.

The present invention has been made by paying attention to the above problem, and is a display device capable of easily grasping the relative positional relationship between the own vehicle and the object in the image displayed to the occupant. It is an object of the present invention to provide a display method and a display device.

In order to achieve the above object, the present invention detects an object around the own vehicle, and obtains an image from a virtual viewpoint including the object from the captured images obtained by a plurality of cameras provided in the own vehicle . In the display method of the display device that displays as visual assistance information, a step of specifying a predetermined camera for detecting the object, an object position specifying step, a virtual viewpoint setting step, and a virtual image generation step among a plurality of cameras. And have.
In the object position specifying step, the position of the object in the front-rear direction of the own vehicle or the left-right direction of the own vehicle with respect to the predetermined camera is specified.
The virtual viewpoint setting step sets the position and the position of the virtual viewpoint in a position opposite to the object with respect to a given camera, to set the orientation of the virtual viewpoint to include the field of view of the subject vehicle and the object.
In the virtual image generation step, an image is generated when viewed from the set virtual viewpoint.

As a result, the object can be easily grasped in the displayed image.

FIG. 5 is an overall system configuration diagram showing a display support system to which the display method and display device of the first embodiment are applied. It is a flowchart which shows the flow of the visual assist control processing executed by the visual assist CPU of Example 1. FIG. It is a top view which shows the positional relationship between the own vehicle and a pedestrian existing in the first position, and the virtual viewpoint at that time. It is a virtual image when viewed from the virtual viewpoint shown in FIG. 3A. It is a top view which shows the positional relationship between the own vehicle and a pedestrian existing in a second position, and a virtual viewpoint at that time. It is a virtual image when viewed from the virtual viewpoint shown in FIG. 4A. It is a top view which shows the positional relationship between the own vehicle and a pedestrian existing in a third position, and a virtual viewpoint at that time. It is a virtual image when viewed from the virtual viewpoint shown in FIG. 5A. It is a top view which shows the positional relationship between the own vehicle and the pedestrian existing in the 4th position, and the virtual viewpoint at that time. It is a virtual image when viewed from the virtual viewpoint shown in FIG. 6A. It is a top view which shows the positional relationship between the own vehicle and a pedestrian after a lapse of a predetermined time from the state shown in FIG. 6A, and the virtual viewpoint at that time. It is a virtual image when viewed from the virtual viewpoint shown in FIG. 6C. It is a top view which shows the positional relationship between the own vehicle and the pedestrian existing in the 4th position, and the virtual viewpoint at that time. It is a virtual image when viewed from the virtual viewpoint shown in FIG. 7A. It is a top view which shows the positional relationship between the own vehicle and a pedestrian after a lapse of a predetermined time from the state shown in FIG. 7A, and the virtual viewpoint at that time. It is a virtual image when viewed from the virtual viewpoint shown in FIG. 7C. This is another first example of a vehicle to which the display device of the first embodiment is applied. This is another second example of the vehicle to which the display device of the first embodiment is applied. 9 is a plan view showing the positional relationship between the own vehicle and the pedestrian existing at the 7th position shown in FIG. 9 and the virtual viewpoint at that time. It is a flowchart which shows the flow of the visual assist control processing executed by the visual assist CPU of Example 2. It is a top view which shows the positional relationship between the own vehicle and the pedestrian existing in the 8th position, and the virtual viewpoint at that time. It is a virtual image when viewed from the virtual viewpoint shown in FIG. 12A. It is a top view which shows the positional relationship between the own vehicle and the pedestrian existing in the 9th position, and the virtual viewpoint at that time. It is a virtual image when viewed from the virtual viewpoint shown in FIG. 13A. It is a top view which shows the positional relationship between the own vehicle and the pedestrian existing in the tenth position, and the virtual viewpoint at that time. It is a virtual image generated when a pedestrian moves to the left of the vehicle from the state shown in FIG. 14A. It is a virtual image generated when a pedestrian vehicle moves to the right of the vehicle from the state shown in FIG. 14A. This is another example of a vehicle to which the display device of the second embodiment is applied.

Hereinafter, a display method of the display device of the present invention and a mode for carrying out the display device will be described with reference to Example 1 shown in the drawings.

(Example 1)
First, the configuration will be described.
The display method and the display device of the display device in the first embodiment are applied to a driving support vehicle equipped with a visual support system that visually assists an occupant in the vehicle. Hereinafter, the configuration of the first embodiment will be described separately for the “overall system configuration”, the “visual assistance control configuration for the occupant”, and the “visual assistance control processing configuration”.

[Overall system configuration]
FIG. 1 is an overall system configuration diagram showing a display support system to which the display method and display device of the first embodiment are applied. Hereinafter, the overall system configuration will be described with reference to FIG.

As shown in FIG. 1, the visual support system of the first embodiment includes an object recognition sensor 21, an in-vehicle camera 22, a steering angle sensor 23, a vehicle speed sensor 24, a range position sensor 25, and a display device D. I have. The display device D includes a visual support CPU 30 (display controller) and a display display 26.

The object recognition sensor 21 is an object detection sensor provided in the own vehicle for detecting the position of an object existing around the vehicle. Here, the "object" is an object necessary for visual assistance of occupants such as pedestrians, bicycles, lane markings (parking borders) provided on the road surface, curbs, fences, walls, oncoming vehicles, and the like. ..
The object recognition sensor 21 includes a front sensor 21a mounted on the front portion of the own vehicle to search the front region of the vehicle, a rear sensor 21b mounted on the rear portion of the own vehicle to search the rear region of the vehicle, and a left side surface of the own vehicle. It has a left side sensor 21c mounted on the vehicle to search the left side region of the vehicle, and a right side sensor 21d mounted on the right side surface of the own vehicle to search the right side region of the vehicle.

The object recognition sensor 21 is typified by a commonly used laser range finder. However, in addition to the laser range finder, the position of the object may be detected by using a clearance sonar that uses ultrasonic waves. The laser range finder is a device that can irradiate an infrared laser on a target and measure the distance to the target based on the degree of reflection, and can acquire distance information to the detected object as point cloud information. ..
The distance information to the object acquired by the object recognition sensor 21 is output to the visual aid CPU 30.

The in-vehicle camera 22 is an object detection sensor provided in the own vehicle for detecting the position of an object existing around the vehicle. The in-vehicle camera 22 includes a front camera 22a mounted at the center position of the front end of the own vehicle to photograph the front region of the vehicle, and a rear camera 22b mounted at the center position of the rear end of the vehicle to photograph the front region of the vehicle. The left side camera 22c mounted on the lower left end position of the left side mirror of the own vehicle to photograph the left side area of the vehicle, and the right side camera 22c mounted on the lower right end position of the right side mirror of the own vehicle to photograph the right side area of the vehicle. It has a right side camera 22d.

The in-vehicle camera 22 is typified by a commonly used monocular camera. However, in addition to the monocular camera, a stereo camera having a plurality of photographing units may be used for shooting. The monocular camera is a camera having one photographing unit, and may be a color camera or a monochrome camera.
The captured image information captured by the in-vehicle camera 22 is output to the visual support CPU 30.

The steering angle sensor 23 is a sensor that acquires steering angle information for estimating the current tire steering angle amount and the like. The steering angle sensor signal is output to the visual aid CPU 30.

The vehicle speed sensor 24 is a sensor that acquires vehicle speed information for calculating the speed and mileage of the own vehicle, and here, a wheel speed sensor provided on the wheel is used. The vehicle speed sensor signal is output to the visual aid CPU 30.

The range position sensor 25 is a sensor for obtaining range position information of the range position (P (parking), R (reverse), D (drive), M (manual mode), etc.) selected by the select lever, and is so-called. It is an inhibitor switch. The range position signal is output to the visual aid CPU 30.

The visual support CPU 30 sets an object existing around the own vehicle based on sensor information from various sensors such as the object recognition sensor 21 and image information from the in-vehicle camera 22, and positions the object with respect to the own vehicle. To identify. Then, a virtual viewpoint is set at a position where the relative positional relationship between the own vehicle and the object is grasped according to the position of the specified object, and an image (virtual image) from the set virtual viewpoint is generated. The generated virtual image information is output to the display 26 for display.

The display 26 can use a liquid crystal display for navigation generally provided in a vehicle, a monitor provided in a remote control terminal, or the like, and displays virtual image information input from the visual support CPU 30 on a screen. Display above.

[Owner's visual support control configuration]
The visual support CPU 30 corresponds to a display controller included in the display device D mounted on the own vehicle, and as shown in FIG. 1, the visual support CPU 30 and the image switching processing unit 30A having the own vehicle state grasping unit 31 and the object position specifying unit 32. , A virtual viewpoint setting unit 33, an image generation unit 34, and an image generation processing unit 30B having an image correction unit 35.

The own vehicle state grasping unit 31 grasps the driving state of the own vehicle based on various sensor information from the steering angle sensor 23, the vehicle speed sensor 24, and the range position sensor 25. The vehicle driving information grasped by the own vehicle state grasping unit 31 is output to the object position specifying unit 32. The driving state of the own vehicle may be estimated from various sensor information.

The object position specifying unit 32 is based on the vehicle driving information from the own vehicle state grasping unit 31, the distance information from the object recognition sensor 21 to the object, and the captured image information from the in-vehicle camera 22. Judge the presence or absence of objects (such as curbs and pedestrians) that exist in the vicinity of. Then, when it is determined that the object exists, the position of the object with respect to the own vehicle is specified.
Here, if the object exists to the left of the center position in the width direction of the own vehicle, the position of the object with respect to the position of the left side camera 22c of the vehicle-mounted camera 22 is specified. That is, the position of the object existing on the left side of the vehicle in the vehicle front-rear direction is specified with reference to the position of the center of the optical axis of the left side camera 22c. Further, if the object exists to the right of the center position in the width direction of the own vehicle, the position of the object with respect to the position of the right side camera 22d of the vehicle-mounted camera 22 is specified. That is, the position of the object existing on the right side of the vehicle in the vehicle front-rear direction is specified with reference to the position of the center of the optical axis of the right side camera 22d.

The presence or absence of the object is determined by the fact that an object having a preset singular point is photographed within the photographing range photographed by the in-vehicle camera 22. Further, the position information of the object is obtained by analyzing the bird's-eye view image generated by converting the captured image taken by the left side camera 22c and the right side camera 22d into a bird's-eye view, and up to the object obtained by the object recognition sensor 21. Obtained by considering the distance. Further, timer information from the timer 32a is input to the object position specifying unit 32, and the identification of the position of the object is repeated at predetermined time intervals. The position information of the object specified by the object position specifying unit 32 is input to the image generation processing unit 30B.

The virtual viewpoint setting unit 33 is based on the vehicle driving information from the own vehicle state grasping unit 31 and the position information of the object from the object position specifying unit 32, and the own vehicle and the object existing around the own vehicle. Set the position of the virtual viewpoint that is judged to be the most appropriate for grasping the relative positional relationship with.
That is, first, it is determined whether or not the object can be displayed at any angle in the front-rear direction of the vehicle. Here, when all parts of the object are photographed within a predetermined range in the front-rear direction of the vehicle with the center positions of the optical axes of the left and right side cameras 22c and 22d in between (the object is substantially in front of the left and right side cameras 22c and 22d). When the object is photographed by either the front camera 22a, the rear camera 22b, or the left and right side cameras 22c, 22d (when the object extends long in the front-rear direction of the vehicle, such as a lane marking or a rim stone). (When), it is determined that the object can be displayed at any angle in the front-rear direction of the vehicle. The "predetermined range sandwiching the center of the optical axis of the left and right side cameras 22c and 22d" is arbitrarily set depending on the state of the captured image and the like.
The "angle in the front direction of the vehicle" is an angle when the front of the vehicle is viewed from the rear of the vehicle, and the "angle in the rear direction of the vehicle" is an angle when the rear of the vehicle is viewed from the front of the vehicle.

Then, when the object can be displayed only at any angle in the front-rear direction of the vehicle, it is determined whether or not the object exists in front of the vehicle from the center positions of the optical axes of the left and right side cameras 22c and 22d. When it is determined that the object is behind the vehicle from the center position of the optical axis of the left and right side cameras 22c and 22d, the virtual viewpoint is a position that includes the object and the front and rear wheels of the vehicle in the field of view, and the object and the vehicle itself. Set from the position in front of the vehicle to the rear of the vehicle. Further, when it is determined that the object exists in front of the vehicle from the center position of the optical axis of the left and right side cameras 22c and 22d, the virtual viewpoint is a position including the object and the front and rear wheels of the own vehicle in the field of view. And, it is set from the position behind the vehicle to the front of the vehicle from the own vehicle.
That is, the virtual viewpoint is set at a position where the relative positional relationship between the own vehicle and the object is grasped according to the position of the object.

On the other hand, when the object can be displayed at any angle in the front-rear direction of the vehicle, the relative speed between the object and the own vehicle is detected, and it is determined whether or not the relative speed is greater than zero. Then, when it is determined that the relative speed is greater than zero, the virtual viewpoint is a position that includes the object and the front and rear wheels of the own vehicle in the field of view, and is from a position behind the object and the own vehicle to the front of the vehicle. Set towards. When it is determined that the relative speed is zero or less, the virtual viewpoint is directed from the position in front of the object and the vehicle to the rear of the vehicle at a position that includes the object and the front and rear wheels of the vehicle in the field of view. And set.
The "case where the relative speed is greater than zero" is a case where the speed of the object is faster than that of the own vehicle and the object moves toward the front of the vehicle when viewed from the own vehicle. Further, "when the relative speed is zero or less" is a case where the speed of the object is slower than that of the own vehicle and the object moves toward the rear of the vehicle when viewed from the own vehicle.
The virtual viewpoint information set by the virtual viewpoint setting unit 33 is output to the image generation unit 34.

The image generation unit 34 converts the viewpoint into an image from a virtual viewpoint set according to the position of the object based on the captured image information from the in-vehicle camera 22 and the virtual viewpoint information from the virtual viewpoint setting unit 33. Then, a virtual image for grasping the relative positional relationship between the own vehicle and the object is generated.
At this time, if the virtual viewpoint is set toward the front of the vehicle, the front and rear of the object and the own vehicle are used by using the image taken by either the left and right side cameras 22c or 22d and the image taken by the front camera 22a. Generate a virtual image that includes the wheels, sides, and vehicle lower area. If the virtual viewpoint is set toward the rear of the vehicle, the image taken by either the left and right side cameras 22c or 22d and the image taken by the rear camera 22b are used to display the object and the front and rear wheels of the own vehicle. Generate a virtual image that includes the sides and the area below the vehicle.
The virtual image may not only be generated by converting, synthesizing, and selecting the image captured by the in-vehicle camera 22, but may also be generated by superimposing a CG image stored in advance on the image generated from the captured image. The CG image is created according to the map information and the road surface information, and is superimposed by performing singular point matching with the captured image.
The virtual image information generated by the image generation unit 34 is output to the image correction unit 35.

Based on the virtual image information from the image generation unit 34, the image correction unit 35 corrects the information superimposed on the virtual image, the information that causes distortion, and the like, and generates a displayable virtual image.
The virtual image information corrected by the image correction unit 35 is output to the display 26 for display.

[Visual support control processing configuration]
FIG. 2 is a flowchart showing the flow of the visual assist control process executed by the visual assist CPU of the first embodiment. Hereinafter, each step of FIG. 2 representing the visual assist control processing configuration will be described.

In step S101, the steering angle sensor 23, the vehicle speed sensor 24, and the range position sensor 25 acquire information on the driving state of the vehicle, grasp the driving state of the own vehicle, and proceed to step S102.

In step S102, following the grasp of the driving state in step S101, it is determined whether or not the object exists within the photographing range photographed by the in-vehicle camera 22. If YES (there is an object), the process proceeds to step S103. If NO (no object), the process proceeds to step S113.
Here, the presence or absence of the object is determined by analyzing the image captured by the in-vehicle camera 22 and considering the sensor information from the object recognition sensor 21.

In step S103, following the determination in step S102 that there is an object, it is determined whether or not the object can be displayed at any angle in the front-rear direction of the vehicle. If YES (displayable from both front and rear of the vehicle), the process proceeds to step S104. If NO (displayable only from one of the front and rear of the vehicle), the process proceeds to step S105.
Here, whether or not the object can be displayed at any angle in the front-rear direction of the vehicle is determined based on the shooting state by the in-vehicle camera 22.

In step S104, following the determination in step S103 that the display can be performed from both the front and rear of the vehicle, the relative speed in the vehicle front-rear direction between the object confirmed to exist in step S102 and the own vehicle is detected, and the detected relative speed. Is greater than zero, that is, whether the speed of the object is faster than that of the own vehicle. If YES (relative velocity> zero), the process proceeds to step S108. If NO (relative velocity ≤ zero), the process proceeds to step S110.
Here, the relative speed between the object and the own vehicle is detected based on the movement (movement direction) of the object within the predetermined time counted by the timer 32a.

In step S105 (object position identification step), following the determination in step S103 that the display can be performed only from one of the front and rear of the vehicle, the position of the object whose existence is confirmed in step S102 is higher than that of the left and right side cameras 22c and 22d. Determine if it is in the rear position. If YES (behind the side camera), the process proceeds to step S106. If NO (in front of the side camera), the process proceeds to step S108.
At this time, the position of the object is determined based on the position of the center of the optical axis of the left and right side cameras 22c and 22d, and the object is located at the center position of the optical axis based on the captured images of the left and right side cameras 22c and 22d. It is determined whether the vehicle is located in the front position of the vehicle or in the rear position of the vehicle.

In step S106 (virtual viewpoint setting step), following the determination in step S105 that the object is located behind the side camera, the position includes the object and the front and rear wheels of the own vehicle in the field of view. A virtual viewpoint is set from the position in front of the vehicle (left and right side cameras 22c, 22d) toward the rear of the vehicle, and the process proceeds to step S107.
Here, when the object is located behind the vehicle with respect to the left and right side cameras 22c and 22d, either the left or right side cameras 22c or 22d can be used to move the front surface of the object (the surface facing the front of the vehicle) and the rear of the object. It is probable that the background has been photographed. Therefore, by setting the virtual viewpoint toward the rear of the vehicle, the relative positional relationship between the own vehicle and the object is determined by using the captured images of the left and right side cameras 22c and 22d, including the object and the front and rear wheels of the own vehicle. It is possible to generate a virtual image to be grasped.
That is, in this step S106, the virtual viewpoint is set at a position where the relative positional relationship between the own vehicle and the object is grasped according to the position of the object.

In step S107 (image generation step), following the setting of the virtual viewpoint in step S106, the virtual image obtained by converting the viewpoint to the image viewed from the virtual viewpoint set in step S106 is converted into the image of the left and right side cameras 22c and 22d. It is generated using the image of either one and the rear camera 22b, and the process proceeds to step S112.
At this time, the generated virtual image includes the object, the front and rear wheels, the side surface, and the vehicle lower region of the own vehicle, and is a virtual image for grasping the relative positional relationship between the own vehicle and the object. Further, a necessary CG image or the like may be superimposed as appropriate according to the virtual image to be generated.

In step S108 (virtual viewpoint setting step), following the determination in step S104 that the relative speed between the object and the own vehicle> zero, or the determination in step S105 that the object is located in front of the side camera. , A position that includes the object and the front and rear wheels of the own vehicle in the field of view, and sets a virtual viewpoint from the position behind the object and the own vehicle (left and right side cameras 22c, 22d) toward the front of the vehicle, and steps. Proceed to S109.
Here, when the relative speed between the object and the own vehicle is greater than zero, it is considered that the object moves toward the front of the vehicle when viewed from the own vehicle. Therefore, by setting the virtual viewpoint toward the front of the vehicle, it is possible to generate a virtual image that grasps the relative positional relationship between the own vehicle and the object without changing the position of the virtual viewpoint even if the object moves. Become.
When the object is located in front of the vehicle with the left and right side cameras 22c and 22d, the rear surface of the object (the surface facing the rear of the vehicle) and the front background of the object by either of the left and right side cameras 22c and 22d. Is considered to have been photographed. Therefore, by setting the virtual viewpoint toward the front of the vehicle, the relative positional relationship between the own vehicle and the object, including the object and the front and rear wheels of the own vehicle, can be determined by using the captured images of the left and right side cameras 22c and 22d. It is possible to generate a virtual image to be grasped.

In step S109 (image generation step), following the setting of the virtual viewpoint in step S108, the virtual image obtained by converting the viewpoint to the image viewed from the virtual viewpoint set in step S108 is converted into the image of the left and right side cameras 22c and 22d. It is generated using the image of either one and the front camera 22a, and the process proceeds to step S112.
At this time, the generated virtual image includes the object, the front and rear wheels, the side surface, and the vehicle lower region of the own vehicle, and is a virtual image for grasping the relative positional relationship between the own vehicle and the object. Further, a necessary CG image or the like may be superimposed as appropriate according to the virtual image to be generated.

In step S110, following the determination in step S104 that the relative speed between the object and the own vehicle is ≤ zero, the position includes the front and rear wheels of the object and the own vehicle in the field of view, and the object and the own vehicle (left and right sides). The virtual viewpoint is set from the position in front of the vehicle to the rear of the vehicle from the cameras 22c and 22d), and the process proceeds to step S111.
Here, when the relative speed between the object and the own vehicle is zero or less, it is considered that the object moves toward the rear of the vehicle when viewed from the own vehicle. Therefore, by setting the virtual viewpoint toward the rear of the vehicle, it is possible to generate a virtual image that grasps the relative positional relationship between the object and the own vehicle without changing the position of the virtual viewpoint even if the object moves. Become.

In step S111, following the setting of the virtual viewpoint in step S110, the virtual image whose viewpoint is converted to the image viewed from the virtual viewpoint set in step S110 is used with either the left and right side cameras 22c or 22d. It is generated using the image with the rear camera 22b, and the process proceeds to step S112.
At this time, the generated virtual image includes the object, the front and rear wheels, the side surface, and the vehicle lower region of the own vehicle, and is a virtual image for grasping the relative positional relationship between the own vehicle and the object. Further, a necessary CG image or the like may be superimposed as appropriate according to the virtual image to be generated.

In step S112, following the generation of the virtual image in step S107, step S109, and step S111, the virtual image generated in these steps is displayed on the display 26 for display, and the process proceeds to return.
As a result, the display 26 displays a virtual image for grasping the relative positional relationship between the object and the own vehicle, and visually assists the occupant who got on the own vehicle.

In step S113, following the determination in step S102 that there is no object, it is determined whether or not a virtual image from a predetermined virtual viewpoint is being displayed on the display display 26. If YES (displaying a virtual image), the process proceeds to step S114. If NO (virtual image is not displayed), the process returns to step S101.

In step S114, following the determination that the virtual image is being displayed in step S113, it is determined that visual assistance by the virtual image is unnecessary because the object does not exist, and the display of the virtual image on the display display 26 is terminated and the process returns. move on.

Next, the action will be described.
The operation of the first embodiment will be described separately by dividing it into a "visual assistance control processing operation", a "virtual viewpoint setting operation", and a "virtual image generation operation".

[Visual assistance control processing action]
Hereinafter, the visual assist control processing operation will be described based on the flowchart of FIG.
As shown in FIG. 3A, the own vehicle S has the front camera 22a of the in-vehicle camera 22 mounted at the center position of the front end portion, the rear camera 22b mounted at the center position of the rear end portion, and the left side camera 22c mounted on the left side mirror. The right side camera 22d is mounted on the right side mirror.
The shooting ranges A to D of the cameras 22a to 22d are areas surrounded by broken lines shown in the figure.

Here, consider a case where the pedestrian H exists at the first position 41, which is a position diagonally forward to the left of the own vehicle S, while the own vehicle S is traveling. At this time, in the flowchart shown in FIG. 2, the process proceeds from step S101 to step S102, and the pedestrian H existing in the shooting ranges A and C of the front camera 22a and the left side camera 22c is set as the “object”.

Next, the process proceeds to step S103, and it is determined whether or not the pedestrian H, which is the object, can be displayed at any angle in the front-rear direction of the vehicle. Here, the pedestrian H is not photographed in the predetermined region X (the shaded area in FIG. 3A) in the vehicle front-rear direction with the optical axis center position α of the left side camera 22c sandwiched. Therefore, the pedestrian H proceeds to step S105, assuming that it can be displayed only at any angle in the front-rear direction of the vehicle.

After proceeding to step S105, it is determined whether or not the position of the pedestrian H is behind the left side camera 22c. In the case of FIG. 3A, since the pedestrian H is located in front of the vehicle from the optical axis center position α of the left side camera 22c, the process proceeds to step S108, and the pedestrian H and the own vehicle S (at least the left side camera) The virtual viewpoint K is set toward the front of the vehicle from the position 41a behind the vehicle from 22c). At this time, the front and rear wheels 11 and 12 of the pedestrian H and the own vehicle S are included in the field of view of the virtual viewpoint K.

After setting the virtual viewpoint K, the process proceeds from step S109 to step S112 to generate a virtual image whose viewpoint is converted to an image viewed from the virtual viewpoint K set at this position 41a, and then display the generated virtual image. It is displayed on the display 26.

At this time, as shown in FIG. 3B, the generated virtual image includes the pedestrian H, which is an object, the front and rear wheels 11, 12, the side surface 13, and the vehicle lower region of the own vehicle S, and the own vehicle S. It is a virtual image that grasps the relative positional relationship between the pedestrian H (object) and the pedestrian H (object). Further, this virtual image is generated by using the images of the left side camera 22c located behind the pedestrian H and the front camera 22a located in front of the pedestrian H, and the back surface of the pedestrian H is projected. Will be done.
That is, when generating a virtual image from the virtual viewpoint K set at the position 41a, the first captured image obtained by the front camera 22a (first camera) arranged at the position immediately before the pedestrian H and the pedestrian Using the second captured image obtained by the left side camera 22c (second camera) arranged at the position immediately after H, an image facing forward from the rear of the own vehicle S is generated.

Of the virtual images shown in FIG. 3B, for example, a part of the side surface 13 of the own vehicle S (area indicated by dots), a part of the road surface R (area indicated by dots), etc. may not be included in the captured image. , Even if it is included, a CG image generated in advance may be superimposed on the region that is greatly distorted by the viewpoint conversion.

Further, as shown in FIG. 4A, when the pedestrian H is present at the second position 42, which is the left side position of the own vehicle S, while the own vehicle S is traveling, the pedestrian proceeds to step S101 → step S102. After setting H as the "object", the process proceeds to step S103, and it is determined whether or not the pedestrian H, which is the object, can be displayed at any angle in the front-rear direction of the vehicle.

Here, since the pedestrian H is not photographed within the predetermined region X in the vehicle front-rear direction with the optical axis center position α of the left side camera 22c in between, the pedestrian H can only be photographed at any angle in the vehicle front-rear direction. Assuming that the display cannot be performed, the process proceeds to step S105. After proceeding to step S105, it is determined whether or not the position of the pedestrian H is behind the left side camera 22c. In the case of FIG. 4A, since the pedestrian H is located slightly behind the vehicle from the optical axis center position α of the left side camera 22c, the process proceeds to step S106. As a result, the virtual viewpoint K is set from the position 42a in front of the pedestrian H and the own vehicle S (left side camera 22c) toward the rear of the vehicle. At this time, the front and rear wheels 11 and 12 of the pedestrian H and the own vehicle S are included in the field of view of the virtual viewpoint K.

After setting the virtual viewpoint K, the process proceeds from step S107 to step S112 to generate a virtual image whose viewpoint is converted to an image viewed from the virtual viewpoint K set at this position 42a, and then display the generated virtual image. It is displayed on the display 26.

At this time, as shown in FIG. 4B, the generated virtual image includes the pedestrian H, which is an object, the front and rear wheels 11, 12, the side surface 13, and the vehicle lower region of the own vehicle S, and the own vehicle S. It is a virtual image that grasps the relative positional relationship between the vehicle and the pedestrian H (object). Further, this virtual image is generated by using the images of the left side camera 22c located in front of the pedestrian H and the rear camera 22b located behind the pedestrian H, and the front of the pedestrian H is projected. To.
That is, when generating a virtual image from the virtual viewpoint K set at the position 42a, the first captured image obtained by the left side camera 22c (first camera) arranged at the position immediately before the pedestrian H and the walking An image of the own vehicle S facing from the front to the rear is generated by using the second captured image obtained by the rear camera 22b (second camera) arranged at the position immediately after the person H.

Similar to the virtual image shown in FIG. 3B, in the virtual image shown in FIG. 4B, the pre-generated CG image is superimposed on the region and the like (the region indicated by the dot in FIG. 4B) not included in the captured image. You may.

Further, as shown in FIG. 5A, when the pedestrian H is present at the third position 43, which is a position diagonally to the left of the own vehicle S while the own vehicle S is traveling, the pedestrian H proceeds to step S101 → step S102 to walk. After setting the person H as the "object", the process proceeds to step S103, and it is determined whether or not the pedestrian H, which is the object, can be displayed at any angle in the front-rear direction of the vehicle.

At this time, since the pedestrian H is not photographed within the predetermined region X in the vehicle front-rear direction with the optical axis center position α of the left side camera 22c in between, the pedestrian H can only be photographed at any angle in the vehicle front-rear direction. Assuming that the display cannot be performed, the process proceeds to step S105. After proceeding to step S105, it is determined whether or not the position of the pedestrian H is behind the left side camera 22c. In the case of FIG. 5A, since the pedestrian H is located behind the vehicle from the optical axis center position α of the left side camera 22c, the process proceeds to step S106. As a result, the virtual viewpoint K is set from the position 43a in front of the pedestrian H and the own vehicle S (left side camera 22c) toward the rear of the vehicle. At this time, the front and rear wheels 11 and 12 of the pedestrian H and the own vehicle S are included in the field of view of the virtual viewpoint K.

After setting the virtual viewpoint K, the process proceeds from step S107 to step S112 to generate a virtual image whose viewpoint is converted to an image viewed from the virtual viewpoint K set at this position 43a, and then display the generated virtual image. It is displayed on the display 26.

At this time, as shown in FIG. 5B, the generated virtual image includes the pedestrian H, which is an object, the front and rear wheels 11, 12, the side surface 13, and the vehicle lower region of the own vehicle S, and the own vehicle S. It is a virtual image that grasps the relative positional relationship between the vehicle and the pedestrian H (object). Further, this virtual image is generated by using the images of the left side camera 22c located in front of the pedestrian H and the rear camera 22b located behind the pedestrian H, and the front of the pedestrian H is projected. To.
That is, when generating a virtual image from the virtual viewpoint K set at the position 43a, the first captured image obtained by the left side camera 22c (first camera) arranged at the position immediately before the pedestrian H and the walking An image facing from the front to the rear of the own vehicle S is generated by using the second captured image obtained by the rear camera 22b (second camera) arranged at the position immediately after the person H.

Similar to the virtual images shown in FIGS. 3B and 4B, of the virtual images shown in FIG. 5B, the areas not included in the captured image (areas indicated by dots in FIG. 5B) are CGs generated in advance. Images may be superimposed.

As described above, in the display method and the display device D of the first embodiment, when an object (pedestrian H in FIGS. 3A to 5B) required for visual assistance is detected around the own vehicle S, the target for the own vehicle S is detected. Identify the position of an object (pedestrian H). Then, the virtual viewpoint K is set at a position where the relative positional relationship between the own vehicle S and the object (pedestrian H) is grasped according to the position of the specified object (pedestrian H).
That is, the set position of the virtual viewpoint K changes according to the position of the object (pedestrian H) with respect to the own vehicle S, and in order to grasp the relative positional relationship between the own vehicle S and the object (pedestrian H). It is set to the optimum position each time.

As a result, as shown in the virtual images shown in FIGS. 3B, 4B, and 5B, it is possible to generate a virtual image in which the relative positional relationship between the own vehicle S and the object (pedestrian H) can be easily grasped. Then, in the image displayed on the display 26 for displaying to the occupant, the relative positional relationship between the own vehicle S and the object (pedestrian H) can be easily grasped. It is possible to effectively provide visual support for the occupants who have boarded the own vehicle S.

Further, since the set position of the virtual viewpoint K is changed according to the position of the object (pedestrian H) with respect to the own vehicle S, the imaging target can be captured even if there is only one camera for lateral photography of the own vehicle. It is possible to visually assist the occupant without providing a defective image such as distortion, for example, a part of the object (pedestrian H) is cut off, or a part of the imaged object is not drawn.

Further, the virtual image generated in the first embodiment is a so-called "side-down view" including the pedestrian H, which is an object, and the front and rear wheels 11, 12, the side surface 13, and the vehicle lower region of the own vehicle S. .. Therefore, even in a relatively narrow road environment without a center line, the degree of approach to pedestrians, bicycles, curbs (road boundaries), oncoming vehicles, etc. existing in the immediate vicinity of the own vehicle S should be displayed in an easy-to-understand manner. Can be done. Then, by setting the virtual viewpoint K according to the position of the object (pedestrian H) with respect to the own vehicle S, the occupant moves the vehicle in parallel with the curb while paying attention to the pedestrian or the bicycle. It is possible to switch and display an appropriate side-down view that makes it easy to grasp the surrounding situation of.

Further, in the first embodiment, an in-vehicle camera 22 (front camera 22a, rear camera 22b, left side camera 22c, right side camera 22d) is mounted as an object detection sensor for detecting the position of the object (pedestrian H). ing. Then, when specifying the position of the object (pedestrian H) with respect to the own vehicle S, the optical axis center position α of the left side camera 22c, which is the object detection sensor, is used as a reference, and the optical axis center position of the left side camera 22c. The position of the object (pedestrian H) with respect to α is specified.
That is, the position of the virtual viewpoint K is adjusted according to the relative positional relationship between the left side camera 22c and the object (pedestrian H).

As a result, as shown in FIG. 4A, the position of the object (pedestrian H) can be easily specified even for the object (pedestrian H) located on the side of the own vehicle S, which is virtual. The viewpoint K can be set quickly, and a virtual image can be generated in a short time. Therefore, even when the speed of the own vehicle S is high, the virtual image can be displayed at an appropriate timing, and the visual support effect can be improved.

[Virtual viewpoint setting action]
In the scenes shown in FIGS. 3A to 5B, the pedestrian H, which is an object, is not photographed in the predetermined region X in the front-rear direction of the vehicle with the optical axis center position α of the left side camera 22c in between, and the pedestrian H is not photographed. Is an example that can be displayed only at any angle in the front-rear direction of the vehicle.

On the other hand, as shown in FIG. 6A, the pedestrian H exists at the fourth position 44 in the predetermined area X in the vehicle front-rear direction with the optical axis center position α of the left side camera 22c in between, and the pedestrian H exists in the predetermined area X. Consider a scene in which all parts of pedestrian H are photographed.
At this time, the pedestrian H is located in front of the left side camera 22c, and can be displayed at any angle in the front-rear direction of the vehicle.
Therefore, in the flowchart shown in FIG. 2, the process proceeds from step S101 → step S102 → step S103 → step S104, the relative speed between the pedestrian H and the own vehicle S is detected, and whether or not the detected relative speed is greater than zero. To judge.

Here, for example, when the own vehicle S is running and the pedestrian H is stopped, the moving speed of the own vehicle S is faster than that of the pedestrian H, and the relative speed is zero or less. Therefore, the process proceeds from step S104 to step S110, and the virtual viewpoint K is set from the position 44a in front of the pedestrian H and the own vehicle S (left side camera 22c) toward the rear of the vehicle. At this time, the front and rear wheels 11 and 12 of the pedestrian H and the own vehicle S are included in the field of view of the virtual viewpoint K.

After setting the virtual viewpoint K, the process proceeds from step S111 to step S112 to generate a virtual image whose viewpoint is converted to an image viewed from the virtual viewpoint K set at this position 44a, and then display the generated virtual image. It is displayed on the display 26.

At this time, as shown in FIG. 6B, the generated virtual image includes the pedestrian H, which is an object, the front and rear wheels 11, 12, the side surface 13, and the vehicle lower region of the own vehicle S, and the own vehicle S. It is a virtual image that grasps the relative positional relationship between the vehicle and the pedestrian H (object). Further, this virtual image is generated by using the images of the left side camera 22c located on the side of the pedestrian H and the rear camera 22b located behind the pedestrian H, and the front of the pedestrian H is projected. To.

After that, with the passage of time, the relative positional relationship between the own vehicle S and the pedestrian H changes. Specifically, as shown in FIG. 6C, since the speed of the own vehicle S is high, the pedestrian H relatively moves to the fifth position 45, which is a position diagonally rearward to the left of the own vehicle S.

Here, the virtual image for visual assistance is an image from the virtual viewpoint K set in a state of facing the rear of the vehicle from the position 44a, and when the own vehicle S is viewed from the front to the rear of the vehicle. It is an image of. Therefore, as shown in FIG. 6D, the pedestrian H is continuously included in the virtual image even when the pedestrian H relatively moves to the diagonally left rear position of the own vehicle S with the passage of time. Can be done.

On the other hand, as shown in FIG. 7A, the pedestrian H exists at the fourth position 44 in the predetermined region X in the vehicle front-rear direction with the optical axis center position α of the left side camera 22c sandwiched, and walks in the predetermined region X. Even when all parts of the person H are photographed, for example, when the own vehicle S is stopped and the pedestrian H is walking toward the front of the vehicle, the pedestrian H and the own vehicle S Relative speed is greater than zero.

At this time, the process proceeds from step S101 → step S102 → step S103 → step S104 → step S108, and a virtual viewpoint is directed toward the front of the vehicle from the position 44b behind the vehicle H and the own vehicle S (left side camera 22c). Set K. At this time, the front and rear wheels of the pedestrian H and the own vehicle S are included in the field of view of the virtual viewpoint K.
After that, the process proceeds from step S111 to step S112 to generate a virtual image whose viewpoint is converted to an image when viewed from the virtual viewpoint K set at this position 44b, and then displays the generated virtual image on the display 26 for display. To do.

As a result, as shown in FIG. 7B, the virtual image whose viewpoint is converted into the image when viewed from the virtual viewpoint K set at the position 44b includes the pedestrian H, which is an object, and the front and rear wheels 11 of the own vehicle S. It is a virtual image that includes 12, the side surface 13 and the vehicle lower region, and grasps the relative positional relationship between the own vehicle S and the pedestrian H (object). Further, this virtual image is generated by using the images of the left side camera 22c located on the side of the pedestrian H and the front camera 22a located in front of the pedestrian H, and the back surface of the pedestrian H is projected. Will be done.

Then, with the passage of time thereafter, the relative positional relationship between the own vehicle S and the pedestrian H changes. In this case, since the speed of the pedestrian H is high, the pedestrian H is as shown in FIG. 7C. Moves relatively to the sixth position 46, which is a position diagonally forward to the left of the own vehicle S.

Here, the virtual image for visual assistance is an image from the virtual viewpoint K set in a state of facing the front of the vehicle from the position 44b, and when the own vehicle S is viewed from the rear of the vehicle to the front. It is an image of. Therefore, as shown in FIG. 7D, the pedestrian H is continuously included in the virtual image even when the pedestrian H relatively moves to the diagonally forward left position of the own vehicle S with the passage of time. Can be done.

In this way, when the object (pedestrian H) can be displayed at any angle in the front-rear direction of the vehicle, it is virtual according to the magnitude of the relative speed between the own vehicle S and the object (pedestrian H). Set the position of the viewpoint K. As a result, even if the relative positional relationship between the own vehicle S and the object (pedestrian H) in the vehicle front-rear direction changes with the passage of time, the object (pedestrian H) does not change the position of the virtual viewpoint K. ) Can be continued to be included in the virtual image. As a result, the angle of the virtual image can be maintained, and the virtual image for grasping the relative positional relationship between the own vehicle S and the object (pedestrian H) does not change significantly, and this relative positional relationship can be easily grasped. It can be possible.

On the other hand, as shown in FIG. 3A, when the object (pedestrian H) is in front of the vehicle from the left side camera 22c, the virtual viewpoint K is viewed from the rear position of the object (pedestrian H). Set to face the front of the vehicle.
As a result, even when the object (pedestrian H) is located in front of the left side camera 22c, a virtual image generated with reference to the virtual viewpoint K facing the front of the vehicle (see FIG. 3B). ), The relative positional relationship between the own vehicle S and the object (pedestrian H) can be grasped.

Further, as shown in FIGS. 4A and 5A, when the object (pedestrian H) is behind the vehicle with respect to the left side camera 22c, the virtual viewpoint K is set to the position in front of the object (pedestrian H). Set so that it faces the rear of the vehicle.
As a result, even when the object (pedestrian H) is located behind the left side camera 22c, the occupant can use the virtual image generated with reference to the virtual viewpoint K facing the rear of the vehicle. (See FIGS. 4B and 5B), the relative positional relationship between the own vehicle S and the object (pedestrian H) can be grasped.

[Virtual image generation action]
In the first embodiment, when the virtual image from the virtual viewpoint K set according to the position of the object (pedestrian H) is generated, the virtual viewpoint K is set to the position facing the front of the vehicle as described above ( In the case of (see FIGS. 3A and 7A), the images generated by the left side camera 22c and the front camera 22a are generated, and the back surface of the pedestrian H is projected in the virtual image as shown in FIGS. 3B and 7B. ..
On the other hand, when the virtual viewpoint K is set to a position facing the rear of the vehicle (see FIGS. 4A, 5A, and 6A), the images generated by the left side camera 22c and the rear camera 22b are generated, and FIGS. As shown in 5B and FIG. 6B, the back surface of the pedestrian H is projected in the virtual image.

In this way, a camera provided at an appropriate position is selected from the in-vehicle cameras 22 according to the orientation in which the virtual viewpoint K is set, and a virtual image is generated using the image captured by the camera. As a result, an appropriate virtual image with few distortions and missing parts can be generated, and the relative positional relationship between the own vehicle S and the object (pedestrian H) can be grasped in the virtual image.

Next, the effect will be described.
In the display method and the display device of the display device of the first embodiment, the effects listed below can be obtained.

(1) An object (pedestrian H) around the own vehicle S is detected, an image from a virtual viewpoint K including the object (pedestrian H) is generated, and the image is used as visual support information for the occupant. In the display method of the display device to be displayed,
An object position specifying step (step S105) for specifying the positions 41 and 42 of the object (pedestrian H) with respect to the own vehicle S based on predetermined information.
The virtual viewpoint K is set at positions 41a and 42a for grasping the relative positional relationship between the own vehicle S and the object (pedestrian H) according to the positions 41 and 42 of the object (pedestrian H). Virtual viewpoint setting step (step S106, step S108) and
An image generation step (step S107, step S109) for generating an image when viewed from the virtual viewpoint K, and
It was configured to have.
As a result, the relative positional relationship between the own vehicle S and the object (pedestrian H) can be easily grasped in the image displayed to the occupant.

(2) An object detection sensor (left side camera 22c) for detecting the position of the object (pedestrian H) is mounted on the own vehicle S.
The object position specifying step (step S105) is configured to specify the positions 41 and 42 of the object (pedestrian H) with respect to the position (optical axis center position α) of the object detection sensor (left side camera 22c). And said.
As a result, in addition to the effect of (1), the position of the object (pedestrian H) can be easily specified, the virtual viewpoint K can be quickly set, and the virtual image can be displayed at an appropriate timing. ..

(3) In the virtual viewpoint setting step (step S106, step S108), the position (first position 41) of the object (pedestrian H) is the position (optical axis) of the object detection sensor (left side camera 22c). When it is in front of the center position α), the virtual viewpoint K is set from the position 41a behind the vehicle to the front of the vehicle from the object (pedestrian H).
As a result, in addition to the effect of (2), even when the object (pedestrian H) is located in front of the object detection sensor (left side camera 22c), in the virtual image, The relative positional relationship between the own vehicle S and the object (pedestrian H) can be easily grasped.

(4) The image generation step (step S107, step S109) is located behind the vehicle (pedestrian H) when generating an image from the virtual viewpoint K set toward the front of the vehicle. The image is generated using the information detected by the object detection sensor (left side camera 22c).
As a result, in addition to the effect of (3), a virtual image can be generated using the information from the object detection sensor (left side camera 22c) provided at an appropriate position according to the direction of the virtual viewpoint K, and the own vehicle can be generated. An appropriate virtual image for grasping the relative positional relationship between S and the object (pedestrian H) can be generated.

(5) In the virtual viewpoint setting step (step S106, step S108), the position (second position 42) of the object (pedestrian H) is the position (optical axis) of the object detection sensor (left side camera 22c). When it is behind the center position α), the virtual viewpoint K is set from the position 42a in front of the vehicle to the rear of the vehicle with respect to the object (pedestrian H).
As a result, in addition to the effect of (2), even when the object (pedestrian H) is located behind the object detection sensor (left side camera 22c), in the virtual image, The relative positional relationship between the own vehicle S and the object (pedestrian H) can be easily grasped.

(6) The image generation step (step S107, step S109) is located in front of the vehicle (pedestrian H) when generating an image from the virtual viewpoint K set toward the rear of the vehicle. The image is generated using the information detected by the object detection sensor (left side camera 22c).
As a result, a virtual image can be generated using the information from the object detection sensor (left side camera 22c) provided at an appropriate position according to the direction of the virtual viewpoint K, and the own vehicle S and the object (pedestrian H) can be generated. ) Can generate an appropriate virtual image for grasping the relative positional relationship.

(7) A display that detects an object (pedestrian H) around the own vehicle S, generates an image from a virtual viewpoint K including the object (pedestrian H), and displays it as visual support information for the occupant. In the display device D provided with the controller (visual support CPU 30),
The display controller (visual support CPU 30) is
Based on the predetermined information, the object position specifying unit 32 for specifying the positions 41 and 42 of the object (pedestrian H) with respect to the own vehicle S, and
Virtual viewpoint setting in which the virtual viewpoint K is set at positions 41a and 42a for grasping the relative positional relationship between the own vehicle S and the object (pedestrian H) according to the position of the object (pedestrian H). Part 33 and
An image generation unit 34 that generates an image from the virtual viewpoint K, and
It was configured to have.
As a result, the relative positional relationship between the own vehicle S and the object (pedestrian H) can be easily grasped in the image displayed to the occupant.

(Modified Example 1)
FIG. 8 is a plan view of the own vehicle S1 to which the display method of the first embodiment and the first modification of the display device are applied.
The own vehicle S1 is mounted as an in-vehicle camera 22 at a front camera 22a mounted at the center position of the front end portion of the own vehicle S1 to photograph the vehicle front region A, and mounted at the center position of the rear end portion of the own vehicle S1 and is mounted on the front of the vehicle. The rear camera 22b for photographing the area B, the first left side camera 22c mounted at the lower left end position of the left side mirror of the own vehicle S1 for photographing the left front area C of the vehicle, and the lower end position of the left rear pillar of the own vehicle S1. The second left side camera 22e mounted on the vehicle to photograph the vehicle left rear region E, and the first right side camera 22d mounted on the vehicle S1 at the lower right end position of the right side mirror of the vehicle S1 to capture the vehicle right front region D. And a second right side camera 22f mounted at the lower end position of the right rear pillar of the own vehicle S1 and photographing the vehicle right rear region F.
That is, in the own vehicle S1, the number of in-vehicle cameras 22 is 6 in the entire vehicle, and the number of cameras for photographing the vehicle side region is 2 per side.

Further, FIG. 9 is a plan view of the own vehicle S2 to which the display method of the first embodiment and the second modification of the display device are applied.
The own vehicle S2 is mounted as an in-vehicle camera 22 at a front camera 22a mounted at the center position of the front end portion of the own vehicle S2 to photograph the vehicle front region A, and mounted at the center position of the rear end portion of the own vehicle S2 and is mounted in front of the vehicle. The rear camera 22b for photographing the area B, the first left side camera 22c mounted on the lower left end position of the left side mirror of the own vehicle S2 and photographing the left front area C of the vehicle, and the lower end position of the left rear pillar of the own vehicle S2. The second left side camera 22e mounted on the vehicle to photograph the vehicle left rear region E, the third left side camera 22g mounted on the left end position of the front bumper of the own vehicle S2 to photograph the vehicle left front region G, and the own vehicle. The first right side camera 22d mounted on the lower right end position of the right side mirror of S2 to photograph the vehicle right front region D, and the vehicle right rear region F mounted on the lower end position of the right rear pillar of the own vehicle S2. It has a second right side camera 22f, and a third right side camera 22j mounted at the right end position of the front bumper of the own vehicle S2 to photograph the vehicle right front region J.
That is, in the own vehicle S2, the number of in-vehicle cameras 22 is eight in the entire vehicle, and the number of cameras for photographing the vehicle side region is three per side.

In a vehicle equipped with a plurality of cameras for photographing the vehicle side area, such as the own vehicle S1 and the own vehicle S2, the vehicle side area is divided according to the number of side cameras and photographed by the left and right side cameras. be able to. Therefore, it is less necessary to set the shooting range of the side cameras to a wider angle (fisheye) than in the case of the first embodiment, and the shooting range of each of the left and right side cameras can be narrowed.
As a result, it is possible to display a wide range of image information in the horizontal direction while suppressing discomfort such as distortion. Further, it is possible to set the virtual viewpoint at a position further behind the position of the virtual viewpoint set when there is one left and right side camera.

Further, as shown in FIGS. 8 and 9, by increasing the number of side cameras, it is possible to expand the predetermined area X in which the object can be displayed regardless of the angle from the front-rear direction of the vehicle. it can.
That is, in the case shown in FIG. 8, the region surrounded by the optical axis center positions α of the first left and right side cameras 22c and 22d, the optical axis center positions β of the second left and right side cameras 22e and 22f, and the front and rear predetermined regions thereof. Is a predetermined area X. Further, in the case shown in FIG. 9, a region surrounded by the optical axis center position γ of the third left and right side cameras 22g, 22j, the optical axis center position β of the second left and right side cameras 22e, 22f, and a predetermined region before and after the region. Is a predetermined area X.

As a result, in particular, as shown in FIG. 9, when the first to third left and right side cameras 22c to 22j for photographing the vehicle side region are provided, the front and rear sides of the own vehicle S2 can be almost the entire side region. It is possible even with the image angle of.
Therefore, the degree of freedom in setting the virtual image is improved, and it is effective in a scene where the vehicle is parked parallel to the curb while the vehicle is brought close to the road boundary such as the curb, and the visual support effect for the occupants is effective. Can be improved.

That is, for example, as shown in FIG. 10, when the pedestrian H (object) exists at the seventh position 47 on the side of the own vehicle S2, first, a plurality of side cameras (first to third left side cameras). The position of the pedestrian H is specified based on the captured image taken by 22c, 22e, 22g), and the image angle is set according to the relative speed with the own vehicle S2.
That is, if the relative speed between the pedestrian H and the own vehicle S2 is greater than zero, the virtual viewpoint is set at the position 47a that faces the front of the vehicle and includes the front and rear wheels of the pedestrian H and the own vehicle S2 in the field of view. If the relative speed is zero or less, the virtual viewpoint is set at a position 47b that faces the rear of the vehicle and includes the front and rear wheels of the pedestrian H and the own vehicle S2 in the field of view.

Then, when generating virtual images from the positions 47a and 47b, the first captured image obtained by the third left side camera 22g (first camera) arranged immediately before the pedestrian H and the pedestrian H Using the second captured image obtained by the first left side camera 22c (second camera) arranged at the position immediately after, a virtual image including the side surface of the own vehicle S2 and the vehicle lower region is generated.

As a result, the side surface 13 of the own vehicle S and the vehicle lower region are mainly based on the captured images of the third left side camera 22g located immediately before the pedestrian H and the first left side camera 22c located immediately after the pedestrian H. It is possible to generate an image (side-down view) including the above, and at the same time, determine the angle (image direction) from the relative speed difference between the captured images of the third and first left side cameras 22g and 22c. Therefore, it is difficult for the occupant to limit the angle (image direction) of the virtual image that makes it easy for the occupant to grasp the relative positional relationship between the own vehicle S2 and the pedestrian H.

(Example 2)
The second embodiment is an example in which a virtual viewpoint is set according to the position of an object existing around the own vehicle in the left-right direction with respect to the center position in the width direction of the own vehicle.

FIG. 11 is a flowchart showing the flow of the visual assistance control processing executed by the visual assistance CPU of the first embodiment. Hereinafter, each step of FIG. 11 showing the visual assistance control processing configuration will be described.

In step S201, the steering angle sensor 23, the vehicle speed sensor 24, and the range position sensor 25 acquire information on the driving state of the vehicle, grasp the driving state of the own vehicle, and proceed to step S202.

In step S202, following the grasp of the driving state in step S201, it is determined whether or not the object exists within the photographing range photographed by the in-vehicle camera 22. If YES (there is an object), the process proceeds to step S203. If NO (no object), the process proceeds to step S211.
Here, the presence or absence of the object is determined by analyzing the image captured by the in-vehicle camera 22 and considering the sensor information from the object recognition sensor 21.

In step S203, following the determination in step S202 that there is an object, it is determined whether or not the object can be displayed at any angle in the left-right direction of the vehicle. If YES (displayable from either the left or right side of the vehicle), the process proceeds to step S204. If NO (displayable only from one of the left and right sides of the vehicle), the process proceeds to step S205.
Here, whether or not the object can be displayed at any angle in the left-right direction of the vehicle is determined based on the shooting state by the vehicle-mounted camera 22.

In step S204, following the determination that the display can be performed from either the left or right side of the vehicle in step S203, it is determined whether or not the object whose existence has been confirmed in step S202 is moving toward the right side of the own vehicle. If YES (moving to the right), the process proceeds to step S207. If NO (no movement to the right), the process proceeds to step S208.
Here, the movement of the object is detected based on the movement (movement direction) of the object within the predetermined time counted by the timer 32a. In addition, "no movement to the right" includes the case where the object is moving to the left of the own vehicle and the case where the object is stopped on the spot.

In step S205 (object position identification step), following the determination in step S203 that the display can be performed only from the left or right side of the vehicle, the object whose existence has been confirmed in step S202 can be photographed by the left side camera 22c. Judge whether or not. If YES (capable of shooting with the left side camera), the process proceeds to step S206. If NO (shooting is not possible with the left side camera), the process proceeds to step S207.
Here, "capable of shooting with the left side camera 22c" means that the position of the object is located on the left side of the center position in the width direction of the own vehicle. On the other hand, "cannot be photographed by the left side camera 22c" means that the image is photographed by the right side camera 22d, and the position of the object is located on the right side of the center position in the width direction of the own vehicle. means.

In step S206 (virtual viewpoint setting step), following the determination that the object in step S205 can be photographed by the left side camera 22c, the position includes the object and a part of the own vehicle in the field of view, and the object and the object and a part of the own vehicle are included in the field of view. A virtual viewpoint is set from the position on the right side in front of the vehicle to the left side behind the vehicle (left side of the vehicle) from the center position in the width direction of the own vehicle, and the process proceeds to step S209.
Here, when the object is photographed by the left side camera 22c, it is considered that the right side surface of the object and the left background of the vehicle are photographed by the left side camera 22c. Therefore, by setting the virtual viewpoint toward the left side of the vehicle, it is possible to generate a virtual image for grasping the relative positional relationship between the own vehicle and the object by using the captured image of the left side camera 22c.
That is, in this step S206, the virtual viewpoint is set at a position where the relative positional relationship between the own vehicle and the object is grasped according to the position of the object.

In step S207 (virtual field of view setting step), it is determined that the object in step S204 moves toward the right side of the own vehicle, or that the object in step S205 cannot be photographed by the left side camera. Next, it is a position that includes the object and a part of the own vehicle in the field of view, and is a virtual viewpoint from the position on the left front side of the vehicle to the right side behind the vehicle (right side of the vehicle) from the center position in the width direction of the object and the own vehicle. Is set, and the process proceeds to step S209.
Here, when the object moves toward the right side of the own vehicle, by setting the virtual viewpoint toward the right side of the vehicle, the own vehicle does not change the position of the virtual viewpoint even if the object moves. It is possible to generate a virtual image that grasps the relative positional relationship between the object and the object.
Further, when the object is not photographed by the left side camera 22c, it is considered that the left side surface of the object and the right background of the vehicle are photographed by the right side camera 22d. Therefore, by setting the virtual viewpoint toward the right side of the vehicle, it is possible to generate a virtual image for grasping the relative positional relationship between the own vehicle and the object by using the captured image of the right side camera 22d.
That is, in this step S207, the virtual viewpoint is set at a position where the relative positional relationship between the own vehicle and the object is grasped according to the position of the object.

In step S208, following the determination that the object moves to the left of the own vehicle in step S204, the position includes the object and a part of the own vehicle in the field of view, and the width of the object and the own vehicle. A virtual viewpoint is set from the position on the right side in front of the vehicle to the left side behind the vehicle (left side of the vehicle) from the center position, and the process proceeds to step S209.
Here, when the object moves toward the left side of the own vehicle, by setting the virtual viewpoint toward the left side of the vehicle, the own vehicle does not change the position of the virtual viewpoint even if the object moves. It is possible to generate a virtual image that grasps the relative positional relationship between the object and the object.

In step S209, following the setting of the virtual viewpoint in steps S206, S207, and S208, the left and right side cameras 22c display the virtual image whose viewpoint is converted to the image viewed from the virtual viewpoint set in any step. , 22d and the image of the front camera 22a or the rear camera 22b are used to generate an image, and the process proceeds to step S210.
At this time, the generated virtual image includes the object and a part of the own vehicle (for example, the front wheel, the driver's seat, the side surface of the vehicle, and the area below the vehicle), and grasps the relative positional relationship between the own vehicle and the object. Let it be a virtual image. Further, a necessary CG image or the like may be superimposed as appropriate according to the virtual image to be generated.

In step S210, following the generation of the virtual image in step S209, the virtual image generated in step S209 is displayed on the display 26 for display, and the process proceeds to return.
As a result, the display 26 displays a virtual image for grasping the relative positional relationship between the object and the own vehicle, and visually assists the occupant who got on the own vehicle.

In step S211, following the determination in step S202 that there is no object, it is determined whether or not a virtual image from a predetermined virtual viewpoint is being displayed on the display display 26. If YES (displaying a virtual image), the process proceeds to step S212. If NO (virtual image is not displayed), the process returns to step S201.

In step S212, following the determination that the virtual image is being displayed in step S211, it is determined that visual assistance by the virtual image is unnecessary because there is no object, and the display of the virtual image on the display display 26 is terminated and the process returns. move on.

Next, the action will be described.
The operation of the second embodiment will be described by dividing it into a "visual assist control processing operation" and a "virtual viewpoint setting operation".

[Visual assistance control processing action]
Hereinafter, the visual assist control processing operation will be described based on the flowchart of FIG.
As shown in FIG. 12A, the own vehicle S has the front camera 22a of the in-vehicle camera 22 mounted at the center position of the front end portion, the left side camera 22c mounted on the left side mirror, and the right side camera 22d mounted on the right side mirror. Has been done. Note that the rear camera 22b is not shown here because an example in which the object exists in front of the vehicle will be described.
The shooting ranges A, C, and D of the cameras 22a, 22c, and 22d are areas surrounded by broken lines shown in the figure.

Here, consider a case where the pedestrian H exists at the eighth position 48, which is a position diagonally forward to the left of the own vehicle S. At this time, in the flowchart shown in FIG. 11, the process proceeds from step S201 to step S202, and the pedestrian H existing in the shooting ranges A and C of the front camera 22a and the left side camera 22c is set as the “object”.

Next, the process proceeds to step S203, and it is determined whether or not the pedestrian H, which is the object, can be displayed at any angle in the left-right direction of the vehicle. Here, the pedestrian H is not photographed in the predetermined region Y (the shaded area in FIG. 12A) in the left-right direction of the vehicle with the optical axis center position α'of the front camera 22a in between. Therefore, the pedestrian H proceeds to step S205, assuming that it can be displayed only at any angle in the left-right direction of the vehicle.

After proceeding to step S205, it is determined whether or not the pedestrian H can be photographed by the left side camera 22c, that is, whether or not the position is on the left side of the center position in the width direction of the own vehicle S. In the case of FIG. 12A, since the pedestrian H is within the shooting range C of the left side camera 22c, it is possible to shoot with the left side camera 22c. That is, assuming that the pedestrian H exists on the left side of the own vehicle S, the process proceeds to step S206, from the position 48a on the front right side of the vehicle front and the left side behind the vehicle (left side of the vehicle) with respect to the center position in the width direction of the pedestrian H and the own vehicle S. ), The virtual viewpoint K is set. At this time, the pedestrian H and a part of the own vehicle S are included in the field of view of the virtual viewpoint K.

After setting the virtual viewpoint K, the process proceeds from step S209 to step S210 to generate a virtual image whose viewpoint is converted to an image viewed from the virtual viewpoint K set at this position 48a, and then display the generated virtual image. It is displayed on the display 26.

At this time, as shown in FIG. 12B, the generated virtual image includes a part of the pedestrian H and the own vehicle S, which are objects, and is relative to the own vehicle S and the pedestrian H (object). It is a virtual image that grasps the positional relationship. Further, this virtual image is generated by using the images of the right side camera 22d located on the right side of the pedestrian H and the front camera 22a located in the front part of the own vehicle S, and the right side surface of the pedestrian H is generated. It is projected.

Of the virtual images shown in FIG. 12B, for example, a part of the side surface 13 of the own vehicle S, a part of the road surface R, etc. are not included in the captured image, or even if they are included, they can be greatly changed by changing the viewpoint. A pre-generated CG image may be superimposed on the distorted region.

Further, as shown in FIG. 13A, when the pedestrian H is present at the ninth position 49, which is a position diagonally forward to the right of the own vehicle S, the process proceeds from step S201 to step S202 to make the pedestrian H an “object”. Then, the process proceeds to step S203, and it is determined whether or not the pedestrian H, which is the object, can be displayed at any angle in the left-right direction of the vehicle.

Here, since the pedestrian H is not photographed in the predetermined region Y in the vehicle front-rear direction with the optical axis center position α'of the front camera 22a in between, the pedestrian H can only be photographed at any angle in the vehicle left-right direction. Assuming that the display cannot be performed, the process proceeds to step S205. After proceeding to step S205, it is determined whether or not the pedestrian H can be photographed by the left side camera 22c, that is, whether or not the position is on the left side of the center position in the width direction of the own vehicle S. In the case of FIG. 13A, since the pedestrian H is within the shooting range D of the right side camera 22d, the left side camera 22c makes it impossible to shoot and proceeds to step S207. As a result, assuming that the pedestrian H exists on the right side of the own vehicle S, the pedestrian H and the own vehicle S are directed from the position 49a on the left side in front of the vehicle to the right side behind the vehicle (right side of the vehicle). Set the virtual viewpoint K. At this time, the pedestrian H and a part of the own vehicle S are included in the field of view of the virtual viewpoint K.

After setting the virtual viewpoint K, the process proceeds from step S209 to step S210 to generate a virtual image whose viewpoint is converted to an image viewed from the virtual viewpoint K set at this position 49a, and then display the generated virtual image. It is displayed on the display 26.

At this time, as shown in FIG. 13B, the generated virtual image includes the pedestrian H, which is an object, and a part of the own vehicle S, and the own vehicle S and the pedestrian H (object). It is a virtual image that grasps the relative positional relationship. Further, this virtual image is generated by using the images of the right side camera 22d located on the left side of the pedestrian H and the front camera 22a located in the front part of the own vehicle S, and the left side surface of the pedestrian H is generated. It is projected.

Similar to the virtual image shown in FIG. 12B, in the virtual image shown in FIG. 13B, the area not included in the captured image, the area where the distortion becomes large, and the like can be superposed with the CG image generated in advance. Good.

In this way, even with the display method and display device D of the second embodiment, if an object (pedestrian H in FIGS. 12A to 13B) required for visual assistance is detected around the own vehicle S, the own vehicle The position of the object (pedestrian H) with respect to S is specified. Then, the virtual viewpoint K is set at a position where the relative positional relationship between the own vehicle S and the object (pedestrian H) is grasped according to the position of the specified object (pedestrian H).
That is, the set position of the virtual viewpoint K changes according to the position of the object (pedestrian H) with respect to the own vehicle S, and in order to grasp the relative positional relationship between the own vehicle S and the object (pedestrian H). It is set to the optimum position each time.

As a result, as shown in the virtual images shown in FIGS. 12B and 13B, it is possible to generate a virtual image in which the relative positional relationship between the own vehicle S and the object (pedestrian H) can be easily grasped. Then, in the image displayed on the display 26 for displaying to the occupant, the relative positional relationship between the own vehicle S and the object (pedestrian H) can be easily grasped. It is possible to effectively provide visual support for the occupants who have boarded the own vehicle S.

Further, in the second embodiment, when the object (pedestrian H) can be photographed by the left side camera 22c and exists on the left side of the vehicle, the virtual viewpoint K is set from the center position in the width direction of the own vehicle S. Also set so that it faces the left side of the vehicle from the position on the right side.
As a result, even when the object (pedestrian H) is located on the left side with respect to the left side camera 22c, a virtual image generated with reference to the virtual viewpoint K facing the left side of the vehicle (FIG. In (see 12B), the relative positional relationship between the own vehicle S and the object (pedestrian H) can be grasped.

Further, as shown in FIG. 13A, when the object (pedestrian H) can be photographed by the right side camera 22d and exists on the right side of the vehicle, the virtual viewpoint K is set to the center position in the width direction of the own vehicle S. Set so that it faces the right side of the vehicle from the position on the left side.
As a result, even when the object (pedestrian H) is located on the right side with respect to the right side camera 22d, a virtual image generated with reference to the virtual viewpoint K facing the right side of the vehicle (FIG. In (see 13B), the relative positional relationship between the own vehicle S and the object (pedestrian H) can be grasped.

[Virtual viewpoint setting action]
In the scenes shown in FIGS. 12A to 13B, the pedestrian H, which is an object, is not photographed in a predetermined region Y in the front-rear direction of the vehicle with the optical axis center position α'of the front camera 22a, and the pedestrian H is not photographed. Is an example that can be displayed only at any angle in the left-right direction of the vehicle.

On the other hand, as shown in FIG. 14A, the pedestrian H exists in the tenth position 50 in the predetermined region Y in the left-right direction of the vehicle with the optical axis center position α'of the front camera 22a in between, and the pedestrian H exists in the predetermined region Y. Consider a scene in which all parts of pedestrian H are photographed.
At this time, the pedestrian H is located substantially in front of the front camera 22a, and can be displayed at any angle in the left-right direction of the vehicle.
Therefore, in the flowchart shown in FIG. 11, the process proceeds from step S201 → step S202 → step S203 → step S204, detects the moving direction of the pedestrian H, and moves toward the right side (right side) of the own vehicle S. Judge whether or not.

Here, for example, when the pedestrian H is moving to the right of the own vehicle S, the process proceeds from step S204 to step S207, and the pedestrian H and the own vehicle S are on the left side in front of the vehicle from the center position in the width direction. The virtual viewpoint K is set from the position 50a toward the rear right side of the vehicle (right side of the vehicle). At this time, the pedestrian H and a part of the own vehicle S are included in the field of view of the virtual viewpoint K.

After setting the virtual viewpoint K, the process proceeds from step S209 to step S210 to generate a virtual image whose viewpoint is converted to an image viewed from the virtual viewpoint K set at this position 50a, and then display the generated virtual image. It is displayed on the display 26.

At this time, as shown in FIG. 14B, the generated virtual image includes the pedestrian H, which is an object, and a part of the own vehicle S, and the own vehicle S and the pedestrian H (object). It is a virtual image that grasps the relative positional relationship.

After that, with the passage of time, the relative positional relationship between the own vehicle S and the pedestrian H changes. Specifically, as shown by the alternate long and short dash line in FIGS. 14A and 14B, the pedestrian H moves to the diagonally right forward position 51a of the own vehicle S.

Here, the virtual image for visual assistance is an image from the virtual viewpoint K set in a state of facing the vehicle right from the position 50a, and the own vehicle S is directed from the front left side of the vehicle to the right side of the vehicle. It is an image when viewed. Therefore, as shown in FIG. 14B, the pedestrian H is continuously included in the virtual image even when the pedestrian H relatively moves to the diagonally right forward position 51a of the own vehicle S with the passage of time. be able to.

On the other hand, the pedestrian H exists at the tenth position 50 in the predetermined region Y in the vehicle front-rear direction with the optical axis center position α'of the front camera 22a, and all the parts of the pedestrian H are in the predetermined region X. Consider the case where the pedestrian H is moving to the left side (left side) of the own vehicle S in the case of being photographed. At this time, the process proceeds from step S201 → step S202 → step S203 → step S204 → step S208, and from the position 50b on the front right side of the vehicle to the center position in the width direction of the pedestrian H and the own vehicle S, the left side behind the vehicle (left side of the vehicle) Set the virtual viewpoint K toward. At this time, the pedestrian H and a part of the own vehicle S are included in the field of view of the virtual viewpoint K.
After that, the process proceeds from step S209 to step S210 to generate a virtual image whose viewpoint is converted into an image when viewed from the virtual viewpoint K set at this position 50b, and then displays the generated virtual image on the display 26 for display. To do.

As a result, the virtual image whose viewpoint is converted into an image when viewed from the virtual viewpoint K set at the position 50b includes a part of the pedestrian H and the own vehicle S, which are objects, as shown in FIG. 14C. This is a virtual image for grasping the relative positional relationship between the own vehicle S and the pedestrian H (object). Further, this virtual image is generated by using the images of the left side camera 22c located on the left side of the pedestrian H and the front camera 22a located on the front side of the own vehicle S, and the left side surface of the pedestrian H is generated. It is projected.

Then, with the passage of time thereafter, the relative positional relationship between the own vehicle S and the pedestrian H changes. In this case, as shown by the two-dot chain line in FIGS. 14A and 14C, the own vehicle S is oblique to the left. The pedestrian H moves to the front position 51b.

Here, the virtual image for visual assistance is an image from the virtual viewpoint K set in a state of facing the vehicle left from the position 50b, and the own vehicle S is directed from the front right side of the vehicle to the left side of the vehicle. It is an image when viewed. Therefore, as shown in FIG. 14C, the pedestrian H is continuously included in the virtual image even when the pedestrian H relatively moves to the diagonally left forward position 51b of the own vehicle S with the passage of time. be able to.

In this way, when the object (pedestrian H) can be displayed at any angle in the left-right direction of the vehicle, the position of the virtual viewpoint K is set on the object (pedestrian H) according to the moving direction. As a result, even if the relative positional relationship between the own vehicle S and the object (pedestrian H) in the vehicle width direction changes with the passage of time, the object (pedestrian H) does not change the position of the virtual viewpoint K. ) Can be continued to be included in the virtual image. As a result, the angle of the virtual image can be maintained, and the virtual image for grasping the relative positional relationship between the own vehicle S and the object (pedestrian H) does not change significantly, and this relative positional relationship can be easily grasped. It can be possible.

Next, the effect will be described.
In the display method of the display device of the second embodiment, the effects listed below can be obtained.

(8) In the virtual viewpoint setting step (step S206, step S207), the position (eighth position 48) of the object (pedestrian H) is higher than the position of the object detection sensor (left side camera 22c). When it exists on the left side of the own vehicle S, the virtual viewpoint K is set from the position 48a on the right side of the own vehicle S toward the left side of the vehicle.
As a result, in addition to the effect of (2), even when the object (pedestrian H) is located on the left side of the object detection sensor (left side camera 22c), in the virtual image. , The relative positional relationship between the own vehicle S and the object (pedestrian H) can be easily grasped.

(9) In the virtual viewpoint setting step (step S206, step S207), the position (9th position 49) of the object (pedestrian H) is higher than the position of the object detection sensor (right side camera 22d). When it exists on the right side of the own vehicle S, the virtual viewpoint K is set from the position 49a on the left side of the own vehicle S toward the right side of the vehicle.
As a result, in addition to the effect of (2), even when the object (pedestrian H) is located on the right side of the object detection sensor (right side camera 22d), in the virtual image. , The relative positional relationship between the own vehicle S and the object (pedestrian H) can be easily grasped.

The display method and display device of the display device of the present invention have been described above based on the first and second embodiments, but the specific configuration is not limited to these examples, and claims for patent. Design changes and additions are permitted as long as the gist of the invention according to each claim of the scope is not deviated.

In the second embodiment, the own vehicle uses the front camera 22a mounted on the vehicle front end of the own vehicle S, the left side camera 22c mounted on the left side mirror, and the right side camera 22d mounted on the right side mirror. An example of detecting an object existing in front of S is shown. However, the present invention is not limited to this, and for example, as shown in FIG. 15, the left front end camera 22m mounted on the left end portion of the front bumper of the own vehicle S3 and the right front end camera 22n mounted on the right end portion of the front bumper are used. An object existing in front of the vehicle S3 may be detected.

In the case shown in FIG. 15, the position of the virtual viewpoint is set based on whether the object is in the shooting range M of the left front end camera 22m or the shooting range N of the right front end camera 22n. Specifically, when the object is within the shooting range M of the left front end camera 22 m, the object is directed from the position 52a on the front right side of the vehicle S3 toward the rear left side (left side of the vehicle) of the vehicle S3. Set the virtual viewpoint K. Further, when an object exists within the shooting range N of the right front end camera 22n, the virtual viewpoint K is directed from the position 52b on the left side in front of the vehicle to the right side (right side of the vehicle) in the vehicle direction with respect to the center position in the width direction of the own vehicle S3. To set.

Further, in the case shown in FIG. 15, the area (shooting range A) shot by the front camera 22a in the second embodiment is shot by two cameras (left front end camera 22m, right front end camera 22n). It will be. Therefore, the viewing angle of each camera (left front end camera 22m, right front end camera 22n) can be made narrower than that of the front camera 22a, and distortion of the imaged object can be reduced.

Further, in Example 2 and FIG. 15, an example in which it is determined whether or not the object exists on either the left or right side of the own vehicle S for the front region of the own vehicle S, and the virtual viewpoint K is set according to the position. However, it is not limited to this. Even when the rear region of the own vehicle S is targeted, the target and the own vehicle S can be specified by specifying the position of the object and setting a virtual viewpoint according to the position of the specified object. It is possible to appropriately generate a virtual image that grasps the relative positional relationship with.

Further, in Example 1, an example of specifying the position of the object with respect to the left and right side cameras 22c and 22d is shown with reference to the optical axis center position α of the left and right side cameras 22c and 22d. However, the position of the object with respect to the own vehicle S may be specified based on, for example, the vehicle center position (center position of the vehicle interior) of the own vehicle S, the driver's seat, or the like.

Further, in Examples 1 and 2, when generating a virtual image, an example in which an image taken by the in-vehicle camera 22 is image-processed and used is shown, but the present invention is not limited to this. For example, a virtual image including an object (object) detected by the object recognition sensor 21 and the own vehicle may be generated as a CG image or the like without using a camera image.
Even in this case, the own vehicle and the object can be changed by appropriately changing the position and direction in which the virtual viewpoint is set based on the position of the object (object) detected by the object recognition sensor 21 with respect to the position of the own vehicle. Generate a virtual image that grasps the relative positional relationship with (object).

Further, the virtual images generated in the first and second embodiments are so-called "side-down views" including the vehicle lower region of the own vehicle S, and are set at a position where the virtual viewpoint K looks down on the own vehicle S. The example is shown, but it is not limited to this. If a virtual image is generated to grasp the relative positional relationship between the own vehicle and the object, the virtual viewpoint is located at a position where the field of view is parallel to the ground according to the position of the object with respect to the own vehicle. May be set, or a virtual viewpoint may be set at a position looking up at the own vehicle.

S Own vehicle D Display device 21 Object recognition sensor (object detection sensor)
22 In-vehicle camera (object detection sensor)
22a Front camera (object detection sensor)
22b Rear camera (object detection sensor)
22c Left side camera (object detection sensor)
22d Right side Gamera (object detection sensor)
23 Steering angle sensor 24 Vehicle speed sensor 25 Range position sensor 26 Display for display 30 Visual support CPU (display controller)
31 Own vehicle state grasping unit 32 Object position specifying unit 33 Virtual viewpoint setting unit 34 Image generation unit 35 Image correction unit 11 Front wheel 12 Rear wheel 13 Side surface

Claims (9)

An object around the own vehicle is detected, an image from a virtual viewpoint including the object is generated from images taken by a plurality of cameras provided in the own vehicle, and the image is used as visual assistance information for the occupant. In the display method of the display device to be displayed as
A step of identifying a predetermined camera that detects the object among the plurality of cameras,
An object position specifying step for specifying the position of the object in the front-rear direction of the own vehicle or the left-right direction of the own vehicle with respect to the predetermined camera .
Virtual viewpoint setting step of the set position and the position of the virtual viewpoint in a position opposite to the predetermined camera the object with respect to setting the virtual viewpoint orientation to include said host vehicle and the object in view When,
An image generation step that generates an image when viewed from the virtual viewpoint, and
A display method of a display device, which comprises. In the display method of the display device according to claim 1 ,
In the virtual viewpoint setting step, when the position of the object is in front of the vehicle from the position of the predetermined camera , the virtual viewpoint is set from a position behind the vehicle to the front of the vehicle from the position of the object. Display method of the characteristic display device. In the display method of the display device according to claim 2 ,
Wherein the image generation step, when generating an image from a virtual viewpoint that is set toward the vehicle front, the image using the captured image obtained al a by the predetermined camera positioned rearward than the object A display method for a display device, characterized in that it is generated. In the display method of the display device according to claim 1 ,
In the virtual viewpoint setting step, when the position of the object is behind the vehicle from the position of the predetermined camera , the virtual viewpoint is set from the position in front of the vehicle to the rear of the vehicle from the position of the object. Display method of the characteristic display device. In the display method of the display device according to claim 4 ,
Wherein the image generation step, when generating an image from a virtual viewpoint that is set toward the rear of the vehicle, the image using the captured image obtained al a by the predetermined camera located on the vehicle forward of said object A display method for a display device, characterized in that it is generated. In the display method of the display device according to claim 1 ,
In the virtual viewpoint setting step, when the position of the object is on the left side of the own vehicle with respect to the position of the predetermined camera , the virtual viewpoint is directed to the left side of the vehicle from the position on the right side of the own vehicle. The display method of the display device, which is characterized by setting. In the display method of the display device according to claim 1 ,
In the virtual viewpoint setting step, when the position of the object is on the right side of the own vehicle with respect to the position of the predetermined camera , the virtual viewpoint is directed to the right of the vehicle from the position on the left side of the own vehicle. The display method of the display device, which is characterized by setting. In the display method of the display device according to claim 1,
The plurality of cameras are mounted side by side in the front-rear direction of the own vehicle on the side portion of the own vehicle to photograph a lateral region of the own vehicle .
Said target position specifying step, based on the image captured by the predetermined camera, and specifies the position in the longitudinal direction of the object of the vehicle with respect to the position of the predetermined camera,
In the virtual viewpoint setting step, the virtual viewpoint is set at a position including the object and the front and rear wheels of the own vehicle in the field of view.
In the image generation step, when generating an image from the virtual viewpoint, a first imaging obtained by the first camera, which is the predetermined camera and is arranged at a position immediately before the object in the front-rear direction of the own vehicle. Using the image and the second captured image obtained by the second camera arranged at the position immediately after the object in the front-rear direction of the own vehicle with the predetermined camera, the side surface of the own vehicle and the lower side of the vehicle are used. A display method for a display device, which comprises generating an image including an area. An object around the own vehicle is detected, and an image from a virtual viewpoint including the object is generated from the images taken by a plurality of cameras provided in the own vehicle and displayed as visual support information of the occupant. In a display device equipped with a display controller
The display controller is
Among the plurality of cameras, an object position specifying unit that identifies a predetermined camera that detects the object and specifies the position of the object in the front-rear direction of the own vehicle or the left-right direction of the own vehicle with respect to the predetermined camera . When,
A virtual viewpoint setting unit that sets the position of the virtual viewpoint at a position opposite to the position of the object with respect to the predetermined camera, and sets the direction of the virtual viewpoint so that the own vehicle and the object are included in the field of view. When,
An image generation unit that generates an image from the virtual viewpoint and
A display device characterized by having.