JP5681569B2 - Information processing system, server device, and in-vehicle device - Google Patents

Description

  The present invention relates to image display processing.

  Conventionally, when there is an image of an approaching vehicle approaching from the rear of the vehicle in an image taken by a camera provided behind the vehicle, the position of the approaching vehicle relative to the vehicle is derived from the position of the approaching vehicle image in the image. It was. And when the position of the approaching vehicle with respect to the vehicle becomes close compared with the position of the approaching vehicle detected in the past, the user is notified that the approaching vehicle is approaching the vehicle.

  When the distance between the vehicle and the approaching vehicle falls below a predetermined distance, for example, a light emitter such as an LED provided in the vicinity of the A-pillar of the vehicle is turned on to confirm that the approaching vehicle is approaching the vehicle. The vehicle user was informed. As an example of performing such notification, when an approaching vehicle is approaching the moving position of the vehicle when the lane in which the vehicle travels is changed by an operation of the user of the vehicle, the approaching vehicle exists. To the vehicle user. In such a case, the user of the vehicle may cancel the lane change of the vehicle because there is a risk of a collision between the vehicle and the approaching vehicle. Note that there is Patent Document 1 as a material for explaining the technology related to the present invention.

JP 2004-304416 A

  However, even when the distance between the vehicle that is about to change lanes and the approaching vehicle is less than a predetermined distance, if the approaching vehicle approaches the vehicle at a low speed, the approaching vehicle is moved to the moving position of the vehicle. In some cases, the vehicle can move to the moving position faster than the vehicle moves. That is, depending on the approaching state of the approaching vehicle to the vehicle, it may be possible to change the traveling lane of the vehicle by the operation of the user of the vehicle. For this reason, there is a need for a technique that allows the user to accurately grasp the approaching state of the approaching vehicle with respect to the vehicle.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a technology that allows a user of a vehicle to accurately grasp the approaching state of an approaching vehicle to the vehicle.

In order to solve the above-mentioned problem, an invention according to claim 1 is an image display system used in a vehicle, and combines the plurality of photographed images with an acquisition unit that obtains a plurality of photographed images of a camera mounted on the vehicle. And generating means for generating a composite image showing the state of the periphery of the vehicle viewed from a virtual viewpoint, detecting means for detecting the presence of an approaching vehicle approaching from the rear of the vehicle, and relative to the vehicle. Derivation means for deriving the position of the approaching vehicle; and display means for displaying the composite image, wherein the generation means is the composition viewed from the virtual viewpoint according to the position of the vehicle and the position of the approaching vehicle. image der is, the approaching vehicle to generate a specific composite image by increasing the height of the virtual viewpoint closer to the vehicle.

Further, the invention of claim 2 is an image display system used in a vehicle, wherein an acquisition means for acquiring a plurality of photographed images of a camera mounted on the vehicle is combined with the plurality of photographed images to obtain a virtual viewpoint. Generating means for generating a composite image showing the state of the periphery of the vehicle viewed from the above, detecting means for detecting the presence of an approaching vehicle approaching from behind the vehicle, and the position of the approaching vehicle relative to the vehicle. Derivation means for deriving and display means for displaying the composite image, wherein the generation means is the composite image viewed from the virtual viewpoint according to the position of the vehicle and the position of the approaching vehicle, A specific composite image is generated in which the viewpoint position of the virtual viewpoint is the position of the driver of the approaching vehicle and the visual field direction of the virtual viewpoint is the direction in which the vehicle exists .

The invention of claim 3 is an image display system used in a vehicle, wherein an acquisition means for acquiring a plurality of captured images of a camera mounted on the vehicle is combined with the plurality of captured images to obtain a virtual viewpoint. Generating means for generating a composite image showing the state of the periphery of the vehicle viewed from the above, detecting means for detecting the presence of an approaching vehicle approaching from behind the vehicle, and the position of the approaching vehicle relative to the vehicle. Derivation means for deriving and display means for displaying the composite image, wherein the generation means is the specific composition that is the composite image viewed from the virtual viewpoint according to the position of the vehicle and the position of the approaching vehicle An image is generated, and the display means displays one captured image of the captured images from the camera when the approaching vehicle does not exist, and displays the one captured image when the approaching vehicle exists. Both Displaying the specific composite image.

According to a fourth aspect of the present invention, there is provided an image display system used in a vehicle, wherein an acquisition means for acquiring a plurality of captured images of a camera mounted on the vehicle and the plurality of captured images are combined to generate a virtual viewpoint. Generating means for generating a composite image showing the state of the periphery of the vehicle viewed from the above, detecting means for detecting the presence of an approaching vehicle approaching from behind the vehicle, and the position of the approaching vehicle relative to the vehicle. Derivation means for deriving and display means for displaying the composite image, wherein the generation means is the specific composition that is the composite image viewed from the virtual viewpoint according to the position of the vehicle and the position of the approaching vehicle An image is generated, and the display means displays the specific composite image viewed from the virtual viewpoint with the viewpoint position as the position of the driver of the approaching vehicle and the visual field direction as the direction in which the vehicle exists, and the viewpoint position as the viewpoint position. And the specific synthetic image viewed from the virtual viewpoint to present a direction of position and to the approaching vehicle the viewing direction of the door mirror vicinity of both displays are switched in response to user's operation of the vehicle.

According to a fifth aspect of the present invention, in the image display system according to the third aspect , the display means determines the size of the one captured image and the size of the specific composite image in accordance with an operation of a user of the vehicle. Switch to display .

The invention of claim 6 is the image display system according to claim 1 , further comprising operation detection means for detecting an operation of a user of the vehicle accompanying a change in a traveling lane of the vehicle, wherein the display means includes: When the user operation is detected, the specific composite image is displayed .

Further, the invention of claim 7 is the image display system according to claim 6 , wherein the generating means sets the position of the virtual viewpoint to the front of the vehicle, and sets the visual direction of the virtual viewpoint to the rear of the vehicle. The specific composite image including a vehicle image and an approaching vehicle image is generated .

In addition, according to an eighth aspect of the present invention, in the image display system according to the sixth aspect , the generating means sets the viewpoint position of the virtual viewpoint as the position of the driver of the approaching vehicle, and sets the viewing direction of the virtual viewpoint as the vehicle. The specific composite image is generated as the direction in which the image exists .

Further, the invention of claim 9 is the image display system according to claim 6 , in which the predicting means for predicting the moving position when the vehicle moves by changing the traveling lane, and the approaching vehicle at the moving position. Computation means for computing a first movement time for movement, and judgment means for judging whether or not the vehicle can move to the movement position based on the first movement time, further comprising: the movement means An image is generated by superimposing a warning image indicating the determination result on whether or not the synthesized image is superimposed .

According to a tenth aspect of the present invention, in the image display system according to the ninth aspect , the generation unit generates an image in which the warning image is superimposed on the movement position .

The invention according to claim 11 is the image display system according to claim 9 , wherein a measuring means for measuring a second moving time during which the vehicle moves to the moving position, and a memory for learning and storing the second moving time. Means for determining whether the movement is possible based on the first movement time and the stored second movement time .

The invention according to claim 12 is the image display system according to any one of claims 6 to 11 , wherein when the user's operation is detected and the approaching vehicle does not exist, the generation means Generating a vehicle composite image that is the composite image including the image of the vehicle and an image of an adjacent lane, with the viewpoint position of the virtual viewpoint being the front of the vehicle and the visual field direction of the virtual viewpoint being the rear of the vehicle; The means displays the vehicle composite image .

The invention according to claim 13 is the image display system according to any one of claims 1 to 12 , wherein the position acquisition means for acquiring the position of the vehicle, and the destination of the vehicle according to the position of the vehicle Notification means for notifying the user of guide information for guiding the route to the user, and the display means displays the composite image when the notification means notifies the guide information .

In addition, according to a fourteenth aspect of the present invention, in the image display system according to the thirteenth aspect , the generation unit changes the virtual viewpoint of the composite image according to the content of the guidance information notified by the notification unit. .

In the image display system according to claim 14 , the guidance information includes information for guiding the change of the traveling lane and information for guiding the direction change at the intersection .

According to a sixteenth aspect of the present invention, there is provided an image processing apparatus used in a vehicle, wherein an acquisition unit that acquires a plurality of captured images of a camera mounted on the vehicle and the plurality of captured images are combined to generate a virtual viewpoint. Generating means for generating a composite image showing the state of the periphery of the vehicle viewed from the above, detecting means for detecting the presence of an approaching vehicle approaching from behind the vehicle, and the position of the approaching vehicle relative to the vehicle. Deriving means for deriving, wherein the generating means is the composite image viewed from the virtual viewpoint according to the position of the vehicle and the position of the approaching vehicle, and the virtual vehicle as the approaching vehicle approaches the vehicle A specific composite image in which the height of the viewpoint is increased is generated .

The invention of claim 17 is an image display method used in a vehicle, wherein (a) a step of acquiring a plurality of photographed images of a camera mounted on the vehicle, and (b) a combination of the plurality of photographed images. And (c) detecting the presence of an approaching vehicle approaching from the rear of the vehicle; (d) for the vehicle; A step of deriving a relative position of the approaching vehicle; and (e) a step of displaying and displaying the composite image, wherein the step (b) depends on the position of the vehicle and the position of the approaching vehicle. In addition, the composite image viewed from the virtual viewpoint, and a specific composite image in which the height of the virtual viewpoint is increased as the approaching vehicle approaches the vehicle is generated .

  According to the first aspect of the present invention, the generation unit generates a specific composite image that is a composite image viewed from a virtual viewpoint corresponding to the position of the vehicle and the relative position of the approaching vehicle, thereby determining the position of the approaching vehicle. The vehicle user can accurately grasp.

  According to a second aspect of the present invention, the generating means generates a specific composite image including a vehicle image and an approaching vehicle image with the viewpoint position of the virtual viewpoint as the front of the vehicle and the visual field direction of the virtual viewpoint as the rear of the vehicle. By generating, the user of the vehicle can accurately grasp the position of the approaching vehicle viewed from the front of the vehicle.

  According to the invention of claim 3, the generation means generates a specific composite image in which the height of the virtual viewpoint is increased as the approaching vehicle approaches the vehicle, whereby the vehicle image and the approaching vehicle image are obtained. It is included in the composite image, and the user of the vehicle can accurately grasp the position of the approaching vehicle.

  According to the invention of claim 4, the generating means generates the specific composite image with the viewpoint position of the virtual viewpoint as the position of the driver of the approaching vehicle and the visual field direction of the virtual viewpoint as the direction in which the vehicle exists. The vehicle user can accurately grasp the position of the vehicle from the viewpoint of the driver of the approaching vehicle.

  According to the invention of claim 5, the display means displays one captured image of the captured images from the camera when there is no approaching vehicle, and together with one captured image when there is an approaching vehicle. By displaying the specific composite image, the user of the vehicle can recognize the presence of the approaching vehicle at an early stage, and can accurately grasp the position of the approaching vehicle.

  Further, according to the invention of claim 6, the display means includes a specific composite image viewed from a virtual viewpoint in which the viewpoint position is the position of the driver of the approaching vehicle and the visual field direction is the direction in which the vehicle exists, and the viewpoint position is the vehicle position. A specific composite image viewed from a virtual viewpoint having a position near the door mirror and a viewing direction as the direction in which the approaching vehicle is present is displayed by switching according to the operation of the user of the vehicle. The combined image can be displayed to accurately grasp the position of the approaching vehicle.

  According to the seventh aspect of the present invention, the display means switches between the size of the one captured image and the size of the specific composite image in accordance with the operation of the user of the vehicle, so that it corresponds to the traveling state of the vehicle. Thus, the size of the image desired by the user of the vehicle can be adjusted as compared with the sizes of other images.

  According to the invention of claim 8, the display means displays the specific composite image when the user's operation is detected, so that the user of the vehicle can accurately grasp the position of the approaching vehicle and the lane of the vehicle. It is possible to easily determine whether the change can be made.

  According to the invention of claim 9, the generating means generates a specific composite image including a vehicle image and an approaching vehicle image with the position of the virtual viewpoint as the front of the vehicle and the visual direction of the virtual viewpoint as the rear of the vehicle. By doing so, the user of the vehicle can accurately grasp the position of the approaching vehicle viewed from the front of the vehicle, and can easily determine whether or not the lane of the vehicle can be changed.

  According to the invention of claim 10, the generating means generates the specific composite image with the viewpoint position of the virtual viewpoint as the position of the driver of the approaching vehicle and the visual field direction of the virtual viewpoint as the direction in which the vehicle exists. The user of the vehicle can accurately grasp the position of the vehicle from the driver's viewpoint of the approaching vehicle, and can easily determine whether the lane of the vehicle can be changed.

  According to the eleventh aspect of the present invention, the generation unit superimposes a warning image indicating the determination result of the movement on the composite image, so that the user of the vehicle accurately grasps the approaching state of the approaching vehicle to the vehicle. Thus, it is possible to easily determine whether the lane of the vehicle can be changed.

  According to the invention of claim 12, the display means displays the photographed image in a display mode corresponding to the current vehicle position, so that the user who travels for the first time at the point corresponding to the vehicle position can display the photographed image. An optimal display mode can be provided.

  According to the invention of claim 13, the determination means determines whether the vehicle 9 can move to the movement position by determining based on the first movement time and the stored second movement time. The user of the vehicle 9 can easily determine according to the user's specific lane change time.

  According to the invention of claim 14, when the user's operation is detected and there is no approaching vehicle, the generation means sets the viewpoint position of the virtual viewpoint to the front of the vehicle and sets the visual direction of the virtual viewpoint to the vehicle. As a rear, a vehicle composite image that is a composite image including an image of the vehicle is generated, and the display means displays the vehicle composite image, so that the user can determine whether the lane of the vehicle can be changed regardless of the presence of an approaching vehicle. Easy to judge.

  According to the invention of claim 15, the display means displays the composite image when the notification means notifies the guide information, so that the user can see the composite image viewed from the virtual viewpoint position corresponding to the guide information. I can confirm.

  According to the sixteenth aspect of the present invention, the generation means changes the virtual viewpoint of the composite image according to the content of the guidance information notified by the notification means, so that the vehicle traveling state corresponding to the content of the guidance information The user of the vehicle can confirm the composite image corresponding to the above.

  According to the invention of claim 17, the guidance information includes information for guiding the change of the traveling lane and information for guiding the change of the direction at the intersection, so that the change of the traveling lane and the intersection The user of the vehicle can confirm the composite image corresponding to the change in the traveling direction.

FIG. 1 is a block diagram of an image display system. FIG. 2 is a diagram illustrating a position where the in-vehicle camera is arranged in the vehicle. FIG. 3 is a diagram for explaining a method of generating a composite image. FIG. 4 is a diagram for mainly explaining the viewpoint position of the virtual viewpoint. FIG. 5 is a diagram for mainly explaining the viewing direction of the virtual viewpoint. FIG. 6 is a diagram for mainly explaining the viewing direction of the virtual viewpoint. FIG. 7 is a diagram illustrating a composite image viewed from a virtual viewpoint displayed on the display. FIG. 8 is a diagram illustrating a composite image viewed from a virtual viewpoint displayed on the display. FIG. 9 is a process flowchart of the image processing apparatus. FIG. 10 is a diagram for mainly explaining the viewpoint position of the virtual viewpoint. FIG. 11 is a diagram for mainly explaining the viewing direction of the virtual viewpoint. FIG. 12 is a diagram for mainly explaining the viewing direction of the virtual viewpoint. FIG. 13 is a diagram illustrating a composite image viewed from a virtual viewpoint displayed on the display. FIG. 14 is a diagram illustrating a composite image viewed from a virtual viewpoint displayed on the display. FIG. 15 is a block diagram of the image display system. FIG. 16 is a diagram illustrating a composite image viewed from a virtual viewpoint displayed on the display. FIG. 17 is a process flowchart of the image processing apparatus. FIG. 18 is a process flowchart of the image processing apparatus. FIG. 19 is a block diagram of the image display system. FIG. 20 is a graph illustrating an example of measurement of the lane change time of the vehicle. FIG. 21 is a process flowchart in which the learning unit measures the lane change time. FIG. 22 is a process flowchart in which the learning unit measures the lane change time. FIG. 23 is a diagram for mainly explaining the viewpoint position of the virtual viewpoint. FIG. 24 is a diagram for mainly explaining the viewing direction of the virtual viewpoint. FIG. 25 is a diagram illustrating a composite image viewed from a virtual viewpoint displayed on the display. FIG. 26 is a process flowchart of the image processing apparatus. FIG. 27 is a process flowchart of the image processing apparatus. FIG. 28 is a diagram illustrating a composite image viewed from a virtual viewpoint displayed on the display. FIG. 29 is a process flowchart of the image processing apparatus. FIG. 30 is a process flowchart of the image processing apparatus. FIG. 31 is a diagram for mainly explaining the viewpoint position of the virtual viewpoint. FIG. 32 is a diagram for mainly explaining the viewing direction of the virtual viewpoint. FIG. 33 is a diagram illustrating a composite image viewed from a virtual viewpoint displayed on the display. FIG. 34 is a process flowchart of the image processing apparatus. FIG. 35 is a process flowchart of the image processing apparatus. FIG. 36 shows an example of a virtual viewpoint of a composite image displayed on a display together with a photographed image of one camera. FIG. 37 is a diagram showing an image displayed on the display. FIG. 38 is a diagram showing an image displayed on the display. FIG. 39 is a process flowchart for displaying one photographed image and a composite image together on a display. FIG. 40 is a process flowchart for displaying one photographed image and a composite image together on a display. FIG. 41 is a diagram illustrating switching between the size of the captured image of the front camera and the size of the composite image. FIG. 42 is a diagram illustrating points where the guide information is notified to the user. FIG. 43 is a diagram illustrating an example of the viewpoint position and visual field direction of the virtual viewpoint at a point where the guidance information is notified. FIG. 44 is a diagram illustrating a composite image viewed from a virtual viewpoint. FIG. 45 is a process flowchart in which the image processing apparatus displays the composite image on the display when the display apparatus notifies the guide information. FIG. 46 is a process flowchart in which the image processing apparatus displays the composite image on the display when the display apparatus notifies the guide information.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  <First Embodiment>

  <1. Configuration>

<1-1. Image display system>

  FIG. 1 is a block diagram of the image display system 120. An image processing device 11 that generates a peripheral image showing the periphery of a vehicle (vehicle 9 shown in FIG. 2) on which the photographing unit 5 is mounted and outputs image information, and a display that displays an image output from the image processing device 11 The image display system 120 mainly includes the device 20.

  The display device 20 includes a navigation device having a display for displaying navigation guidance, an audio device having a display for displaying music title information, television images, and the like, a smartphone that can be carried by the user and brought into the vehicle 9, and the like. It is.

  The display device 20 includes a control unit 21 that controls the entire device, a storage unit 22 that stores processing data of the control unit 21, a communication unit 23 that communicates with the image processing device 11, and a touch panel function. A display 24 such as a liquid crystal is provided, and an operation unit 25 that is operated by the user. And the display apparatus 20 is installed in the instrument panel etc. of the vehicle 9 so that the screen of the display 24 can be visually recognized from a user. Various instructions from the user are received by the operation unit 25 and the display 24 as a touch panel.

  The control unit 21 is a computer including a CPU, a RAM, a ROM, and the like, and various functions including a navigation function are realized by the CPU performing arithmetic processing according to a predetermined program.

  The control unit 21 mainly has functions of a display control unit 201 and a position deriving unit 202. The display control unit 201 controls a display mode of a captured image of the in-vehicle camera displayed on the display 24 in accordance with a user operation from the operation unit 25 or the like.

  The position deriving unit 202 derives latitude / longitude information corresponding to the position of the vehicle 9. Such a position of the vehicle 9 is realized by receiving a radio wave signal from a GPS (Global Positioning System) satellite and processing the received signal by the control unit 21. The GPS is a system that measures the position of the vehicle 9 based on radio waves from at least three or more GPS satellites that orbit the earth.

  The storage unit 22 stores a program for the control unit 21 to control the display device 20. Further, the storage unit 22 stores map information 202 that is information of a map image displayed on the display 24.

  The communication unit 23 is an interface for performing communication between the display device 20 and the image processing device 11. The communication unit 23 receives image information and the like from the image processing device, and receives an instruction signal when an operation unit 25 described later is operated by the user. It transmits to the image processing apparatus 11.

  The display 24 is a display unit that displays various images, and normally displays an image based on the function of the display device 20 alone under the control of the display control unit 201. And the surrounding image which shows the mode of the periphery of the vehicle 9 produced | generated with the image processing apparatus 11 on the predetermined conditions is displayed. Thereby, the display device 20 also functions as a device that receives and displays the peripheral image generated by the image processing device 11.

  The operation unit 25 is operated by a user to realize various functions of the display device 20, and is, for example, a hard switch provided in the display device 20. The display mode of the captured image of the in-vehicle camera displayed on the display 24 can be switched by the user operating the operation unit 25.

  The image processing apparatus 11 is an ECU (Electronic Control Unit) having a function of generating a peripheral image of the vehicle 9 and is disposed at a predetermined position of the vehicle 9. The image processing device 11 functions as an image generation device that generates a composite image viewed from a virtual viewpoint based on a captured image obtained by photographing the periphery of the vehicle 9 with the photographing unit 5. Further, the image processing device 11 also functions as a device that processes an image shot by one camera of the shooting unit 5. The plurality of in-vehicle cameras 51, 52, 53, and 54 provided in the photographing unit 5 are arranged at appropriate positions of the vehicle 9 different from the image processing apparatus 11, but details will be described later.

  The image processing apparatus 11 mainly includes a control unit 1 that controls the entire apparatus and an image processing unit 3 that processes a captured image acquired by the imaging unit 5 to generate a peripheral image for display.

  Various instructions from the user received by either the operation unit 25 of the display device 20 or the display 24 are received by the communication unit 61 as a control signal via the communication unit 23 of the display device 20 and are received by the control unit 1. Is input. As a result, the image processing apparatus 11 can operate in response to a user operation on the display device 20.

  The image processing unit 3 is configured as a hardware circuit capable of various image processing, and includes an image acquisition unit 31, a composite image generation unit 32, and an image output unit 33 as main functions.

  The image acquisition unit 31 acquires a plurality of captured images captured by the in-vehicle cameras 51, 52, 53, and 54 of the imaging unit 5 that captures the periphery of the vehicle 9.

  The composite image generation unit 32 generates a composite image viewed from an arbitrary virtual viewpoint around the vehicle, based on a plurality of captured images acquired by the in-vehicle cameras 51, 52, 53, and 54. A method by which the composite image generation unit 32 generates a composite image viewed from a virtual viewpoint will be described later.

  Further, the image output unit 33 displays at least one image of the composite image generated by the composite image generation unit 32 and other images (for example, images taken by one camera) via the communication unit 61. 20 is transmitted. Thereby, a peripheral image showing the periphery of the vehicle is displayed on the display 24 of the display device 20.

  The control unit 1 is a computer including a CPU, a RAM, a ROM, and the like, and various control functions are realized by the CPU performing arithmetic processing according to a predetermined program. The image control unit 111, the detection unit 112, and the derivation unit 113 shown in the drawing are a part of the functions of the control unit 1 realized in this way.

  The image control unit 111 controls image processing executed by the image processing unit 3. For example, the image control unit 111 instructs the composite image generation unit 32 about various parameters necessary for generating a composite image according to the position of the vehicle 9. For example, when the control unit 1 receives an instruction signal of a user operation accompanying switching of the display mode of the captured image, the image control unit 111 performs such parameter instruction.

  The image control unit 111 also corresponds to the position of the vehicle 9 and the position of the vehicle approaching the vehicle 9 from the rear of the vehicle 9 (hereinafter also referred to as “approaching vehicle”. For example, the approaching vehicle 10 shown in FIGS. 4 and 5). The composite image generation unit 32 is instructed to generate a composite image viewed from the virtual viewpoint.

  The detection unit 112 detects the presence of the approaching vehicle 10 approaching from behind the vehicle 9. That is, an image of an approaching vehicle existing behind the vehicle 9 (hereinafter also referred to as “approaching vehicle image”) from a photographed image of the back camera 52 provided behind the vehicle 9. For example, the composite image PT1 and the composite image in FIG. The approaching vehicle image 101) shown in PT2 is detected by processing such as edge detection.

  The detection of the presence of the approaching vehicle 10 approaching the vehicle 9 is based on the difference between the position in the image of a certain vehicle image detected by the detection unit 112 in the past and the position in the image of the vehicle image detected this time. Do it. Another method for detecting the presence of the approaching vehicle 10 is to compare the size of a certain vehicle image detected in the past with the size of the vehicle image detected this time.

  The deriving unit 113 derives the relative position of the approaching vehicle 10 with respect to the vehicle 9. That is, the position of the approaching vehicle 10 relative to the vehicle 9 (the distance of the approaching vehicle 10 relative to the vehicle 9 and the vehicle 9 based on the difference in the position between the vehicle image 91 and the approaching vehicle image 101 in the image detected by the detection unit 112. The direction (angle) of the approaching vehicle 10 is derived.

  The image processing apparatus 11 further includes a nonvolatile memory 4, a communication unit 61, and a signal input unit 62, which are connected to the control unit 1.

  The nonvolatile memory 4 is configured by a flash memory or the like that can maintain stored contents even when the power is turned off. In the nonvolatile memory 4, vehicle type information 40 is stored. The vehicle type information 40 includes data corresponding to the type of vehicle that is required when the composite image generation unit 32 generates a composite image.

  In addition, the nonvolatile memory 4 stores virtual viewpoint information 41 that is information on the viewpoint position and visual field direction of the virtual viewpoint VP corresponding to the position of the approaching vehicle 10 with respect to the vehicle 9. When the image control unit 111 sets the viewpoint position and the viewing direction of the virtual viewpoint VP, the image control unit 111 obtains information on the viewpoint position and the viewing direction of the virtual viewpoint VP corresponding to the position of the approaching vehicle 10 with respect to the vehicle 9. Read from the nonvolatile memory 4.

  The communication unit 61 is an interface for the image processing apparatus 11 to communicate with the display device 20.

  The signal input unit 62 is an interface when signals are input from various devices provided in the vehicle 9. A signal from the outside of the image processing apparatus 11 is input to the control unit 1 via the signal input unit 62. Specifically, signals indicating various information are input to the control unit 1 from the shift sensor 81, the vehicle speed sensor 82, the direction indicator 83, the steering angle sensor 84, and the like.

  From the shift sensor 81, the operation position of the shift lever of the transmission of the vehicle 9, ie, “P (parking)”, “D (forward)”, “N (neutral)”, “R (reverse)”, etc. The shift position is input. From the vehicle speed sensor 82, the traveling speed (km / h) of the vehicle 9 at that time is input.

  From the direction indicator 83, a turn signal indicating a direction instruction based on the operation of the blinker switch, that is, a direction instruction intended by the driver of the vehicle 9 is input. When the turn signal switch is operated, a turn signal is generated, and the turn signal indicates the operated direction (left direction or right direction). When the turn signal switch is in the neutral position, the turn signal is turned off.

  From the steering angle sensor 84, the operation direction of the steering wheel operated by the user is input to the control unit 1. Specifically, the steering angle sensor 84 receives the rotation direction, rotation angle, and rotation speed of the steering wheel operated by the user via the signal input unit 62.

<1-3. Shooting Department>
Next, the photographing unit 5 of the image processing apparatus 11 will be described in detail. The photographing unit 5 is electrically connected to the control unit 1 and operates based on a signal from the control unit 1.

  The imaging unit 5 includes a front camera 51, a back camera 52, a left side camera 53, and a right side camera 54 that are in-vehicle cameras. Each of these vehicle-mounted cameras 51, 52, 53, and 54 includes an image sensor such as a CCD or a CMOS and electronically acquires an image.

  FIG. 2 is a diagram illustrating positions where the in-vehicle cameras 51, 52, 53, 54 are arranged on the vehicle 9. In the following description, the vehicle 9 will be described as an example. Also, when indicating the direction and orientation, the three-dimensional XYZ orthogonal coordinates shown in the figure are used as appropriate. The XYZ axes are fixed relative to the vehicle 9. Here, the X-axis direction is along the left-right direction of the vehicle 9, the Y-axis direction is along the straight traveling direction (front-rear direction) of the vehicle 9, and the Z-axis direction is along the vertical direction. For convenience, the + X side is the right side of the vehicle A1, the + Y side is the rear side of the vehicle A1, and the + Z side is the upper side.

  The front camera 51 is provided in the vicinity of the license plate mounting position at the front end of the vehicle 9, and its optical axis 51 a is directed in the straight traveling direction of the vehicle 9 (−Y side in the Y-axis direction in plan view). The back camera 52 is provided in the vicinity of the license plate mounting position at the rear end of the vehicle 9, and its optical axis 52 a is directed in the reverse direction of the vehicle 9 in the straight traveling direction (+ Y side in the Y-axis direction in plan view). Yes. The left side camera 53 is provided on the left door mirror 93, and the right side camera 54 is provided on the right door mirror 94. The optical axis 53a is directed to the left side of the vehicle 9, and the optical axis 54a is directed to the outside along the right direction of the vehicle 9 (X-axis direction in plan view). Note that the mounting position of the front camera 51 and the back camera 52 is preferably approximately the center in the left and right, but may be slightly shifted in the left and right directions from the center in the left and right.

  As the lenses of these in-vehicle cameras 51, 52, 53, and 54, fish-eye lenses and the like are employed, and the in-vehicle cameras 51, 52, 53, and 54 have an angle of view α of 180 degrees or more. For this reason, by using the four in-vehicle cameras 51, 52, 53, and 54, it is possible to capture the entire periphery of the vehicle 9.

<1-4. Image conversion processing>
Next, a method in which the composite image generation unit 32 of the image processing unit 3 generates a composite image that shows a state in which the periphery of the vehicle 9 is viewed from an arbitrary virtual viewpoint based on a plurality of captured images obtained by the imaging unit 5. explain. When generating a composite image, vehicle type information 40 stored in advance in the nonvolatile memory 4 is used. FIG. 3 is a diagram for explaining a method of generating a composite image.

  When the front camera 51, the back camera 52, the left side camera 53, and the right side camera 54 of the photographing unit 5 are simultaneously photographed, four images respectively indicating the front, rear, left side, and right side of the vehicle 9 are shown. Captured images P1 to P4 are acquired. That is, the four captured images P1 to P4 acquired by the imaging unit 5 include information indicating the entire periphery of the vehicle 9 at the time of shooting.

  Next, each pixel of the four captured images P1 to P4 is projected onto a three-dimensional curved surface SP in a virtual three-dimensional space. The three-dimensional curved surface SP has, for example, a substantially hemispherical shape (a bowl shape), and a center portion (a bottom portion of the bowl) is determined as a position where the vehicle 9 exists. A correspondence relationship is determined in advance between the positions of the pixels included in the captured images P1 to P4 and the positions of the pixels of the three-dimensional curved surface SP. For this reason, the value of each pixel of the solid curved surface SP can be determined based on this correspondence and the value of each pixel included in the captured images P1 to P4.

  The correspondence between the positions of the pixels of the captured images P1 to P4 and the positions of the pixels of the three-dimensional curved surface SP is the arrangement of the four in-vehicle cameras 51, 52, 53, and 54 in the vehicle 9 (the distance between each other, the ground height, Depending on the optical axis angle, etc.). For this reason, the table data indicating this correspondence is included in the vehicle type information 40 stored in the nonvolatile memory 4.

  Further, polygon data indicating the shape and size of the vehicle body included in the vehicle type information 40 is used, and a vehicle image that is a polygon model indicating the three-dimensional shape of the vehicle A1 is virtually provided. The configured vehicle image is arranged in a substantially hemispherical central portion determined as the position of the vehicle 9 in the three-dimensional space where the three-dimensional curved surface SP is set.

  Further, the virtual viewpoint VP is set by the control unit 1 for the three-dimensional space where the solid curved surface SP exists. The virtual viewpoint VP is defined by the viewpoint position and the visual field direction, and is set to an arbitrary visual field position corresponding to the periphery of the vehicle 9 in this three-dimensional space toward an arbitrary visual field direction.

  Then, according to the set virtual viewpoint VP, a necessary area on the three-dimensional curved surface SP is cut out as an image. The relationship between the virtual viewpoint VP and the necessary area in the three-dimensional curved surface SP is determined in advance, and stored in advance in the nonvolatile memory 4 or the like as table data. That is, an image area is cut out according to the set virtual viewpoint VP, rendering is performed on a vehicle image composed of polygons, and the resulting two-dimensional vehicle image is superimposed on the cut out image. Is done. As a result, a composite image showing a state in which the periphery of the vehicle 9 is viewed from an arbitrary virtual viewpoint is generated.

  For example, when a virtual viewpoint VPA is set in which the viewpoint position is directly above the center of the position of the vehicle 9 and the visual field direction is approximately directly below, the vehicle 9 ( Actually, it is a vehicle image, for example, a vehicle image 91) displayed in the composite image PTA and a composite image PTA showing the state of the periphery of the vehicle 9 are generated. Also, as shown in the figure, when a virtual viewpoint VPB is set in which the viewpoint position is the left rear of the position of the vehicle 9 and the viewing direction is substantially forward in the vehicle 9, the entire periphery from the left rear of the vehicle 9 is set. As seen, the vehicle 9 (actually a vehicle image, for example, the vehicle image 91 shown in the composite image PTB) and the composite image PTB showing the state of the periphery of the vehicle 9 are generated.

In the case of actually generating a composite image, it is not necessary to determine the values of all the pixels of the three-dimensional curved surface SP, and only the values of the pixels in the area necessary corresponding to the set virtual viewpoint VP are photographed. By determining based on the images P1 to P4, the processing speed can be improved.
<1-5. Virtual viewpoint position and viewing direction>
Next, the viewpoint position and visual field direction of the virtual viewpoint according to the positions of the vehicle 9 and the approaching vehicle 10 will be described. FIG. 4 is a diagram for mainly explaining the viewpoint position of the virtual viewpoint VP. 5 and 6 are diagrams for mainly explaining the viewing direction of the virtual viewpoint VP.

  In FIG. 4, an axis sh <b> 1 extending in the front-rear direction (Y-axis direction) of the vehicle 9 at a substantially central position between the position of the vehicle 9 and the position of the approaching vehicle 10 in the vehicle width direction (X-axis direction). It is shown. In addition, virtual viewpoints VP1 to VP4 that move so as to draw a parabola on a plane perpendicular to the road surface through the axis sh1 are shown. Specifically, as the approaching vehicle 10 approaches the vehicle 9, the height of the viewpoint position of the virtual viewpoint VP increases. Further, the movement of the virtual viewpoint VP is performed so that the vehicle image 91 and the approaching vehicle image 101 are included in the composite image. By moving the viewpoint position of the virtual viewpoint VP in this way, the image of the vehicle 9 and the image of the approaching vehicle 10 are included in the composite image, and the position of the approaching vehicle 10 is determined by the user of the vehicle 9 (hereinafter referred to as the unit “user”). Can also be accurately grasped. The movement from the virtual viewpoint VP3 to VP4 is performed when the vehicle 9 has overtaken the approaching vehicle 10.

  Specifically, for example, when the viewpoint position of the virtual viewpoint VP1 is 30 meters or more between the vehicle 9 and the approaching vehicle 10, the viewpoint position of the virtual viewpoint VP2 is 10 meters or more when the distance between the vehicle 9 and the approaching vehicle 10 is less than 30 meters. Are set by the image control unit 111 respectively. The viewpoint position of the virtual viewpoint VP3 is after the distance between the vehicle 9 and the approaching vehicle 10 is less than 1 m, and the viewpoint position of the virtual viewpoint VP4 is after the distance between the vehicle 9 and the approaching vehicle 10 is less than 1 m. When the approaching vehicle 10 overtakes the vehicle 9, the image control unit 111 sets the approaching vehicle 10.

FIG. 5 is a diagram for mainly explaining the visual field direction of the virtual viewpoint VP1 and the visual field direction of the virtual viewpoint VP2. The visual field direction of the virtual viewpoint VP1 shown in the left diagram of FIG. 5 is directed to the rear of the vehicle 9 from the viewpoint position in front of the vehicle 9 (−Y direction).
Specifically, the deriving unit 113 derives a straight line CL1 directly connecting the vehicle 9 from the position of the approaching vehicle image 101 in the composite image. That is, the deriving unit 113 derives the direction of the approaching vehicle 10 relative to the vehicle 9 and the distance of the approaching vehicle 10 relative to the vehicle 9 from the position of the approaching vehicle image 101 in the composite image.

  As another derivation method, the vehicle 9 (for example, the center of gravity position CP of the vehicle body) in the front-rear direction (Y-axis direction) of the vehicle 9 and the approaching vehicle 10 and the approaching vehicle 10 (for example, an approaching vehicle existing behind the vehicle 9). The straight line AL with the approaching vehicle body position DP) closest to the vehicle 9 is derived to the deriving unit 113. Then, a straight line BL between the vehicle 9 and the approaching vehicle 10 is derived to the deriving unit 113 in the lateral direction (X direction). As a result, the deriving unit 113 may derive a straight line CL1 connecting the vehicle 9 and the approaching vehicle 10 from the straight line AL and the straight line BL. Deriving such a straight line CL1 corresponds to deriving the position of the approaching vehicle 10 relative to the vehicle 9.

  Then, the visual field direction of the virtual viewpoint VP1 is directed to the direction of the point CE1 that is substantially the center of the straight line CL1. Such a composite image viewed from the viewpoint position and the visual field direction of the virtual viewpoint VP1 includes images of both the vehicle 9 and the approaching vehicle 10. The user can accurately grasp the position of the approaching vehicle 10 relative to the vehicle 9 from the positions of the vehicle image 91 and the approaching vehicle image 101 in the image displayed on the display 24.

  Next, the right diagram in FIG. 5 shows a state in which the approaching vehicle 10 is closer to the vehicle 9 than the left diagram in FIG. 5 described above. The virtual viewpoint VP2 has a viewpoint position in front of the vehicle 9 (−Y direction), and the visual field direction is an abbreviation of a straight line CL2 that connects the position of the vehicle 9 that is behind the vehicle 9 (+ Y direction) and the position of the approaching vehicle 10. It is directed to the central point CE2. Further, the viewpoint position of the virtual viewpoint VP2 is higher than the virtual viewpoint VP1.

  FIG. 6 is a diagram mainly illustrating the visual field direction of the virtual viewpoint VP3 and the visual field direction of the virtual viewpoint VP4. The viewpoint position of the virtual viewpoint VP3 shown in the left diagram of FIG. 6 is at a position almost immediately above the center of the straight line CL3 connecting the vehicle 9 and the approaching vehicle 10, and the visual field direction is vertically downward (from the + Z direction to the −Z direction). Pointed to point CE3.

  The virtual viewpoint VP4 shown in the right diagram of FIG. 6 is located at a point CE4 at the center of the straight line CL4 that connects the position of the vehicle 9 and the position of the approaching vehicle 10 with the viewpoint position behind the vehicle 9 (+ Y direction). Is directed. That is, the visual field direction of the virtual viewpoint VP4 is directed in front of the vehicle 9 (−Y direction).

  Note that the composite image viewed from the virtual viewpoint VP in the viewpoint position and visual field direction shown in FIGS. 4 to 6 includes a vehicle image 91 corresponding to the vehicle 9 and an approaching vehicle image 101 corresponding to the approaching vehicle 10. As described above, the viewpoint position and the visual field direction of the virtual viewpoint VP are viewpoint positions corresponding to the positions of the vehicle 9 and the approaching vehicle 10. Thereby, the user can grasp | ascertain the position of the approaching vehicle 10 correctly.

<1-6. Display image>

  FIGS. 7 and 8 are views showing a composite image viewed from the virtual viewpoint VP displayed on the display 24. FIG. The composite images PT1 to PT4 shown in FIGS. 7 and 8 are temporally continuous composite images.

  A composite image PT1 shown in FIG. 7 is a composite image viewed from the virtual viewpoint VP1, and an approaching vehicle image 101 corresponding to the approaching vehicle 10 is displayed behind the vehicle image 91 corresponding to the vehicle 9. The composite image PT2 is a composite image viewed from the virtual viewpoint VP2, and the approaching vehicle image 101 is displayed behind the vehicle image 91. In the composite image PT2, the approaching vehicle image 101 is closer to the vehicle image 91 than the composite image PT1. By displaying such a composite image, the user can accurately grasp the position of the approaching vehicle.

The composite image PT3 in FIG. 8 is a composite image viewed from the virtual viewpoint VP3, and the vehicle approaching to the side of the vehicle image 91 corresponding to the vehicle 9 viewed vertically downward from the virtual viewpoint position directly above the vehicle 9 and the approaching vehicle 10. An image in a state where the approaching vehicle image 101 corresponding to 10 is located is displayed.
The composite image PT4 is a composite image viewed from the virtual viewpoint VP4, and an image with the approaching vehicle image 101 overtaking the vehicle image 91 is displayed. As shown in FIG. 7 and FIG. 8, both the vehicle image 91 corresponding to the vehicle 9 and the approaching vehicle image 101 corresponding to the approaching vehicle 10 are included in the composite image viewed from the virtual viewpoints VP1, VP2, VP3, and VP4. An image of the vehicle is shown. Thus, by displaying the position of the approaching vehicle relative to the vehicle 9 in time series, the user of the vehicle 9 can accurately grasp the position of the approaching vehicle.

<1-7. Processing flowchart>

  FIG. 9 is a process flowchart of the image processing apparatus 11. In step S101, when the detection unit 112 detects the presence of the approaching vehicle 10 in the image captured by the back camera 52 of the imaging unit 5 (Yes in step S101), the process proceeds to step S102. If the detection unit 112 has not detected the presence of the approaching vehicle 10 (No in step S101), the process ends.

  In step S102, the deriving unit 113 derives the position of the approaching vehicle 10 relative to the vehicle 9, that is, the position of the approaching vehicle 10 relative to the vehicle 9 (step S102), and the process proceeds to step S103.

  In step S103, the image control unit 111 sets the viewpoint position of the virtual viewpoint VP and the visual field direction from the position of the approaching vehicle 10 with respect to the vehicle 9 (step S103), and the process proceeds to step S104. Note that the setting of the viewpoint position and the visual field direction of the virtual viewpoint VP from the position of the approaching vehicle 10 with respect to the vehicle 9 includes information on the viewpoint position and visual field direction of the virtual viewpoint VP corresponding to the position of the approaching vehicle 10 with respect to the vehicle 9. The image control unit 111 reads out and sets from the nonvolatile memory 4.

In step S104, the composite image generation unit 32 generates a composite image viewed from the virtual viewpoint VP (step S104), and the process proceeds to step S105.
In step S105, the image output unit 33 outputs the composite image generated by the composite image generation unit 32 to the display device 20 via the communication unit 61 (step S105).

<Second Embodiment>

  Next, a second embodiment will be described. The differences between the second embodiment and the first embodiment are the viewpoint position and the viewing direction of the virtual viewpoint. That is, in the first embodiment, the virtual viewpoints VP1 to VP1 move so as to draw a parabola on the axis sh1 extending in the front-rear direction of the vehicle 9 at the approximate center between the position of the vehicle 9 and the position of the approaching vehicle 10. VP4 has been described.

  On the other hand, in the second embodiment, as shown in FIG. 10, the vehicle 9 is located approximately at the center of the vehicle body and passes through the axis sh2 extending in the front-rear direction (Y-axis direction) of the vehicle 9. The virtual viewpoint VP that moves so as to draw a parabola on a plane inclined by a predetermined angle (for example, 20 degrees) from VP11 moves to the position of VP14. Other configurations and processes are the same as those in the first embodiment. For this reason, description of parts having the same configuration and processing is omitted.

  <2-1. Virtual viewpoint position and viewing direction>

  FIG. 10 is a diagram for mainly explaining the viewpoint position of the virtual viewpoint VP. 11 and 12 are diagrams for mainly explaining the viewing direction of the virtual viewpoint VP.

As shown in FIG. 10, the virtual viewpoint VP is positioned substantially at the center of the vehicle body of the vehicle 9 (for example, the center of the front bumper) and passes through the axis sh2 extending in the front-rear direction (Y-axis direction) of the vehicle 9 It moves to the positions of VP11 to VP14 so as to draw a parabola on a plane inclined by a predetermined angle from the direction, that is, looking down diagonally downward from the left side information of the vehicle 9.
Specifically, in the movement of the viewpoint position from the virtual viewpoint VP11 to VP12 and from the virtual viewpoint VP12 to VP13, the height of the viewpoint position increases as the approaching vehicle 10 approaches the vehicle 9. Further, the position of the virtual viewpoint VP becomes closer to the vehicle 9 as the linear distance between the vehicle 9 and the approaching vehicle 10 becomes shorter (for example, the distance between the bumper portion behind the vehicle 9 and the bumper portion ahead of the approaching vehicle 10 becomes shorter). From the front of the vehicle to the side of the vehicle 9.

  Then, as the linear distance between the vehicle 9 and the approaching vehicle 10 is further shortened, the position of the virtual viewpoint VP moves from the side of the vehicle 9 to the rear of the vehicle 9. As the viewpoint position moves in this way, the image of the vehicle 9 and the image of the approaching vehicle 10 are included in the composite image, and the user can accurately grasp the position of the approaching vehicle 10.

  The movement from the virtual viewpoint VP13 to VP14 is performed when the vehicle 9 has overtaken the approaching vehicle 10. That is, the viewpoint position of the virtual viewpoint VP is a viewpoint position corresponding to the positions of the vehicle 9 and the approaching vehicle 10.

  Further, for example, the viewpoint position of the virtual viewpoint VP11 is when the distance between the vehicle 9 and the approaching vehicle 10 is 30 m or more, and the viewpoint position of the virtual viewpoint VP12 is when the distance between the vehicle 9 and the approaching vehicle 10 is less than 30 m and 10 m or more. It becomes the viewpoint position of each virtual viewpoint. In addition, when the distance between the vehicle 9 and the approaching vehicle 10 is less than 1 m and the distance between the vehicle 9 and the approaching vehicle 10 is 1 m or more, the viewpoint position of the virtual viewpoint VP13 is after the distance between the vehicle 9 and the approaching vehicle 10 is less than 1 m. When the approaching vehicle 10 passes the vehicle 9, the viewpoint position of each virtual viewpoint is obtained.

  FIG. 11 is a diagram mainly illustrating the visual field direction of the virtual viewpoint VP11 and the visual field direction of the virtual viewpoint VP12. In the virtual viewpoint VP11 shown in the left diagram of FIG. 11, the line of sight of the virtual viewpoint VP11 passes from the viewpoint position diagonally left front of the vehicle 9 to the center of gravity position CP that is the position of the approximate center of gravity of the vehicle body of the vehicle 9. Then, the viewing direction is such that the field of view is directed to a substantially central point CE11 connecting the center of gravity position CP located behind the vehicle 9 (+ Y direction) and the position of the approaching vehicle 10.

  Next, the virtual viewpoint VP12 shown in the right diagram of FIG. 11 passes the line of sight of the virtual viewpoint VP12 from the viewpoint position in front of the vehicle 9 tilted further to the left than the virtual viewpoint VP11 on the center of gravity position CP. The viewing direction is such that the line of sight is directed to a point CE12 at the approximate center of the straight line connecting the position of the approaching vehicle 10.

  FIG. 12 is a diagram for mainly explaining the visual field direction of the virtual viewpoint VP13 and the visual field direction of the virtual viewpoint VP14. The virtual viewpoint VP13 shown in the left diagram of FIG. 12 passes the line of sight of the virtual viewpoint VP13 from the viewpoint position on the left side (−X direction) of the vehicle 9 over the center of gravity position CP, and the center of gravity CP and the right side of the vehicle 9 ( The visual field direction is such that the line of sight is directed to a point CE13 at substantially the center of a straight line connecting the position of the approaching vehicle 10 existing in the (+ X direction).

  The virtual viewpoint VP14 shown in the right figure of FIG. 12 passes from the viewpoint position of the rear side (+ Y direction) of the vehicle 9 to the center of gravity position CP, and the line of sight of the virtual viewpoint VP14 passes through the center of gravity position CP and the vehicle 9 diagonally to the right. The visual field direction is directed to a substantially central point CE14 that connects the position of the approaching vehicle 10 that moves. 10 to 12 include a vehicle image 91 corresponding to the vehicle 9 and an approaching vehicle image 101 corresponding to the approaching vehicle 10 in the synthesized image viewed from the virtual viewpoint VP in the viewpoint position and the viewing direction.

  <2-2. Display image>

  FIG. 13 and FIG. 14 are diagrams showing the composite image viewed from the virtual viewpoint displayed on the display 24. Note that the composite images PT11 to PT14 shown in FIGS. 7 and 8 are temporally continuous composite images. The composite image PT11 is a composite image viewed from the virtual viewpoint VP11, and the approaching vehicle image 101 is displayed behind the vehicle image 91. The composite image PT12 is a composite image viewed from the virtual viewpoint VP12, and the approaching vehicle image 101 is displayed behind the vehicle image 91. In the composite image PT12, the approaching vehicle image 101 is closer to the vehicle image 91 than the composite image PT11. The user of the vehicle can accurately grasp the position of the approaching vehicle by displaying such a composite image.

  The composite image PT13 in FIG. 14 is a composite image viewed from the virtual viewpoint VP13, and an image including the vehicle image 91 and the approaching vehicle image 101 that exists at a side position of the vehicle image 91 is displayed. The composite image PT14 is a composite image viewed from the virtual viewpoint VP14, and an image in a state where the approaching vehicle image 101 overtakes the vehicle image 91 is displayed. Thus, by displaying the position of the approaching vehicle with respect to the vehicle 9 in time series, the user can accurately grasp the position of the approaching vehicle.

  <Third Embodiment>

  Next, a third embodiment will be described. The difference between the third embodiment and the first embodiment is that when the operation detection unit 114 shown in FIG. 15 detects an operation of the user of the vehicle 9 associated with the change of the travel lane of the vehicle 9, the vehicle 9 The composite image generation unit 32 generates a composite image viewed from a virtual viewpoint according to the position of the approaching vehicle 10.

  Further, in the third embodiment, a warning image indicating a determination result of whether or not the lane of the vehicle 9 can be changed is superimposed on the composite image and displayed on the display 24 of the display device 20 in the first embodiment. This is different from the contents explained. Other configurations and processes are the same as those in the first embodiment. For this reason, description of parts having the same configuration and processing is omitted.

<3-1. System configuration>

  FIG. 15 is a block diagram of the image display system 120a. The control unit 1a of the image processing apparatus 11a includes an operation detection unit 114, a prediction unit 115, a calculation unit 116, in addition to the image control unit 111, the detection unit 112, and the derivation unit 113 described in the first embodiment. And the judgment part 117 is provided.

  The operation detection unit 114 detects a user operation associated with a change in the travel lane of the vehicle 9. For example, direction indication based on the user's operation of the blinker switch, that is, an ON state of a turn signal indicating a direction indication intended by the user from the direction indicator 83, and a turn signal when the turn signal switch is in a neutral position Detects the off state. As another example, the steering angle sensor 84 detects a steering wheel rotation direction, rotation angle, and rotation speed signal operated by the user.

  The prediction unit 115 predicts the movement position when the vehicle 9 moves while changing the travel lane. Specifically, the lane change time 42a, which is information on the time when the vehicle 9 changes the travel lane by the user's operation, is stored in advance in the nonvolatile memory 4a. Then, when predicting the movement position of the vehicle 9, the prediction unit 115 reads the lane change time 42 a, and the prediction unit 115 predicts the movement position based on the lane change time 42 a and the traveling speed of the vehicle 9.

  Here, the time for changing the travel lane of the vehicle 9 is the time from the start of the change of the travel lane of the vehicle 9 to the end of the change of the travel lane of the vehicle 9 in accordance with the user's operation. For example, after the operation detection unit 114 detects that the turn signal is turned on when the user operates the direction indicator 83, the rotation direction of the steering wheel is changed to the direction in which the lane of the vehicle 9 is changed, and then the steering wheel A predetermined time (for example, 5 seconds) until the rotation direction is changed to a direction in which the vehicle 9 moves straight (a state in which the rotation angle of the steering wheel is approximately 0 degrees, that is, a state in which the steering wheel is in a substantially neutral position). It is.

  The calculating part 116 calculates the time for the approaching vehicle 10 to move to the moving position where the vehicle 9 moves by changing the traveling lane. For example, the calculation unit 116 calculates information on the traveling speed of the approaching vehicle 10 from a change in the position of the approaching vehicle image 101 in successive captured images, and calculates the travel time from the information on the traveling speed to the moving position of the approaching vehicle 10. The calculation unit 116 calculates.

  The determination unit 117 determines whether or not the vehicle 9 can be moved to the movement position, that is, whether or not the lane of the vehicle 9 can be changed, from the movement time of the vehicle 9 to the movement position and the movement time of the approaching vehicle 10 to the movement position. To do. When the determination unit 117 determines that it is difficult to move the vehicle 9 to the movement position, the determination unit 117 generates an instruction signal for generating a composite image in which a warning image described later is superimposed on the composite image generation unit 32. Output.

If the determination unit 117 determines that the vehicle 9 can move to the movement position, the determination unit 117 may not output an instruction signal for displaying a warning image to the composite image generation unit 32. Then, the determination unit 117 may output an instruction signal for displaying an image other than the warning image. Here, the images other than the warning image include, for example, an image indicating the movement position of the vehicle 9 (hereinafter also referred to as “movement position image”) and an image indicating the approaching state of the approaching vehicle 10 (hereinafter referred to as “approaching state”). At least one image.
The lane change time 42a of the non-volatile memory 4a is a time from when the vehicle 9 starts changing the travel lane to when the change of the travel lane ends in accordance with the operation of the user of the vehicle 9.

<3-2. Display image>

  FIG. 16 is a view showing a composite image viewed from the virtual viewpoint displayed on the display 24. The composite image PT1a is a composite image viewed from the virtual viewpoint VP1, and the approaching vehicle image 101 is displayed behind the vehicle image 91. A moving position image tr indicating the position of the vehicle 9 after the lane change is displayed on the side of the vehicle image 91 superimposed on the composite image.

  Further, an approaching state image ad1 displayed in the traveling direction of the approaching vehicle image 101 is displayed superimposed on the composite image. The approach state image ad1 shows the amount of movement of the approaching vehicle 10 corresponding to the approaching vehicle image 101 during the time when the vehicle 9 corresponding to the vehicle image 91 moves to the movement position. And in the approaching state to the moving position of the approaching vehicle image 101 shown in the composite image PT1a, the vehicle 9 corresponding to the vehicle image 91 can change the lane in which it travels. Thus, the user can clearly grasp whether or not the lane change with respect to the moving position of the vehicle 9 is possible.

  The composite image PT2a is a composite image generated based on a photographed image photographed at a timing different from that of PT1a. The composite image PT2a is a composite image viewed from the virtual viewpoint VP2, and the approaching vehicle image 101 is displayed behind the vehicle image 91. In the composite image PT2a, the approaching vehicle image 101 corresponding to the approaching vehicle 10 is closer to the vehicle image 91 corresponding to the vehicle 9 than the composite image PT1a, and represents a state in which it is difficult to change the lane of the vehicle 9. .

  Specifically, on the side of the vehicle image 91, a moving position image tra indicating the position of the vehicle 9 after the lane change is displayed superimposed on the composite image. The display mode of this tra is changed compared to the movement position image tr shown in the composite image PT1a, and a warning image indicating that the movement of the vehicle 9 corresponding to the vehicle image 91 is difficult is superimposed. ing. That is, a warning image indicating that there is a possibility of contact with the approaching vehicle 10 when the vehicle 9 moves to the movement position of the vehicle 9 (for example, a color indicating a warning state for a range corresponding to the movement position) Are superimposed. Thereby, the user can grasp | ascertain clearly whether the lane change with respect to the movement position of the vehicle 9 is possible.

  Further, a part of the approaching state image ad2 displayed in the traveling direction of the approaching vehicle image 101 is projected and superimposed on the composite image so as to be included in the range of the movement position image tra. This indicates the amount of movement of the approaching vehicle 10 corresponding to the approaching vehicle image 101 during the time when the vehicle 9 corresponding to the vehicle image 91 moves to the movement position, and when the vehicle 9 changes the lane to the movement position. A warning image indicating that there is a risk of coming into contact with the approaching vehicle 10 is displayed superimposed on the composite image.

 Further, a warning line image LN indicating that it is difficult to change the traveling lane of the vehicle 9 corresponding to the vehicle image 91 on the lane marking at a substantially central position between the vehicle image 91 and the approaching vehicle image 101 is synthesized as a warning image. It is superimposed on the image. As a result, the user of the vehicle 9 can accurately grasp the approaching state of the approaching vehicle 10 and easily determine whether or not the lane of the vehicle can be changed.

  Other combinations of warning images shown in the composite image PT2a (for example, only the warning line image LN is superimposed and projected, or only the moving position image tra and the approach state image ad2 are projected and projected). Is possible.

  In the third embodiment, when the operation detection unit 114 detects the user's operation of the vehicle 9 associated with the change of the travel lane of the vehicle 9, a warning image is displayed on the composite image according to the position of the approaching vehicle 10 with respect to the vehicle 9. The process of superimposing and displaying on the display 24 has been described. On the other hand, when the operation detection unit 114 detects a user operation associated with a change in the travel lane of the vehicle 9, a composite image (described in the first embodiment) is displayed without displaying a superimposed warning image. Only a composite image viewed from a virtual viewpoint according to the positions of the vehicle 9 and the approaching vehicle 10 may be displayed on the display 24. Accordingly, the user can accurately grasp the position of the approaching vehicle 10 and easily determine whether or not the lane of the vehicle 9 can be changed.

  <3-3. Processing flowchart>

  17 and 18 are process flowcharts of the image processing apparatus 11a. In step S201, when the operation detection unit 114 detects a user operation associated with a change in the travel lane of the vehicle 9 (Yes in step S201), the process proceeds to step S101, and then the process from step S101 to step S104 is performed. Is called. In step S104, a composite image viewed from the virtual viewpoint VP is generated, and the process proceeds to step S202. In addition, in the process of step S201, when the operation detection part 114 has not detected the user's operation accompanying the change of the lane in which the vehicle 9 travels (step S201 is No), the process ends.

  In step S202, the prediction unit 115 derives the movement position associated with the change of the travel lane of the vehicle 9 (step S202), and the process proceeds to step S203.

  In step S203, the calculation part 116 calculates the movement time to the movement position of the approaching vehicle 10 (step S203), and proceeds to the process of step S204.

  In step S204, when the movement time of the vehicle 9 to the movement position is longer than the movement time of the approaching vehicle 10 to the movement position (Yes in step S204), the determination unit 117 displays a warning image on the composite image generation unit 32. Is output (step S205), and the process proceeds to step S206. If the moving time of the vehicle 9 to the moving position is shorter than the moving time of the approaching vehicle 10 to the moving position (No in step S204), the process proceeds to step S105, and the composite image generated in step S104 is displayed. The image output unit 33 outputs to the display device 20 (step S105).

  In step S206, the composite image generation unit 32 superimposes the warning image on the composite image. Then, the composite image on which the warning image is superimposed is output to the display device 20 by the image output unit 33 (step S206).

  <Fourth embodiment>

  Next, a fourth embodiment will be described. The difference between the fourth embodiment and the third embodiment is that the user's own lane change time (for example, the user's lane change of the vehicle 9 is changed from the lane change time in the lane change of the user of the vehicle 9 multiple times). The average time) is learned, and the time is stored in the nonvolatile memory 4b. That is, learning means that the learning unit 118 derives the user's unique lane change time from the lane change time in the lane change of the user of the vehicle 9 multiple times.

  In addition, storing means storing the lane change time unique to the user of the vehicle 9 derived by the learning unit 118 as the lane change time 42b of the nonvolatile memory 4b. Thereby, when the judgment part 117 judges whether the movement to the movement position of the vehicle 9 is possible, it becomes possible to judge whether the lane change according to the operation time of the vehicle at the time of the lane change for each user. Other configurations and processes are the same as those of the third embodiment. For this reason, description of parts having the same configuration and processing is omitted.

  <4-1. System configuration>

  FIG. 19 is a block diagram of the image display system 120b. The control unit 1b of the image processing apparatus 11b includes a learning unit 118 in addition to the configuration described in the third embodiment. The learning unit 118 adds the time required for multiple lane changes from the start of the change of the travel lane of the vehicle 9 to the end of the change of the travel lane of the vehicle 9 in accordance with the user's operation, and obtains the user's unique lane change time. measure.

The lane change time 42b of the nonvolatile memory 4b stores information on the lane change time derived by the learning unit 118. That is, the non-volatile memory 4b stores the lane change time 42b, which is a value obtained as a result of learning the user's lane change operation, instead of the constant lane change information.
The determination unit 117 determines whether or not the vehicle 9 can move to the movement position based on the time when the approaching vehicle 10 moves to the movement position and the lane change time 42b that is the user's unique lane change time. Thereby, the user can easily determine whether or not the vehicle 9 can move to the moving position according to the user's unique lane change time.

  <4-2. Lane change time graph>

FIG. 20 is a graph showing an example of measurement of the lane change time of the vehicle 9. When the user operates the direction indicator 83 at time t1, the learning unit 118 of the control unit 1b starts learning the lane change time because the operation detection unit 14 detects the turn signal ON state.
Next, at the time t <b> 2 when the turn signal of the direction indicator 83 is in the on state, the rotation angle of the steering wheel in the same direction as the direction in which the traveling lane of the vehicle 9 is changed is detected by the operation detection unit 14. Then, when the inclination of the steering angle graph of the steering wheel is lowered to the right at time t3, the turn signal is turned off. Thereafter, when the steering angle of the steering wheel becomes approximately 0 degrees (the steering wheel is in the neutral position), the learning unit 118 learns the time from time t1 to t4 as one lane change time of the user of the vehicle 9 To do. And the said lane change time is memorize | stored in the non-volatile memory 4b as time which the user required for one lane change.

  Thereafter, each time the lane of the vehicle 9 is changed, the learning unit 118 adds up the lane change time, and divides the accumulated time by the number of lane changes, so that it is non-volatile as the lane change time 42b that is the user's unique lane change time. Stored in the memory 4b. The lane change time 42b is read from the nonvolatile memory 4b when the prediction unit 115 predicts the movement position and when the determination unit 117 determines whether the vehicle 9 can move to the movement position.

  <4-3. Processing flowchart>

21 and 22 are process flowcharts in which the learning unit 118 measures the lane change time. In step 401, when the user operates the direction indicator 83 and the turn signal is turned on (Yes in step S401), the process proceeds to step S402. If the user is not operating the direction indicator 83 (No in step S401), the process ends.
In step S402, when the turn signal is turned on, the direction indicator flag of the control unit 1 is turned on (step S402). In step 403, the learning unit 118 starts measuring the lane change time ( The process proceeds to step S403) and step S404.

In step S404, when the steering wheel is steered to the side in which the direction indicator 83 is operated in one of the left and right directions of the direction indicator 83 (Yes in step S404), the process proceeds to step S405. For example, when the direction indicator 83 is operated to the right side, the steering wheel is steered to the right side to move the vehicle 9 to a traveling lane adjacent to the right side of the lane in which the vehicle 9 travels.
When the steering wheel is not steered to the side where the direction indicator 83 is operated in either the left or right direction of the direction indicator 83, that is, the steering wheel is steered to the side opposite to the side where the direction indicator 83 is operated. In the case (No in step S404), the process proceeds to step S406.

  In step S405, the steering flag of the steering wheel corresponding to the side on which the direction indicator 83 is operated is turned on (step S405), and the process proceeds to step S406.

  In step S406, when the steering wheel is steered to the side opposite to the side on which the direction indicator 83 is operated (step S406 is Yes), the process proceeds to step S407. For example, when the direction indicator 83 is operated to the right side, the steering wheel is operated to the right side, and then the steering wheel is turned back to the opposite side (left side). When the steering wheel is not steered to the side opposite to the side on which the direction indicator 83 is operated (No at Step S406), the process proceeds to Step S408.

  In step S407, the opposite side steering flag corresponding to the side opposite to the side on which the direction indicator 83 is operated is turned on, and the process proceeds to step S408.

  In step S408, when the steering flag and the opposite side steering flag are on (step S408 is Yes), the process proceeds to step S409. If the steering flag and the opposite side steering flag are not in the on state (No in step S408), the process ends.

  In step S409, if the steering angle of the steering wheel is approximately 0 degrees (step S409 is Yes), the process proceeds to step S410. Note that if the steering angle of the steering wheel is not approximately 0 degrees (No in step S409), the process ends.

  In step S410, the learning unit 118 derives the user's unique lane change time (step S410), and the process proceeds to step S411. In step S411, each flag is turned off (step S411). By repeatedly performing such processing every time the lane is changed, it is possible to accurately determine whether the vehicle 9 can move to the moving position according to the lane change time of the user.

  <Fifth embodiment>

  Next, a fifth embodiment will be described. The difference between the fifth embodiment and the third embodiment is that the viewpoint position of the virtual viewpoint is the position of the driver of the approaching vehicle 10 and the viewing direction is the direction in which the vehicle 9 exists. Other configurations and processes are the same as those of the third embodiment. For this reason, description of parts having the same configuration and processing is omitted.

  <5-1. Viewpoint and view direction of virtual viewpoint>

  FIG. 23 is a diagram for mainly explaining the viewpoint position of the virtual viewpoint VP31. As shown in FIG. 23, the position of the driver of the approaching vehicle 10 is the viewpoint position of the virtual viewpoint VP31. Specifically, the position at which the driver of the vehicle 10 is seated (the position of the driver's seat) is the viewpoint position. Although the derivation unit 113 derives the position of the driver of the approaching vehicle 10, detailed description will be given with reference to FIG.

  FIG. 24 is a diagram for mainly explaining the visual field direction of the virtual viewpoint VP31. The viewing direction of the virtual viewpoint VP31 shown in FIG. 24 is the direction in which the vehicle 9 exists. That is, the direction is substantially the same as the traveling direction (−Y direction) of the approaching vehicle 10 viewed from the position of the driver of the approaching vehicle 10.

  Here, the position of the approaching vehicle 10 with respect to the vehicle 9 that is the position of the driver of the approaching vehicle 10 is derived from the position in the image of the approaching vehicle image 101 in the composite image from the center of gravity position CP (for example, The deriving unit 113 derives a straight line CL connecting to the position of the front bumper of the approaching vehicle 10. Then, the deriving unit 113 derives a straight line CLa obtained by adding the correction value cl to the straight line CL. The correction value cl is a distance from the position of the approaching vehicle 10 (for example, the position of the front bumper of the approaching vehicle 10) to the seating position UP that is the position of the driver of the approaching vehicle 10.

  In addition to this, the position of the driver of the approaching vehicle 10 may be derived as follows. The deriving unit 113 derives the position in the front-rear direction (Y-axis direction) of the approaching vehicle 10 with respect to the vehicle 9 shown in FIG. 24 as the position obtained by adding the correction value al to the straight line AL shown in FIG. 5 of the first embodiment. . Further, the deriving unit 113 derives the position in the lateral direction (X-axis direction) of the approaching vehicle 10 with respect to the vehicle 9 as the position obtained by adding the correction value bl to the straight line BL. Then, the deriving unit 113 derives a straight line CLa that connects the center of gravity position CP of the vehicle 9 to the seating position UP by using, for example, the three square theorem.

  Then, the seating position UP becomes the position of the driver of the approaching vehicle 10, and this position becomes the viewpoint position of the virtual viewpoint VP31. Further, the direction substantially the same as the traveling direction (−Y direction) of the approaching vehicle 10 is the visual field direction of the virtual viewpoint VP31.

  Each correction value such as the correction value cl is recorded in the non-volatile memory 4a, and is, for example, a correction value used when deriving a straight line from the position of the front bumper portion of the approaching vehicle 10 to the seating position UP. .

  <5-2. Display image>

  FIG. 25 is a diagram illustrating a composite image viewed from the virtual viewpoint VP31 displayed on the display 24. As illustrated in FIG. The composite image PT31 is a composite image viewed from the virtual viewpoint VP31, and the vehicle image 91 is projected forward. In other words, a composite image viewed from the visual field direction where the vehicle 9 exists is displayed on the display 24 at the viewpoint position of the driver of the approaching vehicle 10. Thus, the user can accurately grasp the position of the vehicle 9 from the viewpoint of the driver of the approaching vehicle 10. In addition, the one part image of the approaching vehicle 10 is an image figure previously stored in the non-volatile memory 4b.

  In addition, in the composite image PT31, the movement position image tra is projected forward on the composite image, and a part of the approach state image ad2 displayed in the traveling direction of the approaching vehicle 10 is part of the movement position image tra. The image is superimposed on the composite image so as to be included in the range. Further, the warning line image LN is projected and superimposed on the composite image as a warning image. In such a case, the user can clearly determine that it is difficult to change the lane of the vehicle 9.

<5-3. Processing flowchart>

  26 and 27 are process flowcharts of the image processing apparatus 11a. In step S102a, when the detecting unit 112 detects the presence of the approaching vehicle 10 in step S101 of the previous process, the deriving unit 113 derives the position of the driver of the approaching vehicle 10, and the process proceeds to step S103a.

In step S103a, the viewpoint position of the virtual viewpoint is set as the driver position of the approaching vehicle 10, the viewing direction of the virtual viewpoint is set as the direction in which the vehicle 9 exists (step S103a), and the process proceeds to step S104. Thereafter, the processing described in the third embodiment is performed.

<Sixth Embodiment>

  Next, a sixth embodiment will be described. The difference between the sixth embodiment and the third embodiment is that, when the vehicle 9 changes lanes by a user operation, the composite image is displayed when the detection unit 112 does not detect the presence of the approaching vehicle 10. It is the point which explained about. In such a case, a composite image viewed from the virtual viewpoint VP in which the viewpoint position is in front of the vehicle 9 and the viewing direction is behind the vehicle 9 is displayed on the display 24.

<6-1. Display image>

  FIG. 28 is a diagram showing a composite image viewed from the virtual viewpoint VP displayed on the display 24. A composite image including the vehicle image 91 is displayed on the composite image PT1b. The composite image PT1b is when the detection unit 112 does not detect the presence of the approaching vehicle 10 when the operation detection unit 114 detects the operation of the user of the vehicle 9 (the approaching vehicle 10 does not exist behind the vehicle 9). The composite image generation unit 32 generates a composite image viewed from the following virtual viewpoint. That is, the composite image generation unit 32 includes a composite image viewed from the virtual viewpoint VP including at least a part of the vehicle image 91, the viewpoint position in front of the vehicle 9, and the visual field direction in the rear of the vehicle 9. Accordingly, the user can easily determine whether or not the lane of the vehicle 9 can be changed regardless of the presence or absence of the approaching vehicle 10.

<6-2. Processing flowchart>

29 and 30 are processing flowcharts of the image processing apparatus 11a. In step S101, when the detection unit 112 does not detect the presence of the approaching vehicle 10 (No in step S101), the process proceeds to step S301.
In step S301, the image control unit 111 sets the viewpoint position and viewing direction of the virtual viewpoint VP (step S301). That is, the image control unit 111 sets a virtual viewpoint VP that includes at least a part of the vehicle image 91, the viewpoint position is the front of the vehicle 9, and the visual field direction is the rear of the vehicle 9. Then, the composite image generation unit 32 generates a composite image viewed from the virtual viewpoint VP (step S104). Thereafter, the processing described in the third embodiment is performed.

<Seventh embodiment>

  Next, a seventh embodiment will be described. The difference between the seventh embodiment and the sixth embodiment is that when the vehicle 9 changes lanes by a user operation, the viewpoint position is a position near the door mirror of the vehicle 9 and the visual field direction is that of the vehicle 9. That is, the composite image generation unit 32 generates a composite image viewed from the virtual viewpoint VP41 that is behind.

<7-1. Viewpoint and view direction of virtual viewpoint>

  FIG. 31 is a diagram for mainly explaining the viewpoint position of the virtual viewpoint VP41. As shown in FIG. 31, the viewpoint position of the virtual viewpoint VP41 is a position near the door mirror of the vehicle 9. Note that FIG. 31 illustrates the case where the vehicle 9 changes the driving lane in the right (+ X direction) lane, so the viewpoint position of the virtual viewpoint VP41 is set in the vicinity of the right (+ X direction) drag mirror. When the vehicle 9 changes the traveling lane on the left side (−X direction), the viewpoint position of the virtual viewpoint may be set near the door mirror on the left side (−X direction).

  FIG. 32 is a diagram for mainly explaining the viewing direction of the virtual viewpoint VP41. A virtual viewpoint VP41 shown in FIG. 32 has the vicinity of the door mirror of the vehicle 9 as the viewpoint position and the visual field direction as the rear of the vehicle 9 (+ Y direction). That is, the field of view is directed in the direction opposite to the traveling direction of the vehicle 9 (−Y direction).

<7-2. Display image>

FIG. 33 is a diagram showing a composite image viewed from the virtual viewpoint VP41 displayed on the display 24. As shown in FIG. The composite image P41 is a composite image viewed from the virtual viewpoint VP41, and is displayed as a composite image including an image displayed on the right door mirror of the vehicle 9. Thereby, even when there is no approaching vehicle, the user can easily determine whether or not the lane of the vehicle 9 can be changed.
<7-3. Processing flowchart>
34 and 35 are process flowcharts of the image processing apparatus 11a. In step S101, when the detection unit 112 does not detect the presence of the approaching vehicle 10 (No in step S101), the process proceeds to step S401.

  In step S401, the image control unit 111 sets the viewpoint position and the viewing direction of the virtual viewpoint VP41 whose viewing direction is behind the vehicle 9 at the viewing position near the door mirror of the vehicle 9 (step S401). Then, the composite image generation unit 32 generates a composite image viewed from the virtual viewpoint VP41 (step S104). Thereafter, the processing described in the sixth embodiment is performed.

<Eighth Embodiment>

  Next, an eighth embodiment will be described. In the eighth embodiment, a captured image captured by one camera such as the front camera 51 is displayed on the display 24. When the photographed image of one camera is displayed on the display 24 and the presence of the approaching vehicle 10 is detected by the detection unit 112, the image is viewed from a virtual viewpoint corresponding to the positions of the vehicle 9 and the approaching vehicle 10. The combined image is displayed on the display 24 together with the image photographed by one camera.

  Here, the virtual viewpoint according to the positions of the vehicle 9 and the approaching vehicle 10 is, for example, a virtual viewpoint (for example, as shown in FIG. 36) in which the viewpoint position is the position of the driver of the approaching vehicle 10 and the viewing direction is the direction in which the vehicle 9 exists. This is a virtual viewpoint VP52a). Further, for example, it is a virtual viewpoint (for example, virtual viewpoint VP52b shown in FIG. 36) in which the viewpoint position is a position near the door mirror of the vehicle 9 and the visual field direction is the direction in which the approaching vehicle 10 exists. As will be described later, these virtual viewpoints are changed according to a user operation, and the display mode of the composite image displayed on the display 24 is switched by this change. As a result, the user can recognize the presence of the approaching vehicle 10 at an early stage, and can accurately determine the position of the approaching vehicle 10.

<8-1. Viewpoint and view direction of virtual viewpoint>
FIG. 36 shows an example of a virtual viewpoint of a composite image displayed on the display 24 together with a photographed image of one camera. Specifically, the viewpoint position is the position of the driver of the approaching vehicle 10, the viewing direction is the direction in which the vehicle 9 exists, the virtual viewpoint VP 52 a is the position near the door mirror of the vehicle 9, and the viewing direction is the approaching vehicle 10. A virtual viewpoint VP52b with the existing direction is shown. And these virtual viewpoints are changed according to a user's operation, and the display mode of the composite image displayed on the display 24 is switched by this change.

  <8-2. Display image>

  37 and 38 are diagrams showing images displayed on the display 24. FIG. Note that the images PT51, PT51a, and PT51b in FIGS. 37 and 38 are temporally continuous images. The image PT51 is a captured image taken only by the front camera provided in the vehicle 9, and is a vehicle image 102 of a forward vehicle traveling in front of the vehicle 9 (in the -Y direction) (hereinafter also referred to as "front vehicle image 102"). Is displayed on the display 24. A composite image as described later is not superimposed on the image PT51. That is, the presence of the approaching vehicle 10 approaching from the rear of the vehicle 9 is not detected by the detection unit 112.

  Next, the image PT51a shows an image displayed on the display 24 when the detecting unit 112 detects the presence of the approaching vehicle 10. A forward vehicle image 102 is displayed in the image TP51a, and a composite image PT52a viewed from the virtual viewpoint VP52a of the position of the driver of the approaching vehicle 10 is superimposed on the image PT51a and displayed. That is, the composite image PT52a is displayed on the display 24 together with the image PT51a. Thereby, the user can recognize presence of approaching vehicle 10 at an early stage, and can grasp the position of approaching vehicle 10 correctly.

  The viewpoint position of the virtual viewpoint of the composite image PT52 is the position of the driver of the approaching vehicle 10, and the visual field direction is the direction in which the approaching vehicle 10 exists. Since the distance of the approaching vehicle 10 with respect to the vehicle 9 is less than a predetermined distance, the visual field direction including the vehicle image 91 (the direction diagonally forward left when viewed from the driver of the approaching vehicle 10) This is a composite image viewed from the virtual viewpoint.

  Further, the partner viewpoint button B1 and the own vehicle viewpoint button B2 are displayed superimposed on the image PT51a. When the user operates one of the partner viewpoint button B1 and the own vehicle viewpoint button B2, a composite image viewed from the virtual viewpoint corresponding to each button is displayed on the display 24. That is, the composite image is displayed on the display 24 together with the photographed image photographed by the front camera 51.

  Since the partner viewpoint button B1 is selected in the image PT51a, the composite image PT52a viewed from the virtual viewpoint VP52a at the position of the driver of the approaching vehicle 10 is displayed on the display 24 together with the image PT51a.

  Next, in the image PT51b shown in FIG. 38, the own vehicle viewpoint button B2 is selected by the user's operation. In response to the user operation, the composite image displayed on the display 24 together with the image captured by the front camera is switched from the composite image PT52a to the composite image PT52b. Thereby, the composite image viewed from the virtual viewpoint desired by the user is displayed, and the position of the approaching vehicle can be accurately grasped.

  <8-3. Processing flowchart>

  39 and 40 are process flowcharts for displaying a composite image together with one captured image on the display 24. In step S501, an image captured by the front camera 51 is processed by the image processing unit 3 (step S501), and is output from the image output unit 33 to the display device 20 via the communication unit 61 (step S502). Proceed to the process.

  In step S101, when the detection unit 112 detects the presence of the approaching vehicle 10 (step S101 is Yes), the process proceeds to step S503, and a composite image viewed from a virtual viewpoint corresponding to the positions of the vehicle 9 and the approaching vehicle 10 is displayed. The image output unit 33 outputs to the display device 20 via the communication unit 61 (step S503), and the process proceeds to step S504. As a result, a composite image is displayed on the display 24 together with the photographed image photographed by the front camera 51.

  Note that the viewpoint position and the visual field direction of the virtual viewpoint of the composite image are the composites viewed from the viewpoint position and the visual field direction of the virtual viewpoint selected by the user operation when the detection unit detects the approaching vehicle in the past. An image is displayed on the display 24.

  Moreover, when the detection part 112 has not detected presence of the approaching vehicle 10 by step S101 (step S101 is No), a process is complete | finished.

  Returning to step S504, when the partner viewpoint button B1 is selected by the user's operation when the composite image PT52b is displayed on the display 24 together with the image PT51a of the captured image captured by the front camera 51 on the display 24 ( If step S504 is YES, the process proceeds to step S505.

  If the partner viewpoint button B1 is not selected by the user's operation (No in step S504), the process proceeds to step S507. The same process is performed when the partner viewpoint button B1 has already been selected by a past user operation.

  In step S505, the composite image PT52a viewed from the virtual viewpoint VP52a of the position of the driver of the approaching vehicle 10 is generated by the composite image generation unit 32 (step S505), and the process proceeds to step S506.

  In step S506, the composite image PT52a is output to the display device 20 by the image output unit 33 (step S506), and the process proceeds to step S507.

  In step S507, when the composite image PT52a viewed from the virtual viewpoint VP52a is displayed on the display 24 together with the image PT51a of the captured image captured by the front camera 51 on the display 24, the vehicle viewpoint button B2 is operated by the user's operation. Is selected (step S507 is Yes), the process proceeds to step S508.

  If the vehicle viewpoint button B2 is not selected by the user's operation (No in step S507), the process ends. The same processing is also performed when the vehicle viewpoint button B2 has already been selected by a past user operation.

  In step S508, the composite image PT52b viewed from the virtual viewpoint VP52b near the door mirror of the vehicle 9 is generated by the composite image generation unit 32 (step S508), and the process proceeds to step S509.

  In step S509, the composite image PT52b is output to the display device 20 by the image output unit 33 (step S509).

<Ninth embodiment>

  Next, a ninth embodiment will be described. In the ninth embodiment, the size of the image captured by the front camera 51 of the vehicle 9 described in the eighth embodiment and the size of the composite image are switched by a user operation and displayed on the display 24.

<9-1. Display image>

  FIG. 41 is a diagram illustrating switching between the size of the captured image of the front camera 51 and the size of the composite image. 41 shows a state in which the composite image PT52c viewed from the virtual viewpoint corresponding to the positions of the vehicle 9 and the approaching vehicle 10 is displayed on the display 24 together with the image PT51c photographed by the front camera 51. . Note that the image PT51c is displayed on the display 24 as an image that is substantially the same size as the screen of the display 24, and the composite image PT52c is displayed on the display 24 as approximately 1/10 the size of the image PT51c.

  It should be noted that an image switching button B3 for switching between the size of one captured image and the size of the composite image by the user's operation and displaying it on the display 24 is displayed superimposed on the image PT51c.

  In the lower part of FIG. 41, when the image switching button B3 is operated by the user, the size of the composite image PT52c is set to be approximately the same as the image PT51c before the user operates the image switching button B3. That is, it is displayed on the display 24 as a composite image PT52d that is larger than the size of the composite image PT52c before switching.

  In addition, with such image switching, the size of the image PT51c is set to be approximately the same size as the synthesized image PT52c before the user operates the image switching button B3 of the vehicle 9. That is, the image PT51d that is reduced in size than the image PT51c before switching is displayed on the display 24. Thereby, according to the driving | running | working state of the vehicle 9, the size of the image which a user desires can be adjusted compared with the size of another image.

<Tenth Embodiment>

  Next, a tenth embodiment will be described. In the tenth embodiment, when the vehicle 9 travels using the navigation function of the display device 20, the vehicle 9 is viewed from the virtual viewpoint according to the notification of the guidance information from the display device 20 to the user. A composite image is provided to the user.

  The purpose of this embodiment is as follows. When the user is driving the vehicle 9 along the route guidance to the destination on the navigation screen displayed on the display device 20, another image (from the navigation image at a point where it is not necessary to change the course of the vehicle 9). For example, the display device 20 image may be switched to a composite image viewed from a virtual viewpoint. In such a case, the user may not be able to see the navigation image that the user originally wants to see, and may feel annoyed. Therefore, in the following tenth embodiment, a configuration will be described in which a composite image viewed from a virtual viewpoint is displayed on the display device 20 at a timing required for the user to operate the vehicle 9.

<10-1. Guidance information point>

  FIG. 42 is a diagram illustrating points where the guide information is notified to the user. In FIG. 42, the vehicle 9 is traveling in the left lane LA of the road, and the approaching vehicle 10 that is located behind the vehicle 9 is traveling in the right lane RB divided by the lane marking. In addition, a route RU that is a route that should be traveled to reach the destination when the destination to which the vehicle 9 arrives is set in the display device 20 by the operation of the user of the vehicle 9 is shown.

  Next, when the vehicle 9 travels along the route RU, when the vehicle 9 approaches the point PN1 on the left lane LA, guidance information indicating that the position of the vehicle 9 is changed from the left lane LA to the right lane RB. Is notified from the display device 20 to the user. When such guidance information is notified, a composite image viewed from a virtual viewpoint (for example, a composite image viewed from a virtual viewpoint in the viewing direction of the vehicle 9 at the viewpoint position of the driver of the approaching vehicle 10) Is generated by the composite image generation unit 32. The generated composite image is displayed on the display 24. In this way, the user can confirm the composite image viewed from the virtual viewpoint position corresponding to the guidance information notified from the display device 20.

  When the vehicle 9 approaches the point PN2 on the intersection CR, the display device 20 informs the user of guidance information that the vehicle 9 is turned right at the intersection CR and moved from the right lane RB to the lane MC. Is done. When such guidance information is notified, the viewpoint position is the front of the vehicle 9 so as to include at least a part of the vehicle image 91 corresponding to the vehicle 9 when viewed from the virtual viewpoint, and the field of view A composite image viewed from a virtual viewpoint with the direction behind the vehicle 9) is generated by the composite image generation unit 32. Thereby, the user can confirm the composite image according to the change of the travel lane of the vehicle 9 and the change of the traveling direction at the intersection.

  The case where the vehicle 9 approaches the point where the guidance information is notified refers to, for example, a case where the distance of the straight line between the vehicle 9 and the point where the guidance information is notified becomes 50 m, but other distances may be used. .

<10-2. Virtual viewpoint position and viewing direction>

  FIG. 43 is a diagram illustrating an example of the viewpoint position and visual field direction of the virtual viewpoint at a point where the guidance information is notified. The left diagram in FIG. 43 shows a virtual image in which a composite image is generated when the user is informed of a guidance for changing the lane from the left lane LA to the right lane RB when the vehicle 9 approaches the point PN1. It shows the viewpoint position and viewing direction of the viewpoint VP61. Specifically, the virtual viewpoint VP61 is a viewpoint position of the driver of the approaching vehicle 10 and is a virtual viewpoint in the visual field direction where the vehicle 9 exists, and a composite image viewed from the virtual viewpoint VP61 is generated by the composite image generation unit 32. .

  43, when the vehicle 9 approaches the point PN2, when the user is notified of guidance information that the vehicle 9 turns right from the right lane RB at the intersection CR and moves to the lane MC. It shows the viewpoint position and visual field direction of the virtual viewpoint VP62 where the composite image is generated. Specifically, the virtual viewpoint VP62 is a virtual viewpoint in which the viewpoint position is the front of the vehicle 9 and the visual field direction is the rear of the vehicle 9 so as to include at least a part of the vehicle image 91 corresponding to the vehicle 9. Then, the composite image viewed from the virtual viewpoint VP62 is generated by the composite image generation unit 32.

  <10-3. Display image>

  FIG. 44 is a diagram illustrating a composite image viewed from the virtual viewpoints VP61 and VP62. The composite image PT61 is a composite image displayed on the display 24 when the user is notified of the lane change guidance information of the vehicle 9. Specifically, it is a composite image viewed from the virtual viewpoint of the driver of the approaching vehicle 10, and the vehicle image 91 a of the vehicle 9 traveling in front of the approaching vehicle 10 is displayed on the display 24. In addition, a movement position image tr that is the movement position of the vehicle 9 when the vehicle 9 changes lanes from the left lane LA to the right lane RB is displayed on the display 24. Further, an approaching state image ad1 indicating the traveling direction of the approaching vehicle image 101 is displayed superimposed on the composite image.

  The composite image PT62 is a composite image that is displayed on the display 24 when the direction change guidance information at the intersection of the vehicle 9 is notified to the user. Specifically, the virtual viewpoint VP62 is a composite image viewed from the virtual viewpoint behind the vehicle 9 so as to include at least a part of the vehicle image 91 corresponding to the vehicle 9, and the vehicle image 91b corresponding to the vehicle 9 is displayed. The image including it is projected. Thus, the virtual viewpoint of the composite image is changed according to the content of the guidance information notified to the user. Thereby, the user can confirm the composite image according to the driving | running | working condition of the vehicle corresponding to the content of guidance information.

In the composite image PT62, the approaching vehicle image 101 corresponding to the approaching vehicle 10 is not shown in the image. However, when the approaching vehicle 10 is approaching the vehicle 9, the approaching vehicle image 101 corresponding to the approaching vehicle is displayed. You may make it project the composite image containing.
<10-4. Processing flowchart>
45 and 46 are processing flowcharts in which the image processing device 11a displays a composite image on the display 24 when the display device 20 notifies the guidance information. In step S601, the image processing device 11a acquires a photographed image around the vehicle 9 photographed by the in-vehicle camera provided in the vehicle 9 (step S601), and the process proceeds to step S602.

  In step S602, when the vehicle 9 travels along the route RU to the destination, the image processing device 11a acquires guidance information to be notified to the user from the display device 20 (step S602), and the processing in step S603 is performed. move on.

  In step S603, when the image processing apparatus 11a acquires the first guidance information to the user (for example, guidance information corresponding to the change of the travel lane of the vehicle 9) from the display device 20 via the communication unit 23 (step S603). Yes), the process proceeds to step S604. If the image processing apparatus 11a has not acquired the first guidance information for the user from the display device 20 via the communication unit 23 (No in step S603), the process ends.

  In step S604, when the presence of the approaching vehicle 10 approaching from the rear of the vehicle 9 is detected by the back camera 52 provided in the vehicle 9 (step S604 is Yes), the process proceeds to step S605.

  In step S605, since the approaching vehicle 10 exists, the first virtual viewpoint (for example, the virtual viewpoint VP61 in the visual field direction where the vehicle 9 exists at the viewpoint position of the driver of the approaching vehicle 10) is set by the image processing apparatus 11a (step S605). S605), the process proceeds to step S607.

  Returning to step S604, if the presence of the approaching vehicle 10 approaching from the rear of the vehicle 9 is not detected by the detection unit 112 based on the image captured by the back camera 52 (step S604 is No), the process proceeds to step S606. .

  In step S606, since the approaching vehicle 10 does not exist, the second virtual viewpoint (for example, the viewpoint position is the front of the vehicle 9 so as to include at least a part of the vehicle image 91 corresponding to the vehicle 9, and the viewing direction is the vehicle 9). The rear virtual viewpoint VP62) is set by the image processing apparatus 11a (step S606), and the process proceeds to step S607.

  In step S607, the composite image generation unit 32 generates a composite image viewed from the virtual viewpoint set by the image processing apparatus 11a (step S607), and the composite image is output to the display device 20 via the communication unit 61 (step S607). S608), the process proceeds to step S609.

  In step S609, when the image processing apparatus 11a acquires second guidance information for the user (for example, guidance information corresponding to a change in the traveling direction of the vehicle 9 turning right at the intersection) from the display device 20 via the communication unit 23. In step S609, the process proceeds to step S610. If the image processing apparatus 11a has not acquired the second guidance information for the user from the display device 20 via the communication unit 23 (No in step S609), the process ends.

  In step S610, when the presence of the approaching vehicle 10 approaching from the rear of the vehicle 9 is detected by the image captured by the back camera 52 provided in the vehicle 9 (step S610 is Yes), the process proceeds to step S611. Proceed to

  In step S611, since the approaching vehicle 10 exists, the first virtual viewpoint (for example, the virtual viewpoint VP61 in the visual field direction where the vehicle 9 exists at the viewpoint position of the driver of the approaching vehicle 10) is set by the image processing device 11a (step S611). S611), the process proceeds to step S613.

  Returning to step S610, if the detection unit 112 has not detected the presence of the approaching vehicle 10 approaching from the rear of the vehicle 9 by the back camera 52 (No in step S610), the process proceeds to step S612.

  In step S612, since the approaching vehicle 10 does not exist, the second virtual viewpoint (for example, the viewpoint position is the front of the vehicle 9 so as to include at least a part of the vehicle image 91 corresponding to the vehicle 9, and the visual field direction is the vehicle 9). The rear virtual viewpoint VP62) is set by the image processing apparatus 11a (step S612), and the process proceeds to step S613.

  In step S613, the composite image generation unit 32 generates a composite image viewed from the virtual viewpoint set by the image processing device 11a (step S613), and the composite image is output to the display device 20 via the communication unit 61 (step S613). Step S614).

  <Modification>

  Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications are possible. Below, such a modification is demonstrated. In addition, all the forms including the form demonstrated in the said embodiment and the form demonstrated below are combinable suitably.

  In the above embodiment, the approaching vehicle 10 is a vehicle that travels behind the vehicle 9 and is a vehicle that travels behind the vehicle 9 in addition to a vehicle that travels in the lane adjacent to the lane in which the vehicle 9 travels. And the vehicle which drive | works the same lane as the lane which the vehicle 9 drive | works is also included.

  In the above embodiment, the approaching vehicle 10 includes a vehicle that travels on the side of the vehicle 9 in addition to a vehicle that travels behind the vehicle 9.

  In the above embodiment, it has been described that the virtual viewpoint information 41 is recorded in the nonvolatile memory 4 in advance and the image processing unit 3 reads the virtual viewpoint information VP when the viewpoint position and the visual field direction of the virtual viewpoint VP are set. In addition to this, when setting the viewpoint position and viewing direction of the virtual viewpoint VP, the image control unit 111 calculates and derives the viewpoint position and viewing direction of the virtual viewpoint based on the position of the approaching vehicle 10 with respect to the vehicle 9. It may be.

  Moreover, in the said embodiment, when the detection part 112 detects presence of the approaching vehicle 10 from the back of the vehicle 9, from the picked-up image of the back camera 52 with which the back of the vehicle 9 was equipped, the vehicle 9 in an image It has been described that the approaching vehicle 10 existing behind is detected. In addition, a camera other than the back camera may be used as a camera for detecting the presence of the approaching vehicle 10.

  Further, in the above embodiment, the detection of the presence of the approaching vehicle 10 and the acquisition of the position information of the approaching vehicle 10 relative to the vehicle 9 are reflected from the object among the transmitted waves transmitted to the vehicle 9 outside the vehicle 9. The vehicle 9 may be provided with a device such as a radar device that receives information on the distance, angle, and relative speed of the object from the vehicle 9 by receiving waves.

  In the above embodiment, the lane where the approaching vehicle 10 exists is described as the lane adjacent to the right side (+ X direction) where the vehicle 9 travels, but the lane adjacent to the left side (−X direction) where the vehicle 9 travels. Even when the approaching vehicle 10 exists, the contents of the above embodiment can be similarly applied.

  In the above embodiment, the case where the vehicle image 91 and the approaching vehicle image 101 are included in the composite image may be not only the entire image of the vehicle image 91 and the approaching vehicle image 101 but also a part of the images.

  In the above-described embodiment, the position of at least one of the vehicle image 91 and the approaching vehicle image 101 in the composite image displayed on the display 24 may be set as a mirrored position.

  Further, in the above embodiment, when there is an approaching vehicle approaching from the left and right directions of the vehicle 9, the value of the straight line with the vehicle 9 in the front-rear direction (Y-axis direction) is small (the vehicle 9 is closest). The deriving unit 113 derives the approaching vehicle 10. Then, the position of the approximate center of the straight line in the lateral direction (X-axis direction) connecting the approaching vehicle and the vehicle 9 may be the viewpoint position of the virtual viewpoint VP, and the rear (+ Y direction) of the vehicle 9 may be the viewing direction. .

  Moreover, in the said embodiment, according to the position of the approaching vehicle 10 with respect to the vehicle 9, you may make it not display on the display 24 the synthesized image seen from the virtual viewpoint. In other words, when the position of the approaching vehicle 10 with respect to the vehicle 9 is far away from the predetermined position, there is less risk of a collision between the vehicle 9 and the approaching vehicle 10, so that the composite image is not displayed on the display 24. Also good.

  In the above embodiment, the size of at least one of the vehicle image 91 and the approaching vehicle image 101 displayed in the composite image may be changed according to the position of the approaching vehicle 10 with respect to the vehicle 9. Good.

  Further, in the first embodiment, the axis sh1 is a lane marking that divides, for example, a lane in which the vehicle 9 travels and a lane in which the approaching vehicle 10 travels.

  In the second embodiment, the virtual viewpoint VP is moved in the counterclockwise direction with respect to the axis sh2, but the virtual viewpoint VP is moved in the axis sh2 according to the position of the approaching vehicle 10 with respect to the vehicle 9. It is good also as clockwise with reference to.

  Further, in the first and second embodiments, when the operation detection unit 114 detects a user operation associated with a change in the travel lane of the vehicle 9, the composite image generation unit 32 generates a composite image and generates the composite image. The synthesized image may be displayed on the display 24.

  In the third embodiment, the superimposed image is superimposed and displayed on the composite image according to at least one of the time for the approaching vehicle 10 to reach the moving position of the vehicle 9 and the position of the approaching vehicle 10 and the vehicle 9. The type and number of warning images to be changed may be changed.

  Further, in the third embodiment, before the process of determining whether or not the approaching vehicle 10 is present (the process of step S101), the process of determining the presence or absence of an operation accompanying the change of the traveling lane of the vehicle 9 (the process of step S201). However, the processing may be performed at a different processing position. For example, the process of step S201 may be performed before the process of comparing the travel time of the vehicle 9 and the travel time of the approaching vehicle 10 (process of step S204).

  In the third and fourth embodiments, the lane change time of the vehicle 9 may be derived from the change in the position of the image of the lane marking that divides the lane in which the vehicle 9 travels and the adjacent lane.

  Further, in the fourth embodiment, the learning unit 118 has been described using the information on the steering angle of the steering wheel when deriving the lane change time. May be performed.

  In the fourth embodiment, the lane change time of the user of the vehicle 9 may be recorded for each speed of the vehicle 9. For example, the speed of the vehicle 9 is divided into low speed (for example, 40 km / h), medium speed (for example, 60 km / h), and high speed (for example, 80 km / h), and the lane change time of the user for each speed band is recorded. May be. Moreover, you may derive | lead-out the lane change time of the vehicle 9 based on the image of the lane marking image | photographed with the vehicle-mounted camera.

  In the fifth embodiment, the image processing apparatus 11b includes the operation detection unit 114, the prediction unit 115, the calculation unit 116, and the determination unit 117, and the vehicle 9 changes lanes and a predetermined condition is satisfied. It has been described that a warning image is displayed when the condition is satisfied. In contrast, when the approaching vehicle 10 is detected by the detection unit 112 without the operation detection unit 114, the prediction unit 115, the calculation unit 116, and the determination unit 117, the virtual position of the driver of the approaching vehicle 10 is determined. The composite image viewed from the viewpoint may be generated by the composite image generation unit 32 and displayed on the display 24 of the display device 20.

  Further, in the seventh embodiment, the generation of the composite image viewed from the virtual viewpoint VP41 has been described as being performed when the detection unit 112 does not detect the presence of the approaching vehicle 10, but the presence of the approaching vehicle 10 is described. May be performed when the detection unit 112 detects the above.

  In the eighth and ninth embodiments, a composite image obtained by viewing an image (for example, image PT51a) taken by the front camera 51 together with the vehicle 9 from a virtual viewpoint according to the position of the approaching vehicle 10 is displayed on the display 24. The configuration to project was explained. On the other hand, the image PT51a is not limited to the image captured by the front camera 51, and may be an image captured by another camera (for example, the left side camera 53).

  In the eighth and ninth embodiments, the front vehicle image 102 in the image of the front camera 51 is displayed when the distance between the vehicle traveling ahead of the vehicle 9 and the vehicle 9 is less than a predetermined distance. You may make it display the image of the frame which surrounds the circumference | surroundings. Thereby, the user can perform an operation on the vehicle 9 for avoiding a collision with the preceding vehicle.

  Further, in the tenth embodiment, a case has been described in which guidance information is notified from the display device 20 to the user when a destination is set using the display device 20. In contrast, when the destination information is notified from the display device 20 to the user when the destination is not set, the composite image generation unit 32 may generate a composite image viewed from the virtual viewpoint. . For example, when the destination is not set using the display device 20, when the number of lanes decreases ahead of the lane on which the vehicle 9 travels, it corresponds to guidance information indicating that it is necessary to change lanes in advance. The composite image viewed from the virtual viewpoint may be displayed on the display 24.

  In the tenth embodiment, the viewpoint positions and viewing directions of the virtual viewpoints VP61 and VP62 described as the first virtual viewpoint setting and the second virtual viewpoint setting are examples, and other viewpoint positions and viewing directions are assumed to be virtual. A composite image viewed from the viewpoint may be generated. For example, it is a virtual viewpoint in which the viewpoint position is substantially in the center of the vehicle body of the vehicle 9 and a position just above it, and the visual field direction is directed vertically downward of the vehicle 9.

  In the tenth embodiment, when a composite image viewed from a virtual viewpoint corresponding to the guidance information is displayed on the display 24, not only one composite image but also a plurality of composite images with different viewpoint positions of the virtual viewpoint are displayed. 24 may be displayed.

DESCRIPTION OF SYMBOLS 1 ... Control part 3 ... Image processing part 5 ... Imaging | photography part 10 ... Image processing apparatus 20 ... Display apparatus

Claims (17)

An image display system used in a vehicle,
Acquisition means for acquiring a plurality of captured images of a camera mounted on the vehicle;
Generating means for combining the plurality of captured images to generate a combined image showing a state of the periphery of the vehicle viewed from a virtual viewpoint;
Detecting means for detecting the presence of an approaching vehicle approaching from behind the vehicle;
Deriving means for deriving the position of the approaching vehicle relative to the vehicle;
Display means for displaying the composite image;
With
It said generating means, Ri the composite image der viewed from the virtual viewpoint corresponding to the position of the approaching vehicle and the position of the vehicle, the approaching vehicle is increased the height of the virtual viewpoint closer to the vehicle Generating a specific composite image,
An image display system characterized by the above. An image display system used in a vehicle,
Acquisition means for acquiring a plurality of captured images of a camera mounted on the vehicle;
A composite image showing the surroundings of the vehicle viewed from a virtual viewpoint by combining the plurality of captured images
Generating means for generating an image;
Detecting means for detecting the presence of an approaching vehicle approaching from behind the vehicle;
Deriving means for deriving the position of the approaching vehicle relative to the vehicle;
Display means for displaying the composite image;
With
The generation means is viewed from the virtual viewpoint according to the position of the vehicle and the position of the approaching vehicle.
Generating a specific composite image that is the composite image, wherein the viewpoint position of the virtual viewpoint is the position of the driver of the approaching vehicle, and the visual field direction of the virtual viewpoint is the direction in which the vehicle exists ;
An image display system characterized by the above. An image display system used in a vehicle,
Acquisition means for acquiring a plurality of captured images of a camera mounted on the vehicle;
A composite image showing the surroundings of the vehicle viewed from a virtual viewpoint by combining the plurality of captured images
Generating means for generating an image;
Detecting means for detecting the presence of an approaching vehicle approaching from behind the vehicle;
Deriving means for deriving the position of the approaching vehicle relative to the vehicle;
Display means for displaying the composite image;
With
The generation means generates a specific composite image that is the composite image viewed from the virtual viewpoint according to the position of the vehicle and the position of the approaching vehicle,
The display means displays one captured image of the captured images from the camera when the approaching vehicle does not exist, and displays the specific composite image together with the one captured image when the approaching vehicle exists. Displaying,
An image display system characterized by the above. An image display system used in a vehicle,
Acquisition means for acquiring a plurality of captured images of a camera mounted on the vehicle;
A composite image showing the surroundings of the vehicle viewed from a virtual viewpoint by combining the plurality of captured images
Generating means for generating an image;
Detecting means for detecting the presence of an approaching vehicle approaching from behind the vehicle;
Deriving means for deriving the position of the approaching vehicle relative to the vehicle;
Display means for displaying the composite image;
With
The generation means generates a specific composite image that is the composite image viewed from the virtual viewpoint according to the position of the vehicle and the position of the approaching vehicle,
The display means uses the specific composite image viewed from the virtual viewpoint with the viewpoint position as the position of the driver of the approaching vehicle and the viewing direction as the direction in which the vehicle exists, and the viewpoint position as a position near the door mirror of the vehicle. Switching and displaying the specific composite image viewed from the virtual viewpoint with the view direction as the direction in which the approaching vehicle exists, according to the operation of the user of the vehicle;
An image display system characterized by the above. The image display system according to claim 3 ,
The display means switches between the size of the one captured image and the size of the specific composite image in accordance with an operation of a user of the vehicle;
An image display system characterized by the above. The image display system according to claim 1 ,
Further comprising an operation detecting means for detecting an operation of a user of the vehicle accompanying a change in a traveling lane of the vehicle,
The display means displays the specific composite image when an operation of the user is detected;
An image display system characterized by the above. The image display system according to claim 6 ,
The generating means generates the specific composite image including the image of the vehicle and the image of the approaching vehicle with the position of the virtual viewpoint as the front of the vehicle and the visual field direction of the virtual viewpoint as the rear of the vehicle;
An image display system characterized by the above. The image display system according to claim 6 ,
The generating means generates the specific composite image with a viewpoint position of the virtual viewpoint as a position of a driver of the approaching vehicle and a visual field direction of the virtual viewpoint as a direction in which the vehicle exists;
An image display system characterized by the above. The image display system according to claim 6 ,
Predicting means for predicting a moving position when the vehicle moves by changing a traveling lane;
A calculation means for calculating a first movement time for the approaching vehicle to move to the movement position;
Determination means for determining whether the vehicle can move to the movement position based on the first movement time;
Further comprising
The generating unit generates an image in which a warning image indicating a determination result of the movement is superimposed on the composite image;
An image display system characterized by the above. The image display system according to claim 9 ,
The generating means generates an image in which the warning image is superimposed on the movement position;
An image display system characterized by the above. The image display system according to claim 9 ,
Measuring means for measuring a second moving time during which the vehicle moves to the moving position;
Storage means for learning and storing the second travel time;
Further comprising
The determination means determines whether the movement is possible based on the first movement time and the stored second movement time;
An image display system characterized by the above. The image display system according to any one of claims 6 to 11 ,
When the user's operation is detected and the approaching vehicle does not exist,
The generation means is a vehicle composite image that is the composite image including the image of the vehicle and the image of the adjacent lane, with the viewpoint position of the virtual viewpoint as the front of the vehicle and the visual field direction of the virtual viewpoint as the rear of the vehicle. Produces
The display means displays the vehicle composite image;
An image display system characterized by the above. The image display system according to any one of claims 1 to 12 ,
Position acquisition means for acquiring the position of the vehicle;
According to the position of the vehicle, the user receives guidance information for guiding a route to the destination of the vehicle.
An informing means for informing ;
Further comprising
The display means displays the composite image when the notification means notifies the guidance information;
An image display system characterized by the above. The image display system according to claim 13 ,
The generation means changes the virtual viewpoint of the composite image according to the content of the guidance information notified by the notification means,
An image display system characterized by the above. The image display system according to claim 14 ,
The guidance information includes information for guiding a change in the driving lane and information for guiding a change in direction at an intersection.
An image display system characterized by the above. An image processing apparatus used in a vehicle,
Acquisition means for acquiring a plurality of captured images of a camera mounted on the vehicle;
A composite image showing the surroundings of the vehicle viewed from a virtual viewpoint by combining the plurality of captured images
Generating means for generating an image;
Detecting means for detecting the presence of an approaching vehicle approaching from behind the vehicle;
Deriving means for deriving the position of the approaching vehicle relative to the vehicle;
With
The generation means is the composite image viewed from the virtual viewpoint according to the position of the vehicle and the position of the approaching vehicle, and the height of the virtual viewpoint is increased as the approaching vehicle approaches the vehicle Generating a composite image,
An image processing apparatus. An image display method used in a vehicle,
(A) acquiring a plurality of captured images of a camera mounted on the vehicle;
(B) Combining the plurality of captured images to show the surroundings of the vehicle viewed from a virtual viewpoint
Generating a composite image; and
(C) detecting the presence of an approaching vehicle approaching from behind the vehicle;
(D) deriving a position of the approaching vehicle relative to the vehicle;
(E) displaying the composite image; and
With
The step (b) is the composite image viewed from the virtual viewpoint according to the position of the vehicle and the position of the approaching vehicle, and increases the height of the virtual viewpoint as the approaching vehicle approaches the vehicle. Generating a specific composite image,
An image display method characterized by the above.

Images