JP6257978B2 - Image generation apparatus, image display system, and image generation method - Google Patents

Description

  The present invention relates to a technique for generating an image showing the periphery of a vehicle.

  2. Description of the Related Art Conventionally, an image display system that generates an image showing the periphery of a vehicle such as an automobile and displays this image on a display device in the vehicle is known. By using such an image display system, a user (typically a driver) can check the state of the periphery of the vehicle almost in real time.

  In recent years, an image display system that generates a virtual viewpoint image that shows the periphery of a vehicle viewed from a virtual viewpoint using a plurality of captured images obtained by a plurality of cameras and displays the virtual viewpoint image is also known. . As such a virtual viewpoint image, for example, an in-vehicle viewpoint image showing the periphery of the vehicle viewed from the viewpoint of the driver of the vehicle has been proposed.

  When generating an in-vehicle viewpoint image, the position of the virtual viewpoint is fixed at a position corresponding to the viewpoint of the driver. By checking such an in-vehicle viewpoint image, the user can check the situation around the vehicle from the same viewpoint as the user, and can intuitively grasp the situation around the vehicle.

  In addition, an image display system that continuously generates in-vehicle viewpoint images while changing the direction of the line of sight of the virtual viewpoint and shows the state of the surroundings of the vehicle from the viewpoint of the driver so as to go around the vehicle is also proposed. (For example, refer to Patent Document 1).

JP 2011-66763 A

  In the in-vehicle viewpoint image as described above, the vehicle body of the vehicle with respect to the image of the subject around the vehicle so that the user can intuitively know which direction around the vehicle the in-vehicle viewpoint image indicates. A vehicle image indicating is superimposed.

  As described above, the position of the virtual viewpoint of the in-vehicle viewpoint image is fixed at a position corresponding to the viewpoint of the driver. For this reason, depending on the direction in which the line of sight of the virtual viewpoint is directed, the vehicle image included in the in-vehicle viewpoint image may show more parts than necessary in the vehicle body. Since the vehicle is generally long in the front-rear direction, the vehicle image indicates a portion from the position of the front seat to the rear end in the vehicle body when the line of sight of the virtual viewpoint is directed to the rear of the vehicle. As described above, when the vehicle image included in the in-vehicle viewpoint image shows many parts of the vehicle body, the vehicle image may be changed and the visibility of the image of the subject around the vehicle may be hindered.

  SUMMARY An advantage of some aspects of the invention is that it provides a technique capable of improving the visibility of an image of a subject around a vehicle included in a virtual viewpoint image.

In order to solve the above-described problem, an invention according to claim 1 is an image generation device used in a vehicle, wherein acquisition means for acquiring a plurality of photographed images obtained by a plurality of cameras, and at least of the plurality of photographed images Generating means for continuously generating a virtual viewpoint image showing a periphery of the vehicle and a vehicle body of the vehicle as viewed from a virtual viewpoint in a vehicle interior, using one and the vehicle body data of the vehicle; Control means for changing the angle of the virtual viewpoint line of sight in plan view so that the viewpoint line of sight circulates around the vehicle, and moving the position of the virtual viewpoint in the front-rear direction of the vehicle , The control means moves the position of the virtual viewpoint according to a direction in which the line of sight of the virtual viewpoint is directed.

Further, the invention of claim 2 is an image generation device used in a vehicle, the acquisition means for acquiring a plurality of photographed images obtained by a plurality of cameras, at least one of the plurality of photographed images, and the vehicle Generating means for continuously generating a virtual viewpoint image showing the periphery of the vehicle and the vehicle body of the vehicle as viewed from a virtual viewpoint in a vehicle interior of the vehicle, and the line of sight of the virtual viewpoint Control means for controlling image generation by changing the angle of the line of sight of the virtual viewpoint in plan view so as to go around the vehicle and moving the position of the virtual viewpoint according to the direction of the line of sight.

  According to a third aspect of the present invention, in the image generating apparatus according to the first or second aspect, the control means is configured to adjust the virtual viewpoint in the front-rear direction and the left-right direction of the vehicle along the periphery of the vehicle compartment. Move position.

  According to a fourth aspect of the present invention, in the image generating apparatus according to any one of the first to third aspects, the control unit is configured to change the virtual viewpoint while the angle of the line of sight of the virtual viewpoint is changed in the planar view. And the angle of the line of sight of the virtual viewpoint in the plan view is maintained while the position of the virtual viewpoint is moving.

  In addition, according to a fifth aspect of the present invention, in the image generation apparatus according to any one of the first to third aspects, the control means includes: an angle of the line of sight of the virtual viewpoint in the plan view; and a position of the virtual viewpoint. Change in parallel.

  According to a sixth aspect of the present invention, in the image generating apparatus according to the first or second aspect, the control means moves the position of the virtual viewpoint in a substantially elliptical shape with a major axis along the front-rear direction of the vehicle. .

The invention according to claim 7 is an image generation apparatus used in a vehicle, the acquisition means for acquiring a plurality of photographed images obtained by a plurality of cameras, at least one of the plurality of photographed images, and the vehicle Using the vehicle body data, generating means for continuously generating a virtual viewpoint image showing the periphery of the vehicle and the vehicle body of the vehicle viewed from a virtual viewpoint in the interior of the vehicle, and the line of sight of the virtual viewpoint Control means for changing the angle of the line of sight of the virtual viewpoint in plan view so as to go around the vehicle, and for moving the position of the virtual viewpoint in the front-rear direction of the vehicle, the generating means, Projecting the data of the plurality of photographed images onto a virtual projection plane, generating the virtual viewpoint image using a partial region of the projection plane, and at least one of the plurality of cameras has an optical axis depression angle With other cameras Becomes, the control means, the adjusting the angle of depression of the line of sight of the virtual viewpoint, the non-projection region to be outside the projection area where the data is projected in the projection plane to the not contain the virtual viewpoint image.

  The invention according to claim 8 is the image generation apparatus according to claim 7, wherein the control means is configured such that when the line of sight of the virtual viewpoint is directed in a direction in which the projection area is minimized on the projection plane, The depression angle of the line of sight of the virtual viewpoint is maintained at a specific angle that does not include the virtual viewpoint image in the projection area.

  In addition, according to a ninth aspect of the present invention, in the image generating apparatus according to the seventh aspect, the control unit is configured to control the virtual viewpoint according to a size of the projection area in a direction in which a line of sight of the virtual viewpoint is directed on the projection plane. Change the gaze angle of.

  Further, the invention of claim 10 is the image generation apparatus according to claim 7, wherein the control unit is configured such that the projection area is minimized on the projection plane when the position of the virtual viewpoint is other than the specific position. When the gaze of the virtual viewpoint is directed to the non-projection area, the depression angle of the gaze of the virtual viewpoint is maintained at a specific angle that does not include the virtual viewpoint image, and the position of the virtual viewpoint is the specific position The depression angle of the line of sight of the virtual viewpoint is made smaller than the specific angle.

  The invention according to claim 11 is an image generation apparatus used in a vehicle, wherein an acquisition means for acquiring a plurality of captured images obtained by a plurality of cameras, and data of the plurality of captured images are virtual projection planes. Generating means for continuously generating a virtual viewpoint image showing a periphery of the vehicle as viewed from a virtual viewpoint in a cabin of the vehicle, using a partial area of the projection plane; Control means for changing an angle in plan view of the line of sight of the virtual viewpoint so that the line of sight circulates around the vehicle, and at least one of the plurality of cameras has another angle of depression of the optical axis Unlike the above, the control means adjusts the depression angle of the line of sight of the virtual viewpoint so that the virtual viewpoint image does not include a non-projection area outside the projection area where the data is projected on the projection plane.

  The invention of claim 12 is an image display system used in a vehicle, and displays the image generation device according to any one of claims 1 to 11 and a virtual viewpoint image generated by the image generation device. And a display device.

The invention of claim 13 is an image generation method used in a vehicle, wherein (a) acquiring a plurality of captured images obtained by a plurality of cameras, and (b) at least one of the plurality of captured images. Continuously generating a virtual viewpoint image indicating the periphery of the vehicle and the vehicle body of the vehicle viewed from a virtual viewpoint in the vehicle interior using one and the vehicle body data of the vehicle; and (c) Changing the angle of the virtual viewpoint line of sight in plan view so that the line of sight of the virtual viewpoint circulates around the vehicle, and moving the position of the virtual viewpoint in the front-rear direction of the vehicle. The step (c) moves the position of the virtual viewpoint according to the direction in which the line of sight of the virtual viewpoint is directed.

  The invention of claim 14 is an image generation method used in a vehicle, wherein (a) a step of acquiring a plurality of photographed images obtained by a plurality of cameras, and (b) data of the plurality of photographed images. Projecting a virtual projection image onto a virtual projection plane, and continuously generating a virtual viewpoint image showing the periphery of the vehicle viewed from a virtual viewpoint in a vehicle interior of the vehicle, using a partial area of the projection plane; (C) changing the angle of the virtual viewpoint line of sight in plan view so that the line of sight of the virtual viewpoint circulates around the vehicle, and at least one of the plurality of cameras has an optical axis Unlike other cameras, the step (c) adjusts the depression angle of the line of sight of the virtual viewpoint, and defines a non-projection area outside the projection area on which the data is projected on the projection plane as the virtual viewpoint. Do not include images.

According to the inventions of claims 1 to 10, 12 and 13, since the position of the virtual viewpoint is moved in the longitudinal direction of the vehicle, the portion of the vehicle body indicated by the virtual viewpoint image can be reduced, and the periphery of the vehicle included in the in-vehicle viewpoint image The visibility of the subject image can be improved. Further, since the position of the virtual viewpoint is moved according to the direction in which the line of sight of the virtual viewpoint is directed, the portion of the vehicle body indicated by the virtual viewpoint image can be reduced.

  In particular, according to the second aspect of the present invention, the position of the virtual viewpoint is moved in accordance with the direction in which the line of sight of the virtual viewpoint is directed, so that the portion of the vehicle body indicated by the virtual viewpoint image can be reduced.

  In particular, according to the invention of claim 3, since the position of the virtual viewpoint is moved along the periphery of the passenger compartment of the vehicle, the portion of the vehicle body indicated by the virtual viewpoint image can be reduced.

  In particular, according to the invention of claim 4, the movement of the line of sight of the virtual viewpoint can be simplified, and the virtual viewpoint image can easily show the periphery of the vehicle.

  In particular, according to the invention of claim 5, since the angle of the virtual viewpoint line of sight in plan view and the position of the virtual viewpoint are changed in parallel, the time for the virtual viewpoint line of sight to circulate around the vehicle is shortened. it can.

  In particular, according to the invention of claim 6, the position of the virtual viewpoint is moved so that the major axis is substantially elliptical along the longitudinal direction of the vehicle. The distance to move the viewpoint position can be shortened.

  In particular, according to the seventh aspect of the invention, since the virtual viewpoint image is not included in the non-projection area on the projection plane, it is possible to prevent the user who visually recognizes the virtual viewpoint image from feeling uncomfortable.

  In particular, according to the invention of claim 8, since the depression angle of the line of sight of the virtual viewpoint is maintained, the movement of the line of sight of the virtual viewpoint can be simplified, and the virtual viewpoint image can easily show the periphery of the vehicle.

  In particular, according to the ninth aspect of the present invention, since the depression angle of the line of sight of the virtual viewpoint is changed according to the size of the projection area, the virtual viewpoint image can indicate a relatively far subject away from the vehicle.

  In particular, according to the invention of claim 10, the virtual viewpoint image can indicate a relatively distant subject away from the vehicle with respect to the specific direction of the vehicle while simplifying the movement of the line of sight of the virtual viewpoint.

  Further, according to the inventions of claims 11 and 14, since the virtual viewpoint image does not include the non-projection area on the projection plane, it is possible to prevent the user who visually recognizes the virtual viewpoint image from feeling uncomfortable.

FIG. 1 is a diagram illustrating a configuration of an image display system according to the first embodiment. FIG. 2 is a diagram illustrating the directions in which the four cameras 5 capture images. FIG. 3 is a diagram illustrating the included angles of the optical axes of the four cameras 5. FIG. 4 is a diagram illustrating a method for generating a virtual viewpoint image. FIG. 5 is a diagram illustrating transition of operation modes of the image display system. FIG. 6 is a diagram illustrating an example of a display image including an overhead image. FIG. 7 is a diagram illustrating an example of a display image including an in-vehicle viewpoint image. FIG. 8 is a diagram illustrating transition of the virtual viewpoint according to the first embodiment. FIG. 9 is a diagram illustrating an example of the in-vehicle viewpoint image. FIG. 10 is a diagram illustrating an example of the in-vehicle viewpoint image. FIG. 11 is a diagram illustrating an example of the in-vehicle viewpoint image. FIG. 12 is a diagram illustrating an example of the in-vehicle viewpoint image. FIG. 13 is a diagram illustrating transition of virtual viewpoints in the comparative example. FIG. 14 is a diagram illustrating an example of an in-vehicle viewpoint image generated in the comparative example. FIG. 15 is a diagram illustrating an example of the in-vehicle viewpoint image generated in the comparative example. FIG. 16 is a diagram illustrating a projection plane used for generating a virtual viewpoint image. FIG. 17 is a diagram illustrating the depression angle of the line of sight of the virtual viewpoint according to the first embodiment. FIG. 18 is a diagram illustrating a use region used for generating an in-vehicle viewpoint image according to the first embodiment. FIG. 19 is a diagram showing a flow of operations of the image display system. FIG. 20 is a diagram illustrating the transition of the virtual viewpoint according to the second embodiment. FIG. 21 is a diagram illustrating the transition of the virtual viewpoint according to the third embodiment. FIG. 22 is a diagram illustrating transition of virtual viewpoints according to the fourth embodiment. FIG. 23 is a diagram illustrating the depression angle of the line of sight of the virtual viewpoint according to the fifth embodiment. FIG. 24 is a diagram illustrating an example of the in-vehicle viewpoint image. FIG. 25 is a diagram illustrating an example of the in-vehicle viewpoint image. FIG. 26 is a diagram illustrating a use region used for generating an in-vehicle viewpoint image according to the fifth embodiment. FIG. 27 is a diagram illustrating the depression angle of the line of sight of the virtual viewpoint according to the sixth embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<1. First Embodiment>
<1-1. Configuration>
FIG. 1 is a diagram illustrating a configuration of an image display system 10 according to the first embodiment. This image display system 10 is used in a vehicle (in this embodiment, an automobile), and has a function of generating an image showing an area around the vehicle and displaying it in the vehicle interior. By using this image display system 10, a user (typically a driver) of the image display system 10 can grasp the state around the vehicle in almost real time.

  As shown in the figure, the image display system 10 includes a plurality of cameras 5, an image generation device 2, a display device 3, and operation buttons 4. Each of the plurality of cameras 5 captures the periphery of the vehicle to acquire a captured image, and inputs the acquired captured image to the image generation device 2. The image generation device 2 generates a display image to be displayed on the display device 3 using a captured image showing the periphery of the vehicle. The display device 3 displays the display image generated by the image generation device 2. The operation button 4 receives a user operation.

  Each of the plurality of cameras 5 includes a lens and an image sensor, and electronically obtains a captured image showing the periphery of the vehicle. The multiple cameras 5 include a front camera 5F, a rear camera 5B, a left side camera 5L, and a right side camera 5R. These four cameras 5 are arranged at different positions in the vehicle 9 and photograph different directions around the vehicle 9.

  FIG. 2 is a diagram illustrating the directions in which the four cameras 5 capture images. The front camera 5F is provided at the front end of the vehicle 9, and its optical axis 5Fa is directed forward along the front-rear direction of the vehicle 9. The rear camera 5B is provided at the rear end of the vehicle 9, and its optical axis 5Ba is directed rearward along the front-rear direction of the vehicle 9. The left side camera 5L is provided on the left side mirror 93L on the left side, and its optical axis 5La is directed leftward along the left-right direction of the vehicle 9. The right side camera 5 </ b> R is provided on the right side mirror 93 </ b> R on the right side, and its optical axis 5 </ b> Ra is directed rightward along the left-right direction of the vehicle 9.

  The lenses of these cameras 5 are wide-angle lenses such as fish-eye lenses, and each camera 5 has an angle of view θ of 180 degrees or more. For this reason, it is possible to photograph the entire periphery of the vehicle 9 by using the four cameras 5.

  Further, the front camera 5F, the rear camera 5B, and the side cameras 5L and 5R have different angles of the optical axes (downward angles with respect to the horizontal direction). FIG. 3 is a diagram illustrating the included angles of the optical axes of the four cameras 5. The dashed-dotted line shown in FIG. 3 is along the horizontal direction.

  As shown in FIG. 3, for the left and right side cameras 5L and 5R, the depression angles αL and αR of the optical axes 5La and 5Ra are substantially the same, for example, 80 °. On the other hand, in the front camera 5F and the rear camera 5B, the depression angle of the optical axis is different from that of the side cameras 5L and 5R. The depression angle αF of the optical axis 5Fa of the front camera 5F is, for example, 20 °, and the depression angle αB of the optical axis 5Ba of the rear camera 5B is, for example, 50 °.

  Accordingly, the front camera 5F, the rear camera 5B, and the side cameras 5L and 5R have different distances in the direction away from the vehicle 9 that can be photographed. Of the optical axes of the four cameras 5, the optical axis 5 Fa of the front camera 5 </ b> F is directed to the uppermost side, so the front camera 5 </ b> F can capture an area relatively far from the vehicle 9. On the other hand, since the optical axes 5La and 5Ra of the left and right side cameras 5L and 5R are directed to the lowermost side, the side cameras 5L and 5R can capture only an area relatively close to the vehicle 9.

  Returning to FIG. 1, the display device 3 includes a thin display panel such as a liquid crystal display, and displays various types of information and images. The display device 3 is disposed on an instrument panel or the like of the vehicle 9 so that the user can visually recognize the screen of the display panel. The display device 3 may be disposed in the same housing as the image generation device 2 and integrated with the image generation device 2, or may be a separate device from the image generation device 2. Further, the display device 3 includes a touch panel 31 superimposed on the display panel, and can accept a user operation. The display device 3 may have other functions such as a navigation function for performing route guidance to a destination in addition to the function of displaying.

  The operation button 4 is an operation member that receives a user operation. The operation button 4 is provided, for example, on the steering wheel of the vehicle 9 and mainly receives an operation from the driver. The user can perform various operations on the image display system 10 via the operation buttons 4 and the touch panel 31 of the display device 3. When a user operation is performed on either the operation button 4 or the touch panel 31, an operation signal indicating the content of the operation is input to the image generation device 2.

  The image generation device 2 is an electronic device capable of various image processing. The image generation device 2 includes an image acquisition unit 21, an image generation unit 22, an image adjustment unit 23, and an image output unit 24.

  The image acquisition unit 21 acquires four captured images obtained by the four cameras 5. The image acquisition unit 21 has an image processing function such as a function of converting an analog captured image into a digital captured image. The image acquisition unit 21 performs predetermined image processing on the acquired captured image and inputs the processed captured image to the image generation unit 22.

  The image generation unit 22 is a hardware circuit that performs image processing for generating a virtual viewpoint image. The image generation unit 22 uses the four captured images acquired by the four cameras 5 to generate a virtual viewpoint image indicating the periphery of the vehicle 9 as viewed from the virtual viewpoint. The image generation unit 22 generates virtual viewpoint images continuously (continuously in time) using the most recently obtained captured image in each camera 5. Thereby, the image generation part 22 produces | generates the virtual viewpoint image which shows the periphery of a vehicle substantially in real time. Details of the method for generating the virtual viewpoint image will be described later.

  The image adjustment unit 23 generates a display image to be displayed on the display device 3. The image adjustment unit 23 generates a display image including the virtual viewpoint image generated by the image generation unit 22.

  The image output unit 24 outputs the display image generated by the image adjustment unit 23 to the display device 3 and causes the display device 3 to display the display image. As a result, a virtual viewpoint image that shows the vicinity of the vehicle 9 viewed from the virtual viewpoint almost in real time is displayed on the display device 3.

  The image generation device 2 further includes a control unit 20, an operation reception unit 25, a signal reception unit 26, and a storage unit 27. The control unit 20 is a microcomputer including a CPU, a RAM, a ROM, and the like, for example, and comprehensively controls the entire image generation apparatus 2.

  The operation receiving unit 25 receives an operation signal sent from the operation button 4 and the touch panel 31 when the user performs an operation. Thereby, the operation reception part 25 receives a user's operation. The operation reception unit 25 inputs the received operation signal to the control unit 20.

  The signal receiving unit 26 receives a signal transmitted from another device provided in the vehicle 9 separately from the image generating device 2 and inputs the signal to the control unit 20. The signal receiving unit 26 can receive a signal sent from the shift sensor 95 of the vehicle 9. The shift sensor 95 detects the shift position that is the position of the shift lever of the transmission of the vehicle 9 and sends a signal indicating the shift position to the image generation device 2. Based on this signal, the control unit 20 can determine whether the traveling direction of the vehicle 9 is forward or backward.

  The storage unit 27 is, for example, a nonvolatile memory such as a flash memory, and stores various types of information. The storage unit 27 stores a program 27a as firmware and various data used by the image generation unit 22 to generate a virtual viewpoint image. Data used for generating such a virtual viewpoint image includes vehicle body data 27b indicating the shape and size of the vehicle body of the vehicle 9.

  Various functions of the control unit 20 are realized by executing the program 27a stored in the storage unit 27 (CPU arithmetic processing according to the program 27a). An image control unit 20a shown in the drawing is a part of a functional unit realized by executing the program 27a.

  The image control unit 20a controls the image generation unit 22 that generates a virtual viewpoint image and the image adjustment unit 23 that generates a display image. For example, the image control unit 20a changes the position of the virtual viewpoint related to the virtual viewpoint image generated by the image generation unit 22 and the direction of the line of sight of the virtual viewpoint.

<1-2. Generation of virtual viewpoint image>
Next, a method will be described in which the image generation unit 22 generates a virtual viewpoint image that shows the surroundings of the vehicle 9 as viewed from the virtual viewpoint. FIG. 4 is a diagram illustrating a method in which the image generation unit 22 generates a virtual viewpoint image. The image generation unit 22 generates a virtual viewpoint image with a realistic sensation close to reality by using a virtual stereoscopic projection plane TS for generation of the virtual viewpoint image.

  The front camera 5F, the rear camera 5B, the left side camera 5L, and the right side camera 5R acquire four captured images SF, SB, SL, and SR that respectively indicate the front, rear, left, and right sides of the vehicle 9. . These four captured images SF, SB, SL, and SR include data around the entire vehicle 9.

  The image generation unit 22 projects data (pixel values) included in the four captured images SF, SB, SL, and SR onto a projection surface TS that is a three-dimensional curved surface in a virtual three-dimensional space. The projection surface TS has, for example, a substantially hemispherical shape (a bowl shape), and a central area (bottom part of the bowl) is defined as the position of the vehicle 9. Further, the outside of the position of the vehicle 9 on the projection surface TS corresponds to a region around the vehicle 9.

  A correspondence relationship is determined in advance between the position of the data included in the captured images SF, SB, SL, and SR and the position of the projection plane TS. Table data indicating such a correspondence relationship is stored in the storage unit 27. The image generation unit 22 projects the data included in the four captured images SF, SB, SL, SR onto the corresponding positions on the projection surface TS using the table data.

  The image generation unit 22 projects the data of the captured image SF of the front camera 5F on a portion corresponding to the front of the vehicle 9 on the projection surface TS. In addition, the image generation unit 22 projects the data of the captured image SB of the rear camera 5B on a portion corresponding to the rear of the vehicle 9 on the projection surface TS. Further, the image generation unit 22 projects the data of the captured image SL of the left side camera 5L on the portion corresponding to the left side of the vehicle 9 on the projection plane TS, and the portion corresponding to the right side of the vehicle 9 on the projection plane TS. Data of the direction image SR of the right side camera 5R is projected.

  When the captured image data is projected onto the projection surface TS in this manner, the image generation unit 22 next uses the vehicle body data 27b stored in the storage unit 27 to generate a polygon model indicating the three-dimensional shape of the vehicle 9. Configure virtually. The model of the vehicle 9 is arranged in the central region of the projection plane TS that is the position of the vehicle 9 in a virtual three-dimensional space.

  Next, the image generation unit 22 sets a virtual viewpoint VP for the three-dimensional space under the control of the image control unit 20a. This virtual viewpoint VP is defined by the position and the direction of the line of sight. The image generation unit 22 can set a virtual viewpoint VP having an arbitrary position and an arbitrary line-of-sight direction in a three-dimensional space.

  Next, the image generation unit 22 generates a virtual viewpoint image CP using a partial region of the projection plane TS corresponding to the set virtual viewpoint VP. That is, the image generation unit 22 cuts out data of an area included in a predetermined viewing angle as viewed from the virtual viewpoint VP in the projection plane TS as an image. The clipped image includes an image of a subject around the vehicle 9. At the same time, the image generation unit 22 performs rendering on the model of the vehicle 9 according to the set virtual viewpoint VP, and superimposes the two-dimensional vehicle image 90 as a result on the cut-out image. The vehicle image 90 indicates the shape of the vehicle body of the vehicle 9 viewed from the virtual viewpoint VP. As a result, the image generation unit 22 generates a virtual viewpoint image CP that shows the periphery of the vehicle 9 and the vehicle body of the vehicle 9 as viewed from the virtual viewpoint VP.

  For example, as illustrated in FIG. 4, when a virtual viewpoint VPa is set with the position directly above the vehicle 9 and the line of sight directed downward, a virtual viewpoint image (overhead image) that overlooks the periphery of the vehicle 9 and the vehicle body of the vehicle 9. ) CPa can be generated. When the virtual viewpoint VPb is set with the position being the left rear of the vehicle 9 and the line of sight directed to the front of the vehicle 9, the virtual viewpoint indicating the periphery of the vehicle 9 and the vehicle body of the vehicle 9 as viewed from the left rear of the vehicle 9. An image CPb can be generated.

  The image generation unit 22 according to the present embodiment can set the position of the virtual viewpoint VP in the vehicle interior of the vehicle 9 as well as the outside of the vehicle 9. When the position of the virtual viewpoint VP is set in the vehicle interior of the vehicle 9, it is possible to generate a realistic virtual viewpoint image showing the periphery of the vehicle 9 and the vehicle body (interior) of the vehicle 9 as viewed from the vehicle interior.

<1-3. Operation mode>
Next, the operation mode of the image display system 10 will be described. FIG. 5 is a diagram illustrating transition of operation modes of the image display system 10. The image display system 10 has three operation modes: a normal mode M0, an overhead view mode M1, and a circulation mode M2. These operation modes are switched by the control of the control unit 20 according to the state of the vehicle 9 and the user's operation.

  The normal mode M0 is an operation mode in which the function of the image generation device 2 is not used. When the display device 3 has a navigation function, the display device 3 displays a map image or the like based on the navigation function in the normal mode M0.

  On the other hand, the bird's-eye view mode M1 and the circulation mode M2 are operation modes in which a display image including a virtual viewpoint image generated by the image generation device 2 is displayed on the display device 3 by using the function of the image generation device 2. Therefore, in these operation modes, the user can grasp the state of the periphery of the vehicle 9 almost in real time by confirming the display image displayed on the display device 3.

  The bird's-eye view mode M <b> 1 is an operation mode for displaying a bird's-eye view image that is a virtual viewpoint image for bird's-eye view of the periphery of the vehicle 9 from the viewpoint directly above the vehicle 9. FIG. 6 is a diagram illustrating an example of a display image DP including the overhead image CP1 displayed in the overhead view mode M1. As shown in the figure, this display image DP includes a camera viewpoint image CP2 that is a virtual viewpoint image showing the periphery of the vehicle 9 as viewed from the position of the camera 5 together with the overhead image CP1.

  The virtual viewpoint VP of the camera viewpoint image CP2 is selected based on the traveling direction of the vehicle 9. When the traveling direction of the vehicle 9 is forward, a virtual viewpoint VP similar to the optical axis 5Fa of the front camera 5F is set. When the traveling direction of the vehicle 9 is backward, the virtual viewpoint VP similar to the optical axis 5Ba of the rear camera 5B is set. Is set. The traveling direction of the vehicle 9 is determined by the image control unit 20 a based on a signal sent from the shift sensor 95. By confirming such a bird's-eye view image CP1 and a camera viewpoint image CP2, the user can grasp the state of the entire periphery of the vehicle 9 and the state of the traveling direction of the vehicle 9.

  On the other hand, the circulation mode M2 is an operation mode for displaying an in-vehicle viewpoint image that is a virtual viewpoint image showing the periphery of the vehicle 9 as viewed from the viewpoint inside the vehicle 9. FIG. 7 is a diagram illustrating an example of the display image DP including the in-vehicle viewpoint image CP3. As shown in the figure, this in-vehicle viewpoint image CP3 is a vehicle image 90 showing the vehicle body (interior) of the vehicle 9 viewed from the viewpoint of the vehicle interior in the image of the subject around the vehicle 9 viewed from the viewpoint of the vehicle interior of the vehicle 9. A realistic image in which is superimposed. By confirming such an in-vehicle viewpoint image CP3, the user can confirm the situation around the vehicle 9 from the viewpoint inside the vehicle interior, and can intuitively grasp the situation around the vehicle 9. Further, the user can intuitively grasp which direction around the vehicle 9 indicates the vehicle viewpoint image CP3 based on the vehicle image 90 included in the vehicle viewpoint image CP3.

  The vehicle image 90 is divided into a portion corresponding to the bottom surface of the vehicle body and a portion corresponding to a portion other than the bottom surface of the vehicle body. A portion corresponding to the bottom of the vehicle body in the vehicle image 90 is non-transparent. On the other hand, a portion corresponding to a portion other than the bottom of the vehicle body in the vehicle image 90 is transparent or translucent except for some characteristic features such as tires and frames. Thereby, even if the vehicle image 90 is included in the in-vehicle viewpoint image CP3, the image of the subject around the vehicle 9 can be confirmed.

  In the rounding mode M2, such in-vehicle viewpoint images are continuously displayed on the display device 3, and an animation showing the state of the surroundings of the vehicle 9 viewed from the viewpoint in the vehicle interior is performed so as to go around the vehicle 9. Therefore, the user can check the entire periphery of the vehicle 9 by using the circulation mode M2.

  As shown in FIG. 5, when the image display system 10 is activated, first, the operation mode is a circulation mode M2. In the rotation mode M2, the line of sight of the virtual viewpoint VP makes a round around the vehicle 9. When the circulation of the line of sight of the virtual viewpoint VP is completed, the operation mode is switched to the normal mode M0. In the normal mode M0, when the user presses the operation button 4 for a long time (when the user continuously presses the operation button 4 for a certain period of time), the operation mode is switched to the circulation mode M2.

  In the normal mode M0, when the user presses the operation button 4 with a normal length, or when the shift position of the vehicle 9 is reversed, the operation mode is switched to the overhead view mode M1. Further, when the user presses the operation button 4 in the overhead view mode M1, the operation mode is switched to the normal mode M0.

<1-4. Circulation mode>
Next, the rotation mode M2 will be described in more detail. As described above, in the lap mode M2, an animation showing the state of the periphery of the vehicle 9 as viewed from the viewpoint of the passenger compartment is performed so as to circulate around the vehicle 9.

  In order to perform such an animation, in the circulation mode M2, the image control unit 20a changes the direction of the line of sight of the virtual viewpoint VP so that the line of sight of the virtual viewpoint VP circulates around the vehicle 9. Specifically, the image control unit 20a determines the angle with respect to the front-rear direction of the vehicle 9 in a plan view of the line of sight of the virtual viewpoint VP (the angle with respect to the front-rear direction of the vehicle 9 when the vehicle 9 is viewed from above. Gradually change the viewing angle. Then, the image generation unit 22 continuously generates the in-vehicle viewpoint images (continuously in time) in a state where the image control unit 20a gradually changes the planar viewing angle of the line of sight of the virtual viewpoint VP. .

  An in-vehicle viewpoint image generated in this way is continuously displayed on the display device 3, whereby an animation showing the state of the periphery of the vehicle 9 is performed so as to go around the vehicle 9. By confirming such an animation, the user can easily confirm the state of the entire periphery of the vehicle 9. This animation is performed over 20 seconds, for example.

<1-4-1. Moving the virtual viewpoint position>
In the round mode M2, the image control unit 20a not only changes the direction of the line of sight of the virtual viewpoint VP (planar viewing angle) but also changes the position of the virtual viewpoint VP.

  FIG. 8 is a diagram showing the transition of the virtual viewpoint VP in the circulation mode M2, and shows the vehicle 9 in plan view. Each solid line arrow in the figure represents the virtual viewpoint VP, and the starting point of the solid line arrow indicates the position of the virtual viewpoint VP. The direction of the solid arrow indicates the direction of the line of sight of the virtual viewpoint VP in plan view. That is, the angle of the solid arrow with respect to the front-rear direction of the vehicle 9 corresponds to the planar viewing angle of the line of sight of the virtual viewpoint VP.

  As shown in the figure, the image control unit 20a gradually changes the planar viewing angle of the line of sight of the virtual viewpoint VP so that the line of sight of the virtual viewpoint VP goes around the vehicle 9 clockwise. At the same time, the image control unit 20a linearly moves the position of the virtual viewpoint VP in the front-rear direction and the left-right direction of the vehicle 9 along the periphery of the vehicle interior 91 so as to go around the interior of the vehicle interior 91 of the vehicle 9. Moving.

  First, the image control unit 20a sets the planar viewing angle of the line of sight of the virtual viewpoint VP so that the line of sight of the virtual viewpoint VP is directed forward along the front-rear direction of the vehicle 9, which is the initial direction. At the same time, the image control unit 20a sets the position of the virtual viewpoint VP to the front center position (hereinafter referred to as “front center position”) P1 of the vehicle interior 91, which is the initial position.

  FIG. 9 is a diagram illustrating an example of the in-vehicle viewpoint image CP31 generated by the image generation unit 22 when the line of sight of the virtual viewpoint VP faces the front of the vehicle 9 and the position of the virtual viewpoint VP is the front center position P1. As illustrated in FIG. 9, the in-vehicle viewpoint image CP <b> 31 is an image in which a vehicle image 90 indicating the front of the vehicle 9 is superimposed on an image of a subject in front of the vehicle 9.

  Returning to FIG. 8, next, the image control unit 20a continuously keeps the position of the virtual viewpoint VP rightward toward the front right end position P2 of the vehicle interior 91 while maintaining the planar viewing angle of the line of sight of the virtual viewpoint VP. (Broken arrow A1). That is, the image control unit 20a moves the position of the virtual viewpoint VP to the right along the front edge of the passenger compartment 91 while maintaining the state where the line of sight of the virtual viewpoint VP is directed forward.

  When the position of the virtual viewpoint VP is moved to the front right end position P2, the image control unit 20a stops moving the position of the virtual viewpoint VP, and then the line of sight of the virtual viewpoint VP moves to the right along the left-right direction of the vehicle 9. The planar viewing angle of the line of sight of the virtual viewpoint VP is continuously changed so as to face (dashed arrow A11). That is, the image control unit 20a changes the planar viewing angle of the line of sight of the virtual viewpoint VP by 90 ° by rotating it to the right while maintaining the position of the virtual viewpoint VP at the front right end position P2.

  When the line of sight of the virtual viewpoint VP is directed to the right side of the vehicle 9, the image control unit 20a maintains the plane viewing angle of the line of sight of the virtual viewpoint VP, and changes the position of the virtual viewpoint VP to the position P3 at the rear right end of the passenger compartment 91. Continuously moving backward (dashed arrow A2). That is, the image control unit 20a moves the position of the virtual viewpoint VP rearward along the right edge of the passenger compartment 91 while maintaining the state where the line of sight of the virtual viewpoint VP is directed to the right.

  FIG. 10 illustrates the in-vehicle viewpoint image CP32 generated by the image generation unit 22 when the line of sight of the virtual viewpoint VP is directed to the right of the vehicle 9 and the position of the virtual viewpoint VP moves along the right edge of the passenger compartment 91. It is a figure which shows an example. As shown in FIG. 10, the in-vehicle viewpoint image CP <b> 32 is an image in which a vehicle image 90 indicating the right side of the vehicle 9 is superimposed on an image of a subject on the right side of the vehicle 9.

  Returning to FIG. 8, when the position of the virtual viewpoint VP is moved to the rear rightmost position P3, the image control unit 20a stops the movement of the position of the virtual viewpoint VP, and then the line of sight of the virtual viewpoint VP extends in the front-rear direction of the vehicle 9. The planar viewing angle of the line of sight of the virtual viewpoint VP is continuously changed so as to face backward along the line (broken line arrow A12). That is, the image control unit 20a changes the planar viewing angle of the line of sight of the virtual viewpoint VP by 90 ° by rotating it to the right while maintaining the position of the virtual viewpoint VP at the rear right end position P3.

  When the line of sight of the virtual viewpoint VP is directed to the rear of the vehicle 9, the image control unit 20a keeps the plane view angle of the line of sight of the virtual viewpoint VP and directs the position of the virtual viewpoint VP toward the position P5 at the rear left end of the passenger compartment 91. And move continuously to the left (dashed arrow A3). In other words, the image control unit 20a moves the position of the virtual viewpoint VP to the left along the rear edge of the passenger compartment 91 while maintaining the state where the line of sight of the virtual viewpoint VP is directed rearward. During this movement, the position of the virtual viewpoint VP passes through the rear center position (hereinafter referred to as “rear center position”) P4 of the passenger compartment 91.

  FIG. 11 is a diagram illustrating an example of the in-vehicle viewpoint image CP33 generated by the image generation unit 22 when the line of sight of the virtual viewpoint VP faces the rear of the vehicle 9 and the position of the virtual viewpoint VP is the rear center position P4. As shown in FIG. 11, the in-vehicle viewpoint image CP <b> 33 is an image in which a vehicle image 90 indicating the rear of the vehicle 9 is superimposed on an image of a subject behind the vehicle 9.

  Returning to FIG. 8, when the position of the virtual viewpoint VP is moved to the rear leftmost position P5, the image control unit 20a stops moving the position of the virtual viewpoint VP, and then the line of sight of the virtual viewpoint VP moves in the left-right direction of the vehicle 9. The planar viewing angle of the line of sight of the virtual viewpoint VP is continuously changed so as to face to the left along the line (broken line arrow A13). That is, the image control unit 20a changes the planar viewing angle of the line of sight of the virtual viewpoint VP by 90 ° by rotating it to the right while maintaining the position of the virtual viewpoint VP at the position P5 at the rear left end.

  When the line of sight of the virtual viewpoint VP is directed to the left side of the vehicle 9, the image control unit 20a maintains the plane viewing angle of the line of sight of the virtual viewpoint VP, and moves the position of the virtual viewpoint VP to the position P6 at the front left end of the vehicle interior 91. Continuously moving forward (dashed arrow A4). That is, the image control unit 20a moves the position of the virtual viewpoint VP forward along the left peripheral edge of the passenger compartment 91 while maintaining the state where the line of sight of the virtual viewpoint VP is directed to the left.

  FIG. 12 illustrates the in-vehicle viewpoint image CP34 generated by the image generation unit 22 when the line of sight of the virtual viewpoint VP is directed to the left of the vehicle 9 and the position of the virtual viewpoint VP moves along the left edge of the passenger compartment 91. It is a figure which shows an example. As illustrated in FIG. 12, the in-vehicle viewpoint image CP <b> 34 is an image in which a vehicle image 90 indicating the left side of the vehicle 9 is superimposed on an image of a subject on the left side of the vehicle 9.

  Returning to FIG. 8, when the position of the virtual viewpoint VP is moved to the front left end position P <b> 6, the image control unit 20 a stops moving the position of the virtual viewpoint VP, and then the line of sight of the virtual viewpoint VP extends in the front-rear direction of the vehicle 9. The planar viewing angle of the line of sight of the virtual viewpoint VP is continuously changed so as to face forward along the line (broken line arrow A14). That is, the image control unit 20a changes the planar viewing angle of the line of sight of the virtual viewpoint VP by 90 ° by rotating it to the right while maintaining the position of the virtual viewpoint VP at the front left end position P6.

  When the line of sight of the virtual viewpoint VP is directed to the front of the vehicle 9, the image control unit 20a continuously maintains the position of the virtual viewpoint VP toward the front center position P1 while maintaining the planar viewing angle of the line of sight of the virtual viewpoint VP. (Dashed arrow A5). That is, the image control unit 20a moves the position of the virtual viewpoint VP to the right along the front edge of the passenger compartment 91 while maintaining the state where the line of sight of the virtual viewpoint VP is directed forward.

  Finally, the position of the virtual viewpoint VP returns to the front center position P1, which is the initial position. When the position of the virtual viewpoint VP returns to the front center position P1, the image generation unit 22 generates an in-vehicle viewpoint image similar to the in-vehicle viewpoint image CP31 illustrated in FIG.

  As described above, in the circulation mode M2, the image control unit 20a moves the position of the virtual viewpoint VP according to the direction in which the line of sight of the virtual viewpoint VP is directed. That is, the image control unit 20a moves the position of the virtual viewpoint VP to the front of the passenger compartment 91 when the line of sight of the virtual viewpoint VP is directed to the front of the vehicle 9. Similarly, the image control unit 20a moves the position of the virtual viewpoint VP to the rear of the passenger compartment 91 when the line of sight of the virtual viewpoint VP is directed to the rear of the vehicle 9. Further, the image control unit 20a moves the position of the virtual viewpoint VP to the right side of the passenger compartment 91 when the line of sight of the virtual viewpoint VP is directed to the right side of the vehicle 9, and moves the line of sight of the virtual viewpoint VP to the left side of the vehicle 9. When moving to the left, the position of the virtual viewpoint VP is moved to the left of the passenger compartment 91. As described above, the position of the virtual viewpoint VP is also moved according to the direction in which the line of sight of the virtual viewpoint VP is directed, so that the visibility of the subject image of the vehicle 9 included in the in-vehicle viewpoint image can be improved.

  FIG. 13 is a diagram illustrating transition of the virtual viewpoint VP in the comparative example. In this comparative example, the position of the virtual viewpoint VP is not moved and is fixed at the position P0 corresponding to the viewpoint of the driver in the passenger compartment 91 in the circulation mode. Further, the planar viewing angle of the line of sight of the virtual viewpoint VP is continuously changed so that the line of sight of the virtual viewpoint VP rotates clockwise. FIG. 14 is a diagram illustrating an example of the in-vehicle viewpoint image CP41 generated when the line of sight of the virtual viewpoint VP faces the front of the vehicle 9 in the comparative example. FIG. 15 is a diagram illustrating an example of the in-vehicle viewpoint image CP43 that is generated when the line of sight of the virtual viewpoint VP faces the rear of the vehicle 9 in the comparative example.

  As can be seen by comparing FIGS. 14 and 9 and FIGS. 15 and 11, the in-vehicle viewpoint images CP41 and CP43 of the comparative example are compared with the in-vehicle viewpoint images CP31 and CP33 of the present embodiment. Thus, the vehicle image 90 shows more parts than necessary in the vehicle body. For this reason, the vehicle image 90 may hinder the visibility of the subject image around the vehicle 9. Further, in the in-vehicle viewpoint images CP41 and CP43 of the comparative example, since the distance from the position of the virtual viewpoint VP to the subject around the vehicle 9 is long, the image of the subject to be confirmed becomes small and the visibility of the subject image is improved. It is getting worse.

  On the other hand, in the present embodiment, the position of the virtual viewpoint VP also moves according to the direction in which the image control unit 20a directs the line of sight of the virtual viewpoint VP. For this reason, the part of the vehicle body indicated by the vehicle image 90 included in the in-vehicle viewpoint image can be reduced. Further, since the distance from the position of the virtual viewpoint VP to the subject around the vehicle 9 is short, the image of the subject to be confirmed can be enlarged. As a result, it is possible to improve the visibility of the subject image around the vehicle 9 included in the in-vehicle viewpoint image.

  Since the vehicle is generally long in the front-rear direction, the visibility of the image of the subject included in the in-vehicle viewpoint image is effectively achieved by moving the position of the virtual viewpoint VP in the front-rear direction of the vehicle according to the direction in which the line of sight of the virtual viewpoint VP is directed. Can be improved. In the present embodiment, the image control unit 20a moves the position of the virtual viewpoint VP in the left-right direction as well as the front-rear direction of the vehicle 9 according to the direction in which the line of sight of the virtual viewpoint VP is directed. The visibility of the subject image that is displayed can be further effectively improved. Since the image control unit 20a moves the line of sight of the virtual viewpoint VP along the periphery of the passenger compartment of the vehicle, the vehicle body portion indicated by the vehicle image 90 included in the vehicle interior viewpoint image can be reduced and should be confirmed. The image of the subject can be enlarged.

  In addition, as described above, in the rotation mode M2, the image control unit 20a maintains the position of the virtual viewpoint VP while the plane viewing angle of the line of sight of the virtual viewpoint VP is changed, while the virtual viewpoint VP is moving while the position of the virtual viewpoint is moving. Maintains the planar viewing angle of the viewpoint. That is, the image control unit 20a does not change the planar viewing angle of the line of sight of the virtual viewpoint VP and the position of the virtual viewpoint VP in parallel, and maintains one when changing one. Thereby, since the movement of the line of sight of the virtual viewpoint VP can be simplified without being complicated, the in-vehicle viewpoint image continuously generated in the circulation mode M2 can easily show the surroundings of the vehicle 9.

<1-4-2. Adjustment of the virtual angle of depression>
By the way, the direction of the line of sight of the virtual viewpoint VP is defined by a planar viewing angle and a depression angle (a downward angle with respect to the horizontal direction). In the circulation mode M2, the image control unit 20a also adjusts the depression angle of the line of sight of the virtual viewpoint VP.

  FIG. 16 is a diagram showing a projection plane TS used for generating a virtual viewpoint image. As shown in FIG. 16, on the projection surface TS, an area (hereinafter referred to as “non-projected area”) on which the captured image data is not projected outside the area (hereinafter referred to as “projection area”) R1 onto which the captured image data is projected. It is referred to as a “projection area.”) R2 exists.

  As described above, the front camera 5F, the rear camera 5B, and the side cameras 5L and 5R have different depression angles of the optical axes and different distances in the direction away from the vehicle 9 that can be photographed. The left and right side cameras 5L and 5R can capture only a region relatively close to the vehicle 9, and cannot capture a region relatively distant from the vehicle 9. For this reason, as shown in FIG. 16, a non-projection region R2 having no data is generated in portions corresponding to the left and right sides of the vehicle 9 on the projection surface TS.

  When such a non-projection region R2 is not taken into consideration, there is a possibility that the image generation unit 22 generates an in-vehicle viewpoint image using a region including the non-projection region R2 on the projection surface TS in the circulation mode M2. . That is, the in-vehicle viewpoint image may include the non-projection region R2. Since there is no data in the non-projection area R2, in the in-vehicle viewpoint image including the non-projection area R2, a part of the area is a single color (generally black) without showing the image of the subject around the vehicle 9. . Therefore, the user who visually recognizes such an in-vehicle viewpoint image feels a great sense of incongruity.

  In order to cope with such a problem, the image control unit 20a according to the present embodiment sets the depression angle of the line of sight of the virtual viewpoint VP so that the in-vehicle viewpoint image does not include the non-projection area R2 on the projection surface TS in the rotation mode M2. adjust.

  FIG. 17 is a diagram illustrating the depression angle β of the line of sight of the virtual viewpoint VP adjusted by the image control unit 20a. On the projection surface TS, the size of the projection region R1 (the length in the direction away from the vehicle 9) is minimized and the size of the non-projection region R2 (away from the vehicle 9) on the left and right sides of the entire periphery of the vehicle 9. (Length in the direction) is maximized (see FIG. 16). When the image control unit 20a sets the depression angle β of the line of sight of the virtual viewpoint VP to a specific angle (for example, 60 °), the line of sight of the virtual viewpoint VP is directed to the left or right of the vehicle 9 as described above. In addition, it is possible to prevent the in-vehicle viewpoint image from including the non-projection region R2.

  For this reason, in the rotation mode M2, the image control unit 20a sets the depression angle β of the line of sight of the virtual viewpoint VP to this specific angle (60 °). Then, as illustrated in FIG. 17, the image control unit 20a does not change the line of sight of the virtual viewpoint VP even if the plane view angle of the line of sight of the virtual viewpoint VP changes so that the line of sight of the virtual viewpoint VP circulates around the vehicle 9. The depression angle β is constantly maintained at a specific angle (60 °). That is, the image control unit 20a sets the depression angle β of the line of sight of the virtual viewpoint VP to a specific angle (60 °) even when the line of sight of the virtual viewpoint VP is directed in any direction around the entire vehicle 9.

  FIG. 18 is a diagram illustrating a use region R10 used by the image generation unit 22 for generating an in-vehicle viewpoint image in the projection surface TS when the depression angle β of the line of sight of the virtual viewpoint VP is adjusted as illustrated in FIG. As shown in FIG. 18, the use region R <b> 10 is a region that is relatively close to the vehicle 9 and surrounds the periphery of the vehicle 9. The use area R10 includes only the projection area R1 with data, and does not include the non-projection area R2 without data.

  If the depression angle β of the line of sight of the virtual viewpoint VP is maintained at a specific angle (60 °), the in-vehicle viewpoint image is not projected in the non-projection area R2 even when the line of sight of the virtual viewpoint VP is directed in the direction that minimizes the size of the projection area R1. Not included. For this reason, naturally, even when the line of sight of the virtual viewpoint VP is directed in the other direction, the in-vehicle viewpoint image does not include the non-projection region R2. Therefore, even if the plane view angle of the visual line of the virtual viewpoint VP is changed so that the line of sight of the virtual viewpoint VP circulates around the vehicle 9 in the rotation mode M2, the in-vehicle viewpoint image does not always include the non-projection region R2. can do. As a result, it is possible to prevent the user who has viewed the in-vehicle viewpoint image from feeling uncomfortable.

  Further, since the image control unit 20a maintains the depression angle β of the line of sight of the virtual viewpoint VP in the circulation mode M2, the movement of the line of sight of the virtual viewpoint VP can be simplified without being complicated. For this reason, the in-vehicle viewpoint images continuously generated in the circulation mode M <b> 2 can easily show the surroundings of the vehicle 9.

<1-4-3. Flow of operation>
Next, an operation flow of the image display system 10 in the circulation mode M2 will be described. FIG. 19 is a diagram illustrating a flow of operations of the image display system 10 in the circulation mode M2.

  When the operation mode becomes the circulation mode M2, first, the image control unit 20a sets the virtual viewpoint VP in the vehicle interior (step S11). The image control unit 20a sets the planar viewing angle so that the line of sight of the virtual viewpoint VP faces the front of the vehicle 9 that is the initial direction, and sets the position of the virtual viewpoint VP to the front center position P1 that is the initial position. The image control unit 20a also sets the virtual viewpoint VP at a specific angle (60 °) in which the virtual viewpoint image does not include the non-projection region R2 even when the line of sight of the virtual viewpoint VP is directed to the left or right of the vehicle 9. Set the gaze angle of the line of sight.

  Next, each of the four cameras 5 provided in the vehicle 9 photographs the periphery of the vehicle 9. Then, the image acquisition unit 21 acquires four captured images respectively obtained by the four cameras 5 (step S12).

  Next, the image generation unit 22 generates an in-vehicle viewpoint image using the acquired four captured images (step S13). Then, the image adjustment unit 23 generates a display image DP including the generated in-vehicle viewpoint image, and the image output unit 24 outputs the display image DP to the display device 3 (step S14). Thereby, the display image DP including the in-vehicle viewpoint image is displayed on the display device 3.

  Next, the image control unit 20a changes the planar viewing angle of the line of sight of the virtual viewpoint VP or the position of the virtual viewpoint VP (step S16). And a process returns to step S12 again and the process similar to said step S12-S14 is repeated. Such processing (steps S12 to S14, S16) is repeated at a predetermined cycle (for example, 1/30 second cycle).

  In step S16, the image control unit 20a does not change the planar viewing angle of the line of sight of the virtual viewpoint VP and the position of the virtual viewpoint VP in parallel as described above, and maintains the other when one is changed ( (See FIG. 8.) Further, the image control unit 20a maintains the depression angle of the line of sight of the virtual viewpoint VP at a specific angle regardless of the planar viewing angle of the line of sight of the virtual viewpoint VP and the position of the virtual viewpoint VP (see FIG. 17).

  By such processing, the image generation unit 22 continuously generates in-vehicle viewpoint images while the image control unit 20a gradually changes the virtual viewpoint VP. As a result, in the display device 3, an animation showing the state of the periphery of the vehicle 9 is performed so as to go around the vehicle 9.

  When the line of sight of the virtual viewpoint VP goes around the vehicle 9 and the position of the virtual viewpoint VP returns to the front center position P1 that is the initial position (Yes in step S15), the operation of the image display system 10 in the rotation mode M2 is performed. finish.

<2. Second Embodiment>
Next, a second embodiment will be described. Since the configuration and operation of the image display system 10 of the second embodiment are substantially the same as those of the first embodiment, the following description will be focused on differences from the first embodiment.

  In the first embodiment, in the rotation mode M2, the image control unit 20a does not change the planar viewing angle of the line of sight of the virtual viewpoint VP and the position of the virtual viewpoint VP in parallel. On the other hand, in the second embodiment, the image control unit 20a changes the planar viewing angle of the line of sight of the virtual viewpoint VP and the position of the virtual viewpoint VP in parallel.

  FIG. 20 is a diagram illustrating a transition of the virtual viewpoint VP in the circulation mode M2 according to the second embodiment. Also in the second embodiment, as in the first embodiment, the image control unit 20a moves from the front center position P1 to the front center position P1 again via the positions P2 to P6. The position of the virtual viewpoint VP is linearly moved along the periphery of the passenger compartment 91 so as to go around the inside of the vehicle 91. However, the image control unit 20a continuously moves the position of the virtual viewpoint VP without stopping the position of the virtual viewpoint VP even at the positions P2, P3, P5, and P6.

  Further, the image control unit 20a continuously changes the planar viewing angle of the line of sight of the virtual viewpoint VP even during movement of the position of the virtual viewpoint VP (broken line arrow A15). Thereby, in the rotation mode M2, the planar viewing angle of the line of sight of the virtual viewpoint VP and the position of the virtual viewpoint VP are changed in parallel.

  Also in the second embodiment, in the circulation mode M2, the image control unit 20a moves the position of the virtual viewpoint VP according to the direction in which the line of sight of the virtual viewpoint VP is directed. That is, the image control unit 20a moves the position of the virtual viewpoint VP to the front of the passenger compartment 91 when the line of sight of the virtual viewpoint VP is directed to the front of the vehicle 9. Similarly, the image control unit 20a moves the position of the virtual viewpoint VP to the rear of the passenger compartment 91 when the line of sight of the virtual viewpoint VP is directed to the rear of the vehicle 9. Further, the image control unit 20a moves the position of the virtual viewpoint VP to the right side of the passenger compartment 91 when the line of sight of the virtual viewpoint VP is directed to the right side of the vehicle 9, and moves the line of sight of the virtual viewpoint VP to the left side of the vehicle 9. When moving to the left, the position of the virtual viewpoint VP is moved to the left of the passenger compartment 91. Thereby, the visibility of the subject image of the vehicle 9 included in the in-vehicle viewpoint image can be improved.

  As described above, in the second embodiment, both the planar viewing angle of the line of sight of the virtual viewpoint VP and the position of the virtual viewpoint VP are changed in parallel, so the line of sight of the virtual viewpoint circulates around the vehicle 9. Time can be shortened. That is, it is possible to relatively shorten the time during which an animation showing the state of the periphery of the vehicle 9 is performed so as to go around the vehicle 9 (for example, 15 seconds). Therefore, the user can check the entire surroundings of the vehicle 9 in a relatively short time.

<3. Third Embodiment>
Next, a third embodiment will be described. Since the configuration and operation of the image display system 10 of the third embodiment are substantially the same as those of the second embodiment, the following description will be focused on differences from the second embodiment.

  In the second embodiment, the image control unit 20 a linearly moves the position of the virtual viewpoint VP along the periphery of the passenger compartment 91. On the other hand, in the third embodiment, the image control unit 20a moves the position of the virtual viewpoint VP to a substantially elliptical shape. Also in the third embodiment, the image control unit 20a changes the viewing angle of the line of sight of the virtual viewpoint VP and the position of the virtual viewpoint VP in parallel.

  FIG. 21 is a diagram illustrating the transition of the virtual viewpoint VP in the circulation mode M2 according to the third embodiment. As shown in FIG. 21, in the third embodiment, as a route for moving the position of the virtual viewpoint VP, a substantially elliptical route whose major axis is along the front-rear direction of the vehicle 9 is the vehicle interior 91 of the vehicle 9. Set internally.

  First, the image control unit 20a sets the planar viewing angle so that the line of sight of the virtual viewpoint VP faces the front of the vehicle 9 that is the initial direction, and sets the position of the virtual viewpoint VP to the front center position P1 that is the initial position. To do.

  Next, the image control unit 20a moves the position of the virtual viewpoint VP rearward along the arc on the right side of the substantially elliptical route from the front center position P1 to the rear center position P4 (broken line arrow A6). Further, the image control unit 20a moves the position of the virtual viewpoint VP forward along the arc on the left side of the substantially elliptical route from the rear center position P4 to the front center position P1 (broken line arrow A7). Even during the movement of the position of the virtual viewpoint VP, the image control unit 20a continuously changes the planar viewing angle of the line of sight of the virtual viewpoint VP (broken arrow A16).

  Also in the third embodiment, in the rotation mode M2, the image control unit 20a moves the position of the virtual viewpoint VP according to the direction in which the line of sight of the virtual viewpoint VP is directed. That is, the image control unit 20a moves the position of the virtual viewpoint VP to the front of the passenger compartment 91 when the line of sight of the virtual viewpoint VP is directed to the front of the vehicle 9. Similarly, the image control unit 20a moves the position of the virtual viewpoint VP to the rear of the passenger compartment 91 when the line of sight of the virtual viewpoint VP is directed to the rear of the vehicle 9. Further, the image control unit 20a moves the position of the virtual viewpoint VP to the right side of the passenger compartment 91 when the line of sight of the virtual viewpoint VP is directed to the right side of the vehicle 9, and moves the line of sight of the virtual viewpoint VP to the left side of the vehicle 9. When moving to the left, the position of the virtual viewpoint VP is moved to the left of the passenger compartment 91. Thereby, the visibility of the subject image of the vehicle 9 included in the in-vehicle viewpoint image can be improved.

  As described above, in the third embodiment, since the position of the virtual viewpoint VP is moved in a substantially elliptical shape, the distance to move the position of the virtual viewpoint VP can be shortened. Therefore, it is possible to further shorten the time for the line of sight of the virtual viewpoint VP to go around the vehicle 9, and the user can check the entire surroundings of the vehicle 9 in a shorter time.

  Further, the major axis of the substantially oval shape that is the route for moving the position of the virtual viewpoint VP is along the longitudinal direction of the vehicle 9. For this reason, the position of the virtual viewpoint VP can be greatly moved in front of and behind the passenger compartment 91 of the vehicle 9. Therefore, even when the line of sight of the virtual viewpoint VP is directed forward or rearward of the vehicle 9, the vehicle body portion indicated by the vehicle image 90 included in the in-vehicle viewpoint image can be reduced, and the subject image to be confirmed can be enlarged. can do.

<4. Fourth Embodiment>
Next, a fourth embodiment will be described. Since the configuration and operation of the image display system 10 of the fourth embodiment are substantially the same as those of the first embodiment, the following description will be focused on differences from the first embodiment.

  In the first embodiment, the image control unit 20 a moves the position of the virtual viewpoint VP in the left-right direction as well as the front-rear direction of the vehicle 9. On the other hand, in the fourth embodiment, the image control unit 20 a moves the position of the virtual viewpoint VP only in the front-rear direction of the vehicle 9.

  FIG. 22 is a diagram illustrating the transition of the virtual viewpoint VP in the rotation mode M2 according to the fourth embodiment. As shown in FIG. 22, in the fourth embodiment, a straight route along the front-rear direction of the vehicle 9 is set inside the passenger compartment 91 of the vehicle 9 as a route for moving the position of the virtual viewpoint VP. .

  First, the image control unit 20a sets the planar viewing angle so that the line of sight of the virtual viewpoint VP faces the front of the vehicle 9 that is the initial direction, and sets the position of the virtual viewpoint VP to the front center position P1 that is the initial position. To do.

  Next, the image control unit 20a continuously changes the planar viewing angle of the line of sight of the virtual viewpoint VP so that the line of sight of the virtual viewpoint VP is directed rightward along the left-right direction of the vehicle 9 (broken arrow A17). In other words, the image control unit 20a changes the planar viewing angle of the line of sight of the virtual viewpoint VP by 90 ° by rotating it clockwise while maintaining the position of the virtual viewpoint VP at the front center position P1.

  When the line of sight of the virtual viewpoint VP is directed to the right side of the vehicle 9, the image control unit 20a follows the straight route from the front center position P1 to the rear center position P4 while maintaining the planar viewing angle of the line of sight of the virtual viewpoint VP. The position of the virtual viewpoint VP is moved backward (broken line arrow A8).

  When the position of the virtual viewpoint VP is moved to the rear center position P4, the image control unit 20a stops moving the position of the virtual viewpoint VP, and then the line of sight of the virtual viewpoint VP turns to the left along the left-right direction of the vehicle 9. In this way, the planar viewing angle of the line of sight of the virtual viewpoint VP is continuously changed (broken line arrow A18). That is, the image control unit 20a changes the planar viewing angle of the line of sight of the virtual viewpoint VP by 180 ° by rotating it to the right while maintaining the position of the virtual viewpoint VP at the position P4.

  When the line of sight of the virtual viewpoint VP is directed to the left of the vehicle 9, the image control unit 20a follows the straight route from the rear center position P4 to the front center position P1 while maintaining the planar viewing angle of the line of sight of the virtual viewpoint VP. The position of the virtual viewpoint VP is moved forward (broken line arrow A8).

  When the position of the virtual viewpoint VP is moved to the front center position P1, the image control unit 20a stops the movement of the position of the virtual viewpoint VP, and then the line of sight of the virtual viewpoint VP faces forward along the front-rear direction of the vehicle 9. The plane viewing angle of the line of sight of the virtual viewpoint VP is continuously changed (broken line arrow A19). That is, the image control unit 20a changes the planar viewing angle of the line of sight of the virtual viewpoint VP by 90 ° by rotating it to the right while maintaining the position of the virtual viewpoint VP at the front center position P1.

  Also in the fourth embodiment, in the rotation mode M2, the image control unit 20a moves the position of the virtual viewpoint VP according to the direction in which the line of sight of the virtual viewpoint VP is directed. That is, the image control unit 20a moves the position of the virtual viewpoint VP to the front of the passenger compartment 91 when the line of sight of the virtual viewpoint VP is directed to the front of the vehicle 9. Similarly, the image control unit 20a moves the position of the virtual viewpoint VP to the rear of the passenger compartment 91 when the line of sight of the virtual viewpoint VP is directed to the rear of the vehicle 9. Thereby, the visibility of the subject image of the vehicle 9 included in the in-vehicle viewpoint image can be improved.

  As described above, in the fourth embodiment, since the position of the virtual viewpoint VP is linearly moved only in the front-rear direction, the distance to move the position of the virtual viewpoint VP can be shortened. Therefore, it is possible to shorten the time for the line of sight of the virtual viewpoint VP to circulate around the vehicle 9, and the user can check the entire surroundings of the vehicle 9 in a relatively short time.

  In FIG. 22, the image control unit 20a does not change the planar viewing angle of the line of sight of the virtual viewpoint VP and the position of the virtual viewpoint VP in parallel. However, as in the second embodiment, The planar viewing angle of the line of sight of the virtual viewpoint VP and the position of the virtual viewpoint VP may be changed in parallel.

<5. Fifth embodiment>
Next, a fifth embodiment will be described. Since the configuration and operation of the image display system 10 of the fifth embodiment are substantially the same as those of the first embodiment, the following description will focus on differences from the first embodiment.

  In the first embodiment, in the rotation mode M2, the image control unit 20a maintains the depression angle of the line of sight of the virtual viewpoint VP at a specific angle. On the other hand, in the fifth embodiment, the image control unit 20a changes the depression angle of the line of sight of the virtual viewpoint VP according to the size of the projection area R1 in the direction in which the line of sight of the virtual viewpoint VP is directed on the projection plane TS. To do.

  As described above, on the projection plane TS, the size of the projection region R1 (the length in the direction away from the vehicle 9) is minimized on the left and right sides of the vehicle 9. Therefore, the size of the projection region R1 (the length in the direction away from the vehicle 9) is larger in front of or behind the vehicle 9 on the projection plane TS than in the left and right sides of the vehicle 9. For this reason, when the line of sight of the virtual viewpoint VP is directed forward or rearward of the vehicle 9, the in-vehicle viewpoint image is not projected even if the line of sight of the virtual viewpoint VP is directed upward compared to the case where it is directed leftward or rightward. Does not include R2. The image control unit 20a of the fifth embodiment changes the depression angle of the line of sight of the virtual viewpoint VP in accordance with the size of the projection area R1 in such a range that the in-vehicle viewpoint image does not include the non-projection area R2.

  FIG. 23 is a diagram illustrating the depression angle β of the line of sight of the virtual viewpoint VP in the rotation mode M2 according to the fifth embodiment. As shown in FIG. 23, the image control unit 20a, when turning the line of sight of the virtual viewpoint VP toward the left or right side of the vehicle 9, is the depression angle of the line of sight of the virtual viewpoint VP, as in the first embodiment. β is set to a specific angle (60 °). On the other hand, the image control unit 20a makes the depression angle β of the line of sight of the virtual viewpoint VP smaller than the specific angle (60 °) when the line of sight of the virtual viewpoint VP is directed forward or backward of the vehicle 9.

  First, the image control unit 20a sets the planar viewing angle so that the line of sight of the virtual viewpoint VP faces the front of the vehicle 9, and sets the position of the virtual viewpoint VP to the front center position P1 that is the initial position. Thus, when the position of the virtual viewpoint VP is the front center position P1, the image control unit 20a sets the depression angle β of the line of sight of the virtual viewpoint VP to the smallest minimum angle (for example, 30 °).

  FIG. 24 is a diagram illustrating an example of the in-vehicle viewpoint image CP35 generated by the image generation unit 22 when the line of sight of the virtual viewpoint VP faces the front of the vehicle 9 and the position of the virtual viewpoint VP is the front center position P1. Compared to the in-vehicle viewpoint image CP31 in the first embodiment in FIG. 9, the in-vehicle viewpoint image CP35 in the fifth embodiment in FIG. 24 can show a relatively far subject away from the vehicle 9. .

  The depression angle β of the line of sight of the virtual viewpoint VP in FIG. 9 is a specific angle (60 °), whereas the depression angle β of the line of sight of the virtual viewpoint VP in FIG. 24 is the minimum angle (30 °). Therefore, the line of sight of the virtual viewpoint VP in FIG. 24 is upward from the line of sight of the virtual viewpoint VP in FIG. Since the size of the projection area R1 is relatively large in front of the vehicle 9 on the projection plane TS, the in-vehicle viewpoint image is not projected in the non-projection area even when the depression angle β of the line of sight of the virtual viewpoint VP is set to the minimum angle (30 °). Does not include R2.

  Returning to FIG. 23, next, the image control unit 20 a moves the position of the virtual viewpoint VP from the front center position P <b> 1 toward the front right end position P <b> 2. During this movement, the image control unit 20a gradually changes the depression angle β of the line of sight of the virtual viewpoint VP from the minimum angle (30 °) to the specific angle (60 °). That is, the image control unit 20a gradually turns the line of sight of the virtual viewpoint VP downward.

  Next, the image control unit 20a turns the line of sight of the virtual viewpoint VP to the right of the vehicle 9, and moves the position of the virtual viewpoint VP from the front right end position P2 to the rear right end position P3. On the right side of the vehicle 9 on the projection surface TS, the size of the projection region R1 is the smallest. Therefore, during this movement, the image control unit 20a maintains the depression angle β of the line of sight of the virtual viewpoint VP at a specific angle (60 °) so that the in-vehicle viewpoint image does not include the non-projection region R2.

  Next, the image control unit 20a turns the line of sight of the virtual viewpoint VP toward the rear of the vehicle 9, and moves the position of the virtual viewpoint VP from the position P3 at the rear right end toward the rear center position P4. During this movement, the image control unit 20a gradually changes the depression angle β of the line of sight of the virtual viewpoint VP from the specific angle (60 °) to the minimum angle (30 °). That is, the image control unit 20a gradually turns the line of sight of the virtual viewpoint VP upward. When the position of the virtual viewpoint VP is the rear center position P4, the depression angle β of the line of sight of the virtual viewpoint VP is the minimum angle (30 °).

  FIG. 25 is a diagram illustrating an example of the in-vehicle viewpoint image CP36 generated by the image generation unit 22 when the line of sight of the virtual viewpoint VP is behind the vehicle 9 and the position of the virtual viewpoint VP is the rear center position P4. The in-vehicle viewpoint image CP36 of the fifth embodiment in FIG. 25 can show a relatively far subject away from the vehicle 9 compared to the in-vehicle viewpoint image CP33 of the first embodiment in FIG. .

  The depression angle β of the line of sight of the virtual viewpoint VP in FIG. 11 is a specific angle (60 °), whereas the depression angle β of the line of sight of the virtual viewpoint VP in FIG. 25 is the minimum angle (30 °). Therefore, the line of sight of the virtual viewpoint VP in FIG. 25 is upward from the line of sight of the virtual viewpoint VP in FIG. Since the size of the projection region R1 is relatively large behind the vehicle 9 on the projection surface TS, the in-vehicle viewpoint image is not projected in the non-projection region even when the depression angle β of the line of sight of the virtual viewpoint VP is set to the minimum angle (30 °). Does not include R2.

  Returning to FIG. 23, next, the image control unit 20a moves the position of the virtual viewpoint VP from the rear center position P4 toward the rear left end position P5. During this movement, the image control unit 20a gradually changes the depression angle β of the line of sight of the virtual viewpoint VP from the minimum angle (30 °) to the specific angle (60 °). That is, the image control unit 20a gradually turns the line of sight of the virtual viewpoint VP downward.

  Next, the image control unit 20a turns the line of sight of the virtual viewpoint VP toward the left of the vehicle 9, and moves the position of the virtual viewpoint VP from the position P5 at the rear left end toward the position P6 at the front left end. On the left side of the vehicle 9 on the projection surface TS, the size of the projection region R1 is minimum. Therefore, during this movement, the image control unit 20a maintains the depression angle β of the line of sight of the virtual viewpoint VP at a specific angle (60 °) so that the in-vehicle viewpoint image does not include the non-projection region R2.

  Next, the image control unit 20a turns the line of sight of the virtual viewpoint VP toward the front of the vehicle 9, and moves the position of the virtual viewpoint VP from the front left end position P6 toward the front center position P1. During this movement, the image control unit 20a gradually changes the depression angle β of the line of sight of the virtual viewpoint VP from the specific angle (60 °) to the minimum angle (30 °). That is, the image control unit 20a gradually turns the line of sight of the virtual viewpoint VP upward. When the position of the virtual viewpoint VP returns to the front center position P1, which is the initial position, the depression angle β of the line of sight of the virtual viewpoint VP becomes the minimum angle (30 °).

  FIG. 26 is a diagram illustrating a use region R11 that is used by the image generation unit 22 to generate an in-vehicle viewpoint image in the projection surface TS when the depression angle β of the line of sight of the virtual viewpoint VP is adjusted as illustrated in FIG. As shown in FIG. 26, the use region R <b> 11 is a region that is long in the front-rear direction of the vehicle 9. The use region R11 includes only the projection region R1 with data, and does not include the non-projection region R2 without data. Therefore, even if the planar viewing angle of the line of sight of the virtual viewpoint VP is changed so that the line of sight of the virtual viewpoint VP circulates around the vehicle 9 in the rotation mode M2, the in-vehicle viewpoint image does not include the non-projection region R2. be able to. As a result, it is possible to prevent the user who has viewed the in-vehicle viewpoint image from feeling uncomfortable.

  As described above, in the fifth embodiment, the image control unit 20a determines the size of the projection area R1 in the direction in which the line of sight of the virtual viewpoint VP is directed on the projection plane TS in a range where the in-vehicle viewpoint image does not include the non-projection area R2. In response to the change, the depression angle β of the line of sight of the virtual viewpoint VP is changed. Therefore, the in-vehicle viewpoint image can indicate a relatively distant subject away from the vehicle 9 while preventing the user who has viewed the in-vehicle viewpoint image from feeling uncomfortable. The image control unit 20a makes the depression angle β of the line of sight of the virtual viewpoint VP smaller than the specific angle (60 °) when the line of sight of the virtual viewpoint VP is directed forward or rearward of the vehicle 9. Therefore, the in-vehicle viewpoint image can indicate a relatively distant subject existing in the traveling direction of the vehicle 9.

<6. Sixth Embodiment>
Next, a sixth embodiment will be described. Since the configuration and operation of the image display system 10 of the sixth embodiment are substantially the same as those of the first embodiment, the following description will focus on differences from the first embodiment.

  In the first embodiment, in the rotation mode M2, the image control unit 20a always maintains the depression angle of the line of sight of the virtual viewpoint VP at a specific angle. On the other hand, in the sixth embodiment, the image control unit 20a, when the position of the virtual viewpoint VP is other than the front center position P1 and the rear center position P4, is the virtual viewpoint as in the first embodiment. The depression angle of the line of sight of the VP is maintained at a specific angle (60 °). On the other hand, when the position of the virtual viewpoint VP is the front center position P1 and the rear center position P4, the depression angle of the line of sight of the virtual viewpoint VP is made smaller than the specific angle (60 °).

  FIG. 27 is a diagram illustrating the depression angle β of the line of sight of the virtual viewpoint VP in the rotation mode M2 according to the sixth embodiment.

  First, the image control unit 20a sets the planar viewing angle so that the line of sight of the virtual viewpoint VP faces the front of the vehicle 9, and sets the position of the virtual viewpoint VP to the front center position P1 that is the initial position. Thus, when the position of the virtual viewpoint VP is the front center position P1, the image control unit 20a sets the depression angle β of the line of sight of the virtual viewpoint VP to the smallest minimum angle (30 °). As a result, the image generation unit 22 generates an in-vehicle viewpoint image CP35 that shows a relatively distant subject away from the vehicle 9, as shown in FIG.

  Next, the image control unit 20a gradually decreases the depression angle β of the line of sight of the virtual viewpoint VP from the smallest minimum angle (30 °) to the specific angle (60 °) while maintaining the position of the virtual viewpoint VP at the front center position P1. To change. That is, the image control unit 20a gradually turns the line of sight of the virtual viewpoint VP downward.

  When the depression angle β of the line of sight of the virtual viewpoint VP is set to a specific angle (60 °), the image control unit 20a changes the plane view angle of the line of sight of the virtual viewpoint VP and the position of the virtual viewpoint VP as in the first embodiment. To do. Thereby, the image control unit 20a moves the position of the virtual viewpoint VP from the front center position P1 to the rear center position P4 via the positions P2 and P3. In addition, the image control unit 20 a changes the planar viewing angle so that the line of sight of the virtual viewpoint VP is directed to the right of the vehicle 9 and further toward the rear of the vehicle 9. During this movement, the image control unit 20a maintains the depression angle β of the line of sight of the virtual viewpoint VP at a specific angle (60 °).

  When the position of the virtual viewpoint VP is moved to the rear center position P4, the image control unit 20a sets the depression angle β of the line of sight of the virtual viewpoint VP to a specific angle (60 °) while maintaining the position of the virtual viewpoint VP at the rear center position P4. From 1 to the minimum angle (30 °). That is, the image control unit 20a gradually turns the line of sight of the virtual viewpoint VP upward. When the depression angle β of the line of sight of the virtual viewpoint VP becomes the minimum angle (30 °), the image generation unit 22 generates an in-vehicle viewpoint image CP36 that shows a relatively distant subject away from the vehicle 9, as shown in FIG. .

  Next, the image control unit 20a gradually changes the depression angle β of the line of sight of the virtual viewpoint VP from the minimum angle (30 °) to the specific angle (60 °) while maintaining the position of the virtual viewpoint VP at the rear center position P4. To do. That is, the image control unit 20a gradually returns the line of sight of the virtual viewpoint VP downward.

  When the depression angle β of the line of sight of the virtual viewpoint VP is set to a specific angle (60 °), the image control unit 20a changes the plane view angle of the line of sight of the virtual viewpoint VP and the position of the virtual viewpoint VP as in the first embodiment. To do. Accordingly, the image control unit 20a moves the position of the virtual viewpoint VP from the rear center position P4 to the front center position P1 via the positions P5 and P6. In addition, the image control unit 20 a changes the planar viewing angle so that the line of sight of the virtual viewpoint VP is directed to the left of the vehicle 9 and further toward the front of the vehicle 9. During this movement, the image control unit 20a maintains the depression angle β of the line of sight of the virtual viewpoint VP at a specific angle (60 °).

  As described above, in the sixth embodiment, when the position of the virtual viewpoint VP is other than the specific positions P1 and P4, the depression angle of the line of sight of the virtual viewpoint VP is maintained at the specific angle, while the position of the virtual viewpoint is In the case of specific positions P1 and P4, the depression angle of the line of sight of the virtual viewpoint is made smaller than the specific angle. For this reason, while the movement of the line of sight of the virtual viewpoint VP is simplified, the in-vehicle viewpoint image can indicate a relatively distant subject away from the vehicle 9 when the position of the virtual viewpoint VP is the specific positions P1 and P4. That is, the in-vehicle viewpoint image can indicate a relatively distant subject existing in the traveling direction of the vehicle 9.

<7. Modification>
Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications are possible. Below, such a modification is demonstrated. All the forms including the above-described embodiment and the form described below can be appropriately combined.

  In the above embodiment, the image generation unit 22 generates a virtual viewpoint image using all four captured images of the four cameras 5, but 1 to 3 selected from the four captured images. A virtual viewpoint image may be generated using the captured image (at least one captured image). In this case, the image generation unit 22 may project only the captured image used for generating the virtual viewpoint image among the four captured images onto the projection surface TS.

  In the above embodiment, when the planar viewing angle of the visual line of the virtual viewpoint VP is changed so that the line of sight of the virtual viewpoint VP goes around the vehicle 9, the line of sight of the virtual viewpoint VP goes around the vehicle 9. However, it may be less than one turn or more than one turn. In the above embodiment, the line of sight of the virtual viewpoint VP circulates around the vehicle 9 clockwise, but the line of sight of the virtual viewpoint VP may circulate around the vehicle 9 counterclockwise.

  Moreover, in the said embodiment, although the part corresponded except vehicle body bottom face of the vehicle image 90 was transparent or semi-transparent, it may be non-transparent.

  In the above embodiment, the front camera 5F, the rear camera 5B, and the side cameras 5L and 5R have been described as having different angles of the optical axis, but the present invention is not limited to this. For at least one of a plurality of cameras that acquire a plurality of captured images, if the depression angle of the optical axis is different from that of other cameras, the technique for adjusting the depression angle of the line of sight of the virtual viewpoint described in the above embodiment is preferably used Applicable.

  Moreover, as a method for changing the planar viewing angle of the line of sight of the virtual viewpoint VP and the position of the virtual viewpoint VP, different methods have been described in the first, second, third, and fourth embodiments. May be selected by the user. Further, as methods for changing the depression angle of the line of sight of the virtual viewpoint VP, different methods have been described in the first, fifth, and sixth embodiments, but the user can select which method to execute. May be.

  In addition, the function described as one block in the above embodiment is not necessarily realized by a single physical element, and may be realized by distributed physical elements. Further, the functions described as a plurality of blocks in the above embodiments may be realized by a single physical element. In addition, even if the device in the vehicle and the device outside the vehicle share processing related to any one function and exchange information by communication between these devices, the one function can be realized as a whole. Good.

  In addition, it has been described that all or part of the functions described as being realized by software by executing the program in the above embodiment may be realized by an electrical hardware circuit or by a hardware circuit. All or part of the functions may be realized by software. Further, the function described as one block in the above-described embodiment may be realized by cooperation of software and hardware.

2 image generation device 5 camera 9 vehicle 10 image display system 20a image control unit 21 image acquisition unit 22 image generation unit 90 vehicle image 91 vehicle compartment VP virtual viewpoint

Claims (14)

An image generation device used in a vehicle,
Acquisition means for acquiring a plurality of captured images obtained by a plurality of cameras;
Using at least one of the plurality of photographed images and the vehicle body data of the vehicle, a virtual viewpoint image indicating the periphery of the vehicle and the vehicle body of the vehicle viewed from a virtual viewpoint in the vehicle interior of the vehicle is continuously generated. Generating means for
Control means for changing the angle of the virtual viewpoint line of sight in plan view so that the virtual viewpoint line of sight circulates around the vehicle, and moving the position of the virtual viewpoint in the front-rear direction of the vehicle;
Equipped with a,
The image generation apparatus characterized in that the control means moves the position of the virtual viewpoint in accordance with a direction in which the line of sight of the virtual viewpoint is directed . An image generation device used in a vehicle,
Acquisition means for acquiring a plurality of captured images obtained by a plurality of cameras;
Using at least one of the plurality of photographed images and the vehicle body data of the vehicle, a virtual viewpoint image indicating the periphery of the vehicle and the vehicle body of the vehicle viewed from a virtual viewpoint in the vehicle interior of the vehicle is continuously generated. Generating means for
The angle in the planar view of the virtual viewpoint line of sight is changed so that the line of sight of the virtual viewpoint circulates around the vehicle, and image generation in which the position of the virtual viewpoint is moved in accordance with the direction of the line of sight is controlled. Control means;
An image generation apparatus comprising: The image generation apparatus according to claim 1 or 2,
The image generating apparatus according to claim 1, wherein the control means moves the position of the virtual viewpoint in the front-rear direction and the left-right direction of the vehicle along a periphery of a compartment of the vehicle. In the image generation device according to any one of claims 1 to 3,
The control means includes
While changing the angle of the line of sight of the virtual viewpoint in the plan view, the position of the virtual viewpoint is maintained,
An image generating apparatus characterized by maintaining an angle of the line of sight of the virtual viewpoint in the plan view during movement of the position of the virtual viewpoint. In the image generation device according to any one of claims 1 to 3,
The image generation apparatus characterized in that the control means changes the angle of the line of sight of the virtual viewpoint in the plan view and the position of the virtual viewpoint in parallel. The image generation apparatus according to claim 1 or 2,
The image generation apparatus according to claim 1, wherein the control means moves the position of the virtual viewpoint in a substantially elliptical shape with a long axis along the front-rear direction of the vehicle. An image generation device used in a vehicle,
Acquisition means for acquiring a plurality of captured images obtained by a plurality of cameras;
Using at least one of the plurality of photographed images and the vehicle body data of the vehicle, a virtual viewpoint image indicating the periphery of the vehicle and the vehicle body of the vehicle viewed from a virtual viewpoint in the vehicle interior of the vehicle is continuously generated. Generating means for
Control means for changing the angle of the virtual viewpoint line of sight in plan view so that the virtual viewpoint line of sight circulates around the vehicle, and moving the position of the virtual viewpoint in the front-rear direction of the vehicle;
With
The generation unit projects the data of the plurality of captured images onto a virtual projection plane, generates the virtual viewpoint image using a partial region of the projection plane,
At least one of the plurality of cameras is different from other cameras in the depression angle of the optical axis,
The control means adjusts the depression angle of the line of sight of the virtual viewpoint so that the virtual viewpoint image does not include a non-projection area outside the projection area on which the data is projected on the projection plane. An image generating device. The image generation apparatus according to claim 7.
The control means sets the non-projection area at a specific angle that does not include the virtual viewpoint image when the line of sight of the virtual viewpoint is directed in a direction that minimizes the projection area on the projection plane. An image generating apparatus characterized by maintaining a depression angle. The image generation apparatus according to claim 7.
The image generating apparatus, wherein the control unit changes a depression angle of the line of sight of the virtual viewpoint according to a size of the projection area in a direction in which the line of sight of the virtual viewpoint is directed on the projection plane. The image generation apparatus according to claim 7.
The control means includes
When the position of the virtual viewpoint is other than a specific position, a specific angle at which the virtual viewpoint image does not include the non-projection area when the line of sight of the virtual viewpoint is directed in a direction in which the projection area is minimum on the projection plane And maintaining the depression angle of the line of sight of the virtual viewpoint,
When the position of the virtual viewpoint is the specific position, the depression angle of the line of sight of the virtual viewpoint is made smaller than the specific angle. An image generation device used in a vehicle,
Acquisition means for acquiring a plurality of captured images obtained by a plurality of cameras;
Data of the plurality of photographed images is projected onto a virtual projection plane, and a virtual viewpoint image indicating the periphery of the vehicle viewed from a virtual viewpoint in a vehicle interior of the vehicle is continuously used by using a partial region of the projection plane. Generating means for generating automatically,
Control means for changing an angle of the virtual viewpoint line of sight in plan view so that the line of sight of the virtual viewpoint circulates around the vehicle;
With
At least one of the plurality of cameras is different from other cameras in the depression angle of the optical axis,
The control means adjusts the depression angle of the line of sight of the virtual viewpoint so that the virtual viewpoint image does not include a non-projection area outside the projection area on which the data is projected on the projection plane. An image generating device. An image display system used in a vehicle,
An image generation device according to any one of claims 1 to 11,
A display device for displaying a virtual viewpoint image generated by the image generation device;
An image display system comprising: An image generation method used in a vehicle,
(A) acquiring a plurality of captured images obtained by a plurality of cameras;
(B) Using at least one of the plurality of captured images and the vehicle body data of the vehicle, continuous virtual viewpoint images indicating the periphery of the vehicle and the vehicle body of the vehicle viewed from a virtual viewpoint in the vehicle interior of the vehicle Generating a process,
(C) changing the angle in plan view of the visual line of the virtual viewpoint so that the visual line of the virtual viewpoint goes around the vehicle, and moving the position of the virtual viewpoint in the front-rear direction of the vehicle; ,
Equipped with a,
The step (c) includes moving the position of the virtual viewpoint in accordance with a direction in which the line of sight of the virtual viewpoint is directed . An image generation method used in a vehicle,
(A) acquiring a plurality of captured images obtained by a plurality of cameras;
(B) Projecting the data of the plurality of captured images onto a virtual projection plane, and using a partial area of the projection plane, a virtual viewpoint showing the periphery of the vehicle as viewed from a virtual viewpoint inside the vehicle interior of the vehicle A step of continuously generating images;
(C) changing the angle of the virtual viewpoint line of sight in plan view so that the line of sight of the virtual viewpoint circulates around the vehicle;
With
At least one of the plurality of cameras is different from other cameras in the depression angle of the optical axis,
The step (c) adjusts the depression angle of the line of sight of the virtual viewpoint so that the virtual viewpoint image does not include a non-projection area outside the projection area where the data is projected on the projection plane. A featured image generation method.

Images