JP2015184839A - Image generation device, image display system, and image generation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique with which it is possible to call a user's attention to even an object not included in a target area to be expanded, as well as an object included in the target area.SOLUTION: In an image display system according to the present invention, an image acquisition unit acquires a plurality of photographed images obtained by a plurality of cameras that photograph the periphery of a vehicle, and an image generation unit generates a composite image indicating the periphery of the vehicle looked from virtual viewpoints by using the plurality of photographed images. Then the image generation unit generates a partial expanded image CPa that is one composite image which, while indicating the whole region of the entire periphery of the vehicle, expands the subject included in a partial target region TR of the whole region, and which indicates the subject not included in the target region TR without expanding it. Accordingly, a user visually recognizing the partial expanded image CPa can, while confirming an object included in the target region TR in detail, direct the user's attention to an object not included in the target region TR at the same time.

Description

  The present invention relates to a technique for generating an image showing the periphery of a vehicle.

  2. Description of the Related Art Conventionally, an image display system that generates an image showing the periphery of a vehicle such as an automobile and displays this image on a display device in the vehicle is known. By using such an image display system, a user (typically a driver) can check the state of the periphery of the vehicle almost in real time.

  In recent years, an image display system has also been proposed in which a plurality of captured images obtained by a plurality of cameras are combined to generate a combined image showing the periphery of the vehicle viewed from a virtual viewpoint, and the combined image is displayed ( For example, see Patent Document 1.) In such an image display system, for example, it is possible to generate a composite image (overhead image) showing the entire periphery of the vehicle as if looking down from directly above the vehicle. By visually recognizing such a composite image, the user can check the overall surroundings of the vehicle.

JP 2012-124610 A

  A user who uses the image display system as described above may desire to check a part of the periphery of the vehicle in more detail, for example, behind the vehicle while checking the entire periphery of the vehicle. For this reason, as shown in FIG. 29, a technique for displaying a display image 81 including two images of a composite image 82 showing the entire periphery of the vehicle and an enlarged image 83 showing an enlarged portion of the periphery of the vehicle side by side. Is known (for example, see Patent Document 1). In the example of FIG. 29, the vehicle is shown as a vehicle image 85, and the enlarged image 83 shows the rear of the vehicle in an enlarged manner.

  A user who visually recognizes the display image 81 as shown in FIG. 29 can confirm both the composite image 82 and the enlarged image 83. However, the user usually pays attention to either the composite image 82 or the enlarged image 83 with a specific gravity. For this reason, when the user is paying attention to the enlarged image 83, the user does not notice the object when the object approaches the vehicle outside the area indicated by the enlarged image 83 (for example, in front of the vehicle). there is a possibility.

  The present invention has been made in view of the above problems, and provides a technique that allows a user to pay attention to an object that is not included in the target area as well as an object that is included in the target area to be enlarged. With the goal.

  In order to solve the above-mentioned problem, the invention of claim 1 is an image generation device used in a vehicle, and obtains a plurality of photographed images obtained by a plurality of cameras that photograph the periphery of the vehicle; Generating means for generating a composite image showing the periphery of the vehicle viewed from a virtual viewpoint using the plurality of captured images, the generating means showing the entire area of the entire periphery of the vehicle The subject included in a part of the target area is enlarged, and a partly enlarged image that is one of the combined images shown without enlargement of the subject not included in the target area is generated.

  According to a second aspect of the present invention, in the image generation apparatus according to the first aspect, the generation unit affixes the data of the plurality of captured images to a virtual surface corresponding to the periphery of the vehicle, and The composite image is generated using the pasted data, and the partially enlarged image is generated by changing the coordinates of the portion corresponding to the target region on the virtual plane.

  According to a third aspect of the present invention, in the image generation apparatus according to the second aspect, the virtual surface is a curved surface set in a virtual three-dimensional space.

  According to a fourth aspect of the present invention, in the image generating apparatus according to the third aspect, the generating unit changes the coordinates only in a direction along the ground contact surface of the vehicle.

  The invention according to claim 5 is the image generation apparatus according to claim 3 or 4, wherein the generation means can set the virtual viewpoint at an arbitrary position in the three-dimensional space.

  According to a sixth aspect of the present invention, in the image generating apparatus according to any one of the first to fifth aspects, the partially enlarged image shows a subject in the vicinity of the periphery of the target area in a compressed manner.

  According to a seventh aspect of the present invention, the image generating apparatus according to any one of the first to sixth aspects further comprises setting means for setting the target area in accordance with a user operation.

  The invention according to claim 8 is the image generation apparatus according to any one of claims 1 to 6, wherein the target region is determined according to a detection result of a detection unit that detects a position of an object existing in the vicinity of the vehicle. Setting means for setting.

  The invention of claim 9 is an image display system used in a vehicle, wherein the image generation device according to any one of claims 1 to 8 and the partially enlarged image generated by the image generation device are displayed. And a display device for displaying.

  The invention of claim 10 is an image generation method used in a vehicle, wherein (a) a step of acquiring a plurality of captured images obtained by a plurality of cameras that capture the periphery of the vehicle; Generating a composite image showing the periphery of the vehicle as viewed from a virtual viewpoint using the plurality of captured images, wherein the step (b) shows the entire area of the entire periphery of the vehicle. A subject included in a part of the target region is enlarged, and a partially enlarged image that is one of the combined images shown without enlargement of a subject not included in the target region is generated.

  According to the first to tenth aspects of the present invention, a partially enlarged image is generated in which the subject included in the partial target region of the entire region is enlarged while showing the entire region around the entire vehicle. The user who visually recognizes this partially enlarged image can simultaneously pay attention to an object not included in the target area while confirming in detail the object included in the target area.

  In particular, according to the second aspect of the present invention, a partially enlarged image can be generated by a simple method of changing some coordinates of the virtual surface.

  In particular, according to the third aspect of the present invention, even when the virtual surface is a curved surface, a partially enlarged image can be generated by a simple method of changing some coordinates of the virtual surface.

  In particular, according to the invention of claim 4, since the coordinates are changed only in the direction along the ground contact surface of the vehicle, the amount of calculation for generating a partially enlarged image can be greatly reduced.

  In particular, according to the invention of claim 5, even if the virtual viewpoint is set at any position in the three-dimensional space, a partially enlarged image viewed from the virtual viewpoint can be generated.

  In particular, according to the invention of claim 6, since the partially enlarged image shows the subject in the vicinity of the periphery of the target region in a compressed manner, the continuity between the subject included in the target region and the subject not included in the target region is shown. Can be held.

  In particular, according to the invention of claim 7, the target area is set according to the user's operation, so that the user can confirm the desired subject in detail.

  In particular, according to the invention of claim 8, the target area is set according to the detection result of the detection means, so that the user can confirm in detail the object existing in the vicinity of the vehicle.

FIG. 1 is a diagram illustrating a configuration of an image display system according to the first embodiment. FIG. 2 is a diagram illustrating directions in which four cameras capture images. FIG. 3 is a diagram illustrating a method for generating a composite image. FIG. 4 is a diagram illustrating a method for generating a composite image. FIG. 5 is a diagram illustrating an example of a composite image. FIG. 6 is a diagram for explaining the flick operation. FIG. 7 is a diagram illustrating an example of a composite image. FIG. 8 is a diagram illustrating an example of a composite image. FIG. 9 is a diagram for explaining the pinch-out operation. FIG. 10 is a diagram illustrating an example of a partially enlarged image. FIG. 11 is a diagram illustrating an example of a partially enlarged image. FIG. 12 is a diagram illustrating an example of a partially enlarged image. FIG. 13 is a diagram illustrating a flow of processing of the image display system. FIG. 14 is a diagram illustrating a technique for changing the coordinates of the vertices of the virtual surface. FIG. 15 is a diagram illustrating a technique for changing the coordinates of the vertices of the virtual surface. FIG. 16 is a diagram illustrating a technique for changing the coordinates of the vertices of the virtual surface. FIG. 17 is a diagram illustrating a method for changing the coordinates of the vertices of the virtual surface. FIG. 18 is a diagram for explaining a method of changing the coordinates of the vertices of the virtual surface. FIG. 19 is a diagram illustrating a method for changing the coordinates of the vertices of the virtual surface. FIG. 20 is a diagram for explaining a method of changing the coordinates of the vertices of the virtual surface. FIG. 21 is a diagram for explaining a method of changing the coordinates of the vertices of the virtual surface. FIG. 22 is a diagram illustrating a technique for changing the coordinates of the vertices of the virtual surface. FIG. 23 is a diagram for explaining a method of changing the coordinates of the vertices of the virtual surface. FIG. 24 is a diagram illustrating a technique for changing the coordinates of the vertices of the virtual surface. FIG. 25 is a diagram for explaining a method of changing the coordinates of the vertices of the virtual plane. FIG. 26 is a diagram illustrating a configuration of an image display system according to the fourth embodiment. FIG. 27 is a diagram mainly illustrating a configuration of the object detection device. FIG. 28 is a diagram illustrating a position where the clearance sonar is disposed. FIG. 29 is a diagram illustrating an example of a display image including a composite image.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<1. First Embodiment>
<1-1. Configuration>
FIG. 1 is a diagram illustrating a configuration of an image display system 10 according to the first embodiment. This image display system 10 is used in a vehicle (in this embodiment, an automobile), and has a function of generating an image showing an area around the vehicle and displaying it in the vehicle interior. By using this image display system 10, a user (typically a driver) of the image display system 10 can grasp the state around the vehicle in almost real time.

  As shown in the figure, the image display system 10 mainly includes a plurality of cameras 5, an image generation device 2, a display device 3, and operation buttons 4. Each of the plurality of cameras 5 captures the periphery of the vehicle to acquire a captured image, and inputs the acquired captured image to the image generation device 2. The image generation device 2 uses a plurality of captured images to generate a composite image showing the periphery of the vehicle viewed from a virtual viewpoint. The display device 3 displays the composite image generated by the image generation device 2. The operation button 4 receives a user operation.

  Each of the plurality of cameras 5 includes a lens and an image sensor, and electronically obtains a captured image showing the periphery of the vehicle. The multiple cameras 5 include a front camera 5F, a rear camera 5B, a left side camera 5L, and a right side camera 5R. These four cameras 5F, 5B, 5L, and 5R are arranged at different positions in the vehicle 9 and photograph different directions around the vehicle 9.

  FIG. 2 is a diagram illustrating directions in which the four cameras 5F, 5B, 5L, and 5R are respectively photographed. The front camera 5F is provided at the front end of the vehicle 9, and the optical axis 5Fa is directed in the straight traveling direction of the vehicle 9. The rear camera 5 </ b> B is provided at the rear end of the vehicle 9, and its optical axis 5 </ b> Ba is directed in the direction opposite to the straight traveling direction of the vehicle 9. The left side camera 5L is provided on the left side mirror 93L on the left side, and its optical axis 5La is directed to the left side of the vehicle 9. The right side camera 5 </ b> R is provided on the right side mirror 93 </ b> R on the right side, and its optical axis 5 </ b> Ra is directed to the right side of the vehicle 9.

  A wide-angle lens such as a fisheye lens is adopted as the lens of these cameras 5F, 5B, 5L, and 5R, and each camera 5F, 5B, 5L, and 5R has an angle of view α of 180 degrees or more. For this reason, it is possible to photograph the entire periphery of the vehicle 9 by using the four cameras 5F, 5B, 5L, and 5R.

  Returning to FIG. 1, the display device 3 includes a thin display panel such as a liquid crystal display, and displays various types of information and images. The display device 3 is arranged on an instrument panel or the like of the vehicle 9 so that the user can visually recognize the screen. The display device 3 may be disposed in the same housing as the image generation device 2 and integrated with the image generation device 2, or may be a separate device from the image generation device 2.

  The display device 3 includes a touch panel 31 that receives a user's touch operation on the screen. The touch panel 31 is a capacitance type, and can accept a multi-touch operation such as a flick operation or a pinch-out operation.

  The operation button 4 is an operation member that receives a user operation. The operation button 4 is provided, for example, on the steering wheel of the vehicle 9 and mainly receives an operation from the driver.

  The user can perform various operations on the image display system 10 via the touch panel 31 and the operation buttons 4. When a user operation is performed on either the touch panel 31 or the operation button 4, an operation signal indicating the content of the operation is input to the image generation device 2.

  The image generation device 2 is an electronic device that can perform various types of image processing, and includes an image acquisition unit 21, an image generation unit 22, and an image output unit 24.

  The image acquisition unit 21 acquires captured images respectively obtained by the four cameras 5F, 5B, 5L, and 5R. The image acquisition unit 21 has an image processing function such as a function of converting an analog captured image into a digital captured image. The image acquisition unit 21 performs predetermined image processing on the acquired captured image and inputs the processed captured image to the image generation unit 22. When the four cameras 5F, 5B, 5L, and 5R acquire digital captured images, the image acquisition unit 21 does not have to have a function of converting an analog captured image into a digital captured image.

  The image generation unit 22 is a hardware circuit that performs image processing for generating a composite image. The image generation unit 22 combines the four captured images acquired by the four cameras 5 to generate a combined image that shows the periphery of the vehicle 9 viewed from the virtual viewpoint. The image generation unit 22 attaches a plurality of captured image data to a virtual surface corresponding to the periphery of the vehicle 9 and generates a composite image using the data attached to the virtual surface. Details of the method for generating the composite image will be described later.

  The image output unit 24 outputs the composite image generated by the image generation unit 22 to the display device 3 and causes the display device 3 to display the composite image. As a result, a composite image showing the periphery of the vehicle 9 viewed from the virtual viewpoint is displayed on the display device 3.

  The image generation device 2 further includes a control unit 20, an operation reception unit 25, a signal reception unit 26, and a storage unit 27. The control unit 20 is a microcomputer including a CPU, a RAM, a ROM, and the like, for example, and comprehensively controls the entire image generation apparatus 2.

  The operation reception unit 25 receives an operation signal output from the touch panel 31 and the operation button 4 when the user performs an operation. Thereby, the operation reception part 25 receives a user's operation. The operation reception unit 25 inputs the received operation signal to the control unit 20.

  The signal receiving unit 26 receives a signal from another device provided in the vehicle 9 separately from the image generating device 2 and inputs the signal to the control unit 20. For example, the signal receiving unit 26 receives a signal indicating the shift position, which is the position of the shift lever of the transmission of the vehicle 9, from the shift sensor 91. Based on this signal, the control unit 20 can determine whether the traveling direction of the vehicle 9 is forward or backward.

  The storage unit 27 is, for example, a nonvolatile memory such as a flash memory, and stores various types of information. The storage unit 27 stores a program 27a as firmware and various data used by the image generation unit 22 to generate a composite image. The data used for generating such a composite image includes a correspondence table 27b and an enlargement parameter 27c. The correspondence table 27 b is data in a table format stored in advance in the storage unit 27. On the other hand, the expansion parameter 27c is not data stored in advance in the storage unit 27 but data stored as necessary.

  Various functions of the control unit 20 are realized by the CPU performing arithmetic processing according to the program 27 a stored in the storage unit 27. The display control unit 20a and the parameter setting unit 20b shown in the drawing are a part of functional units realized by the CPU performing arithmetic processing according to the program 27a.

  The display control unit 20 a controls the image generation unit 22 and the image output unit 24 to generate various composite images, and displays the composite images on the display device 3. For example, the display control unit 20 a changes the position of the virtual viewpoint related to the composite image in accordance with the user operation received by the operation receiving unit 25.

  The parameter setting unit 20b sets the expansion parameter 27c according to the user operation received by the operation receiving unit 25. The parameter setting unit 20 b stores the set expansion parameter 27 c in the storage unit 27. The display control unit 20a controls the image generation unit 22 based on the enlargement parameter 27c, so that the image generation unit 22 enlarges and shows a part of the entire region while showing the entire region around the vehicle. An image is generated (details will be described later).

<1-2. Generation of composite image>
Next, a method will be described in which the image generation unit 22 generates a composite image that shows the surroundings of the vehicle 9 viewed from a virtual viewpoint. FIG. 3 is a diagram illustrating a method in which the image generation unit 22 generates a composite image.

  When shooting is performed with each of the front camera 5F, the rear camera 5B, the left side camera 5L, and the right side camera 5R, four captured images SF and SB showing the front, rear, left side, and right side of the vehicle 9, respectively. , SL, SR are acquired. These four photographed images SF, SB, SL, and SR include data around the entire vehicle 9 (subject image).

  First, the image generation unit 22 pastes data (subject images) included in the four captured images SF, SB, SL, and SR onto a virtual surface TS that is a curved surface set in a virtual three-dimensional space ( Project).

  The virtual surface TS is a surface corresponding to an area around the vehicle 9. The virtual surface TS has, for example, a substantially hemispherical shape (a bowl shape), and a central area (bottom part of the bowl) is defined as a vehicle area R0 where the vehicle 9 is located. The image generation unit 22 does not attach the captured image data to the vehicle area R0 in the virtual surface TS, and attaches the captured image data to the area outside the vehicle area R0.

  The virtual surface TS is divided into a large number of sections (polygons) in a mesh shape. Each section constituting the virtual surface TS is a polygon having three or four vertices. Each section is associated with one of the four captured images SF, SB, SL, and SR. The image generation unit 22 pastes the data of each area of the four captured images SF, SB, SL, and SR as a texture on the section of the virtual surface TS corresponding to the area. The area in the captured image and the section of the virtual surface TS to which the data is to be pasted are associated with each other by the correspondence table 27b stored in advance in the storage unit 27.

  As illustrated in FIG. 4, the image generation unit 22 affixes the data of the captured image SF of the front camera 5 </ b> F to the section of the front portion GF corresponding to the front of the vehicle 9 in the virtual plane TS. In addition, the image generation unit 22 affixes the data of the captured image SB of the rear camera 5B to the section of the rear portion GB corresponding to the rear of the vehicle 9 in the virtual plane TS. Furthermore, the image generation unit 22 pastes the data of the captured image SL of the left side camera 5L on the section of the left side portion GL corresponding to the left side of the vehicle 9 on the virtual plane TS, and the right side of the vehicle 9 on the virtual plane TS. The data of the captured image SR of the right side camera 5R is pasted on the section of the right side portion GR corresponding to.

  Returning to FIG. 3, when the data of the captured image is pasted on the virtual surface TS in this way, the image generation unit 22 then virtually configures a vehicle model that shows the three-dimensional shape of the vehicle 9. This vehicle model is arranged in the vehicle region R0 that is the position of the vehicle 9 in the three-dimensional space where the virtual plane TS is set.

  Next, the image generation unit 22 sets the virtual viewpoint VP for the three-dimensional space in which the virtual surface TS is set under the control of the display control unit 20a. This virtual viewpoint VP is defined by the position and the direction of the line of sight. The image generation unit 22 can set the virtual viewpoint VP at an arbitrary position in the three-dimensional space with an arbitrary line-of-sight direction.

  Next, the image generation unit 22 generates a composite image CP using a partial region of the virtual surface TS corresponding to the set virtual viewpoint VP. That is, the image generation unit 22 cuts out, as an image, data pasted on a region included in a predetermined viewing angle from the virtual viewpoint VP in the virtual surface TS. The image generation unit 22 performs rendering on the vehicle model according to the virtual viewpoint VP, and superimposes the resulting vehicle image 90 on the cut-out image. Thereby, the image generation part 22 produces | generates the synthetic | combination image CP which shows the periphery of the vehicle 9 and the vehicle 9 seen from the virtual viewpoint VP.

  For example, as shown in FIG. 3, when a virtual viewpoint VPa is set with the position directly above the vehicle 9 and the line of sight directed toward the center of the vehicle 9, a composite image (overhead image) that overlooks the vehicle 9 and the periphery of the vehicle 9. ) CP1 can be generated. Further, when the virtual viewpoint VPb is set with the position at the left rear of the vehicle 9 and the line of sight toward the center of the vehicle 9, a composite image CP2 showing the vehicle 9 viewed from the left rear of the vehicle 9 and the periphery of the vehicle 9 is displayed. Can be generated. In the present embodiment, the display control unit 20 a controls the image generation unit 22 so that the line of sight of the virtual viewpoint VP is directed toward the center of the vehicle 9 regardless of the position of the virtual viewpoint VP.

<1-3. Display of composite image>
Such a composite image CP is appropriately displayed on the display device 3 according to the state of the vehicle 9 and the user's operation under the control of the display control unit 20a. Normally, the composite image CP is displayed on the display device 3 when the user operates the operation button 4. On the other hand, when the shift position is “R” and the traveling direction of the vehicle 9 is backward, the composite image CP is displayed on the display device 3 without the user operating the operation button 4.

  The display control unit 20a sets the position of the virtual viewpoint VP immediately above the vehicle 9, for example, when the composite image CP is displayed on the display device 3 for the first time. As a result, as shown in FIG. 5, a bird's-eye view image, which is a composite image CP showing the entire area around the vehicle 9 so as to look down from directly above the vehicle 9, is generated and displayed on the display device 3.

  Further, when the composite image CP is displayed on the display device 3, as shown in FIG. 6, when the user performs a flick operation on the screen of the display device 3, the position of the virtual viewpoint VP is changed. . The flick operation is an operation of moving the finger F in one direction after the user touches the screen with the finger F. When the user performs a flick operation, the display control unit 20a moves the position of the virtual viewpoint VP in the direction opposite to the direction of the flick operation.

  For example, when the composite image CP shown in FIG. 5 is displayed and the user performs a flick operation toward the front of the vehicle 9 as shown in FIG. 6, the display control unit 20a displays the virtual viewpoint VP. Is moved to the rear of the vehicle 9. As a result, as shown in FIG. 7, a composite image CP showing the entire area around the entire vehicle 9 as viewed from the rear of the vehicle 9 is generated and displayed on the display device 3.

  When the composite image CP shown in FIG. 7 is displayed and the user performs a flick operation toward the left side of the vehicle 9, the display control unit 20 a sets the position of the virtual viewpoint VP to the right of the vehicle 9. Move towards. As a result, as shown in FIG. 8, a composite image CP indicating the entire area around the vehicle 9 as viewed from the right side of the vehicle 9 is generated and displayed on the display device 3.

  Thus, since the position of the virtual viewpoint VP is changed in response to the user's flick operation, the user can check the entire region around the vehicle 9 from the desired position.

  When the composite image CP is displayed on the display device 3, the user can also perform a pinch-out operation on the screen of the display device 3 as shown in FIG. 9 in addition to the flick operation. The pinch-out operation is an operation of expanding the interval between the fingers F after the user touches the screen with the two fingers F. When the user performs a pinch-out operation, a composite image CP is generated that shows an enlarged subject included in the target region TR specified by the pinch-out operation.

  When the user performs a pinch-out operation, the parameter setting unit 20b sets the enlargement parameter 27c according to the user's operation. The parameter setting unit 20b sets the positions of the two points where the two fingers F are first touched and the amount of movement by which the user spreads the two fingers F as the enlargement parameter 27c. The positions of the two points first touched by the two fingers F define the target area TR to be enlarged. Of the entire region around the vehicle 9, a region corresponding to a circle passing through two points first touched by two fingers F is a target region TR. In addition, the amount of movement by which the user spreads two fingers F defines the enlargement ratio when enlarging the subject included in the target region TR. The parameter setting unit 20b substantially sets the target region TR by setting the enlargement parameter 27c in this way.

  When the enlargement parameter 27c is set in this way, the composite image CP is generated by the image generation unit 22 based on the enlargement parameter 27c. This composite image CP shows the entire area around the entire vehicle 9 while enlarging a subject included in a part of the target area TR of the entire area, and showing a subject not included in the target area TR without expanding. This is “partially enlarged image” CPa.

  For example, when the composite image CP shown in FIG. 5 is displayed and the user performs a pinch-out operation with respect to the rear of the vehicle 9 as shown in FIG. Is set as the target region TR. As a result, as shown in FIG. 10, a partially enlarged image CPa showing an enlarged subject included in the target region TR is generated and displayed on the display device 3.

  As shown in FIG. 10, the partially enlarged image CPa shows the entire area around the entire vehicle 9. In addition, as can be seen by comparing FIG. 10 and FIG. 5, the partially enlarged image CPa shows the subject included in the target region TR in an enlarged manner. On the other hand, the partially enlarged image CPa shows the size of the subject not included in the target region TR and the subject not included in the target region TR without being enlarged or reduced. For this reason, the user who visually recognizes such a partially enlarged image CPa can pay attention to the entire region around the vehicle 9 while confirming in detail the object included in the target region TR. That is, the user can simultaneously pay attention to an object that is not included in the target region TR.

  Further, the partially enlarged image CPa does not necessarily show all the subjects included in the target region TR in an enlarged manner. The partially enlarged image CPa shows the subject near the center of the target region TR in a greatly enlarged manner, while compressing the subject near the periphery of the target region TR. Therefore, continuity between the subject included in the partially enlarged image CPa and the subject not included in the partially enlarged image CPa is maintained. As a result, even if the subject included in the target region TR is enlarged, the data around the enlarged subject can be included in the partially enlarged image CPa without being lost.

  Even when such a partially enlarged image CPa is displayed on the display device 3, when the user performs a flick operation on the screen of the display device 3, the position of the virtual viewpoint VP is changed. Even when the position of the virtual viewpoint VP is changed, the partially enlarged image CPa is displayed on the display device 3.

  For example, when a partially enlarged image CPa shown in FIG. 10 is displayed and the user performs a flick operation toward the front of the vehicle 9, the display control unit 20a sets the position of the virtual viewpoint VP to the vehicle. Move 9 backwards. As a result, as shown in FIG. 11, a partially enlarged image CPa indicating the entire area around the entire vehicle 9 as viewed from the rear of the vehicle 9 is generated and displayed on the display device 3.

  When the partially enlarged image CPa shown in FIG. 11 is displayed and the user performs a flick operation toward the left side of the vehicle 9, the display control unit 20a sets the position of the virtual viewpoint VP to the vehicle 9 Move to the right. As a result, as shown in FIG. 12, a partially enlarged image CPa that shows the entire area around the entire vehicle 9 as viewed from the right side of the vehicle 9 is generated and displayed on the display device 3.

  As shown in FIGS. 11 and 12, even when the position of the virtual viewpoint VP is changed, a partially enlarged image CPa showing an enlarged subject included in the target region TR set behind the vehicle 9 is displayed. Will be. Since the position of the virtual viewpoint VP is changed in response to the user's flick operation, the user can confirm in detail the object included in the target region TR from the desired position.

<1-4. Process flow>
Next, a processing flow of the image display system 10 will be described. FIG. 13 is a diagram illustrating a processing flow of the image display system 10 when the composite image CP is displayed. When displaying the composite image CP, the process shown in FIG. 13 is repeated at a predetermined cycle (for example, 1/30 second cycle).

  First, each of the four cameras 5 provided in the vehicle 9 photographs the periphery of the vehicle 9. Then, the image acquisition unit 21 acquires four captured images respectively obtained by the four cameras 5 (step S11).

  Next, the image generation unit 22 pastes the acquired four captured image data on the virtual surface TS of the virtual three-dimensional space (step S12). In accordance with the correspondence table 27b stored in the storage unit 27, the image generation unit 22 pastes the data of the four photographed images on each of a number of sections constituting the virtual surface TS.

  Next, the display control unit 20a determines whether or not the enlargement parameter 27c is stored in the storage unit 27 (step S13).

  When the enlargement parameter 27c is stored in the storage unit 27 (Yes in step S13), the target area TR to be enlarged is set. Therefore, in this case, in order to generate a partially enlarged image CPa, the image generation unit 22 changes the coordinates of the portion corresponding to the target region TR in the virtual surface TS (step S14). Thereby, the data (namely, the image of the subject included in the target area TR) attached to the portion corresponding to the target area TR of the virtual surface TS is enlarged. A method for changing the coordinates of the virtual surface TS in this way will be described later.

  On the other hand, when the enlargement parameter 27c is not stored in the storage unit 27 (No in step S13), such a change in the coordinates of the virtual surface TS is not performed.

  Next, the image generation unit 22 sets a virtual viewpoint VP for the three-dimensional space, and cuts out the data pasted on the virtual surface TS based on the virtual viewpoint VP as a composite image CP (step S15). If the coordinates of the virtual surface TS have been changed in step S14, the image generation unit 22 cuts out the composite image CP in this way, thereby partially enlarging the image CPa that shows the subject included in the target region TR. Is generated.

  Next, the image output unit 24 outputs the composite image CP generated by the image generation unit 22 to the display device 3 (step S16). As a result, the composite image CP is displayed on the display device 3. When the partially enlarged image CPa is generated, the partially enlarged image CPa is displayed on the display device 3.

<1-5. Changing virtual plane coordinates>
Next, a method will be described in which the image generation unit 22 changes the coordinates of the portion corresponding to the target region TR in the virtual surface TS in order to generate the partially enlarged image CPa.

  As described above, the virtual surface TS is composed of a number of sections. The position of each vertex of this section is represented by the coordinates of a three-dimensional orthogonal coordinate system (XYZ) set in a virtual three-dimensional space. As shown in FIG. 3, the X-axis direction and the Y-axis direction are set in the horizontal direction along the ground plane of the vehicle 9. Further, the Z-axis direction is set to a vertical direction orthogonal to the ground contact surface of the vehicle 9. The image generation unit 22 changes the coordinates of the vertices of such a virtual surface TS.

  14 to 19 are diagrams for explaining a method of changing the coordinates of the vertex of the virtual surface TS. 14 to 19 show a cross section of the virtual surface TS viewed from the horizontal direction.

  Each point P arranged along the virtual surface TS shown in FIG. 14 represents the coordinates before the change of the vertex. Among the vertices of the virtual surface TS, the coordinates of the vertices included in the portion corresponding to the target region TR (hereinafter referred to as “change target portion”) are changed.

  As shown in FIG. 15, it is assumed that a part from the point Pa to the point Pb in the virtual surface TS is designated as the change target part and r is designated as the enlargement ratio based on the enlargement parameter 27c.

  First, a reference point O (Xo, Yo, Zo) that is the center of the change target portion is obtained. The reference point O is obtained as a point where a straight line Ls connecting the center point Pc of the points Pa and Pb and the position of the virtual viewpoint VP intersects the virtual surface TS.

  Hereinafter, the straight line Ls connecting the position of the virtual viewpoint VP and the reference point O is referred to as “visual line”, and the points Pa and Pb are referred to as “end points” of the change target portion. The distance from the line of sight Ls to the end points Pa and Pb corresponds to the radius of the change target portion, and hereinafter, the radius of the change target portion is assumed to be d. In this description, for the sake of simplicity, the reference point O and the end points Pa and Pb overlap with the point P representing the coordinates of the vertex, but these do not necessarily overlap with the point P.

  Next, as shown in FIG. 16, the point P included in the change target portion is moved in the direction orthogonal to the visual line Ls to be a point P ′.

  FIG. 17 is a diagram for explaining a method of paying attention to one point P and moving the point P to the point P ′. FIG. 17 shows an enlarged view of the range indicated by the symbol A in FIG.

  First, a unit vector along the visual line Ls from the reference point O to the position of the virtual viewpoint VP is defined as a vector u, and the vector u is expressed by the following equation (1).

また、基準点Ｏから点Ｐに向かうベクトルＯＰとベクトルｕとの内積の結果となる値をｍとすると、値ｍは次の数２で表される。

Further, when a value resulting from the inner product of the vector OP and the vector u from the reference point O toward the point P is m, the value m is expressed by the following formula 2.

また、点Ｐを通り視直線Ｌｓに直交する直線と視直線Ｌｓとの交点をＨとすると、この交点Ｈの座標（Ｈｘ，Ｈｙ，Ｈｚ）は、基準点Ｏの座標（Ｘｏ，Ｙｏ，Ｚｏ）、値ｍ、及び、ベクトルｕを用いて、次の数３で求められる。

If the intersection of the straight line Ls that passes through the point P and is orthogonal to the viewing line Ls is H, the coordinates (Hx, Hy, Hz) of the intersecting point H are the coordinates (Xo, Yo, Zo) of the reference point O. ), The value m, and the vector u.

したがって、点Ｐから視直線Ｌｓまでの距離Ｒは、点Ｐの座標（ｘ，ｙ，ｚ）と交点Ｈの座標（Ｈｘ，Ｈｙ，Ｈｚ）とを用いて、次の数４で求められる。

Therefore, the distance R from the point P to the line of sight Ls is obtained by the following equation 4 using the coordinates (x, y, z) of the point P and the coordinates (Hx, Hy, Hz) of the intersection H.

点Ｐに関する拡大増加率Ｗは、この距離Ｒと、変更対象部分の半径ｄと、拡大率ｒとを用いて、次の数５により求められる。拡大増加率Ｗは、距離Ｒが小さいほど大きな値となる。ただし、距離Ｒが０の場合（点Ｐが基準点Ｏの場合）は、拡大増加率Ｗは０とされる。

The enlargement increase rate W with respect to the point P is obtained by the following equation 5 using the distance R, the radius d of the change target portion, and the enlargement rate r. The enlargement increase rate W increases as the distance R decreases. However, when the distance R is 0 (when the point P is the reference point O), the enlargement increase rate W is 0.

次に、点Ｐの移動先となる点Ｐ’から視直線Ｌｓまでの距離Ｒ’が、次の数６により求められる。なお、距離Ｒ’が変更対象部分の半径ｄを超える場合は、距離Ｒ’を半径ｄとする。

Next, the distance R ′ from the point P ′ to which the point P is moved to the visual line Ls is obtained by the following equation (6). When the distance R ′ exceeds the radius d of the change target portion, the distance R ′ is set as the radius d.

また、交点Ｈから点Ｐに向かうベクトルＨＰを次の数７で表現する。

A vector HP from the intersection H to the point P is expressed by the following equation (7).

したがって、点Ｐの移動先となる点Ｐ’の座標（ｘ’，ｙ’，ｚ’）は、交点Ｈの座標（Ｈｘ，Ｈｙ，Ｈｚ）、距離Ｒ、距離Ｒ’、及び、ベクトルＨＰを用いて、次の数８で求められる。

Therefore, the coordinates (x ′, y ′, z ′) of the point P ′ to which the point P is moved are the coordinates (Hx, Hy, Hz) of the intersection H, the distance R, the distance R ′, and the vector HP. And obtained by the following equation (8).

このような演算は、変更対象部分に含まれる全ての点Ｐ（基準点Ｏを除く）に関して行われる。これにより、図１６に示すように、変更対象部分に含まれる全ての点Ｐ（基準点Ｏを除く）が点Ｐ’に移動される。点Ｐから点Ｐ’までの移動量は、点Ｐから視直線Ｌｓまでの距離Ｒに応じて決定される。視直線Ｌｓまでの距離Ｒが小さい点Ｐほど移動量が大きくなり、視直線Ｌｓまでの距離Ｒが大きな点Ｐほど移動量が小さくなる。

Such a calculation is performed for all points P (except the reference point O) included in the change target portion. Thereby, as shown in FIG. 16, all the points P (except the reference point O) included in the change target portion are moved to the point P ′. The amount of movement from the point P to the point P ′ is determined according to the distance R from the point P to the visual line Ls. The amount of movement increases as the point P has a smaller distance R to the visual line Ls, and the amount of movement decreases as the point P has a larger distance R to the visual line Ls.

  When the point P is moved to the point P ′ in this way, as shown in FIG. 18, the point P ′ is further moved to the point P ″ on the virtual surface TS. The point P ″ It is an intersection of a straight line along the viewing straight line Ls and the virtual plane TS. As a result, as shown in FIG. 19, the virtual surface TS is in a state where a point P ″ where the point P included in the change target portion is moved and a point P which is not included in the change target portion exists. Point P ″ represents the coordinates after the change of the vertex included in the change target portion.

  By moving the point P to the point P ″ in this way, the coordinates of the vertices included in the change target portion are changed. By such a change in the coordinates of the vertices of the virtual surface TS, the points attached to the virtual surface TS The size of the data (subject image) is also changed, as described above, since the amount of movement is larger at the point P where the distance R to the visual line Ls is smaller, the subject image near the center of the change target portion is greatly enlarged. On the other hand, since the movement amount is small at the point P where the distance R to the visual line Ls is large, the image of the subject near the periphery of the change target portion is compressed. While the subject near the center of the region TR is shown greatly enlarged, a partially enlarged image CPa showing the subject near the periphery of the target region TR is generated.

  As described above, in the image display system 10 according to the present embodiment, the image acquisition unit 21 acquires a plurality of captured images obtained by the plurality of cameras 5 that capture the periphery of the vehicle 9, and the image generation unit 22 includes a plurality of images. A composite image CP showing the periphery of the vehicle 9 viewed from the virtual viewpoint VP is generated using the captured image. Then, the image generation unit 22 enlarges the subject included in the target region TR that is a part of the entire region while showing the entire region around the vehicle 9, and does not enlarge the subject that is not included in the target region TR. A partially enlarged image CPa that is one composite image CP shown in FIG. Therefore, the user who visually recognizes this partially enlarged image CPa can simultaneously pay attention to an object not included in the target region TR while confirming in detail the object included in the target region TR.

  In addition, the image generation unit 22 pastes data of a plurality of captured images on the virtual surface TS corresponding to the periphery of the vehicle 9 and generates a composite image CP using the data pasted on the virtual surface TS. Then, the image generation unit 22 generates a partially enlarged image CPa by changing the coordinates of the change target portion corresponding to the target region TR in the virtual surface TS. For this reason, the partially enlarged image CPa can be generated by a simple technique of changing some coordinates of the virtual surface TS. Further, even when the position of the virtual viewpoint VP is changed, it is possible to generate a partially enlarged image CPa viewed from the changed virtual viewpoint VP.

  Further, the image generation unit 22 considers the position of the virtual viewpoint VP and changes the coordinates of the vertex of the virtual surface TS toward the orthogonal direction of the visual line Ls. For this reason, regardless of the position of the virtual viewpoint VP, the image of the subject included in the target region TR can be enlarged in a direction that can be viewed from the virtual viewpoint VP.

<2. Second Embodiment>
Next, a second embodiment will be described. Since the configuration and operation of the image display system 10 of the second embodiment are substantially the same as those of the first embodiment, the following description will be focused on differences from the first embodiment.

  The image display system 10 according to the second embodiment is different from the first embodiment only in a method in which the image generation unit 22 changes the coordinates of the vertices of the virtual surface TS. In the first embodiment, the position of the virtual viewpoint VP is considered when changing the coordinates of the vertex of the virtual surface TS. On the other hand, in the second embodiment, the amount of calculation is reduced by not considering the position of the virtual viewpoint VP when changing the coordinates of the vertex of the virtual surface TS.

  Also in the present embodiment, the coordinates before the change of the vertex of the virtual surface TS are arranged along the virtual surface TS as shown by a point P shown in FIG. 20 to 23 are diagrams for explaining a method of changing the coordinates of the vertex of the virtual surface TS in the present embodiment. 20 to 23 show a cross section of the virtual surface TS viewed from the horizontal direction.

  As shown in FIG. 20, it is assumed that a part from the end point Pa to the end point Pb in the virtual surface TS is designated as the change target part based on the enlargement parameter 27c, and r is designated as the enlargement ratio.

  Also in the present embodiment, first, the reference point O (Xo, Yo, Zo) is obtained. In this case, the position of the virtual viewpoint VP (the line of sight Ls) is not considered, and only the horizontal coordinate is considered. That is, only the X coordinate and the Y coordinate are considered, and the center point between the end point Pa and the end point Pb is derived. A point on the virtual plane TS having the X and Y coordinates of the central point is set as a reference point O. In the present embodiment, the horizontal distance from the reference point O to the end points Pa and Pb is the radius d of the portion to be changed.

  Next, as shown in FIG. 21, the point P included in the part to be changed is moved in the direction extending the straight line from the reference point O to the point P to be a point P ′. The amount of movement from the point P to the point P ′ takes into account the horizontal distance from the point P to the reference point O.

  Assuming that the horizontal distance from the point P to the reference point O is R, the distance R is the horizontal coordinate (x, y) of the point P and the horizontal coordinate (Xo, Yo) of the reference point O. And obtained by the following equation (9).

点Ｐに関する拡大増加率Ｗは、この距離Ｒと、変更対象部分の半径ｄと、拡大率ｒとを用いて、上述した数５により求められる。したがって、本実施の形態でも、拡大増加率Ｗは、距離Ｒが小さいほど大きな値となる。ただし、距離Ｒが０の場合（点Ｐが基準点Ｏの場合）は、拡大増加率Ｗは０とされる。

The enlargement increase rate W with respect to the point P is obtained by the above-described Expression 5 using the distance R, the radius d of the change target portion, and the enlargement rate r. Therefore, also in the present embodiment, the enlargement increase rate W increases as the distance R decreases. However, when the distance R is 0 (when the point P is the reference point O), the enlargement increase rate W is 0.

  Further, assuming that the horizontal distance from the point P ′ to which the point P is moved to the reference point O is R ′, the distance R ′ can be obtained by the above-described equation (6). Then, the coordinates (x ′, y ′, z ′) of the point P ′, which is the destination of the point P, are obtained using the coordinates (Xo, Yo, Zo), the distance R, and the distance R ′ of the reference point O. The following equation 10 is obtained.

このような演算は、変更対象部分に含まれる全ての点Ｐ（基準点Ｏを除く）に関して行われ、図２１に示すように、変更対象部分に含まれる全ての点Ｐ（基準点Ｏを除く）が点Ｐ’に移動される。本実施の形態でも、基準点Ｏまでの距離Ｒが小さい点Ｐほど移動量が大きくなり、基準点Ｏまでの距離Ｒが大きな点Ｐほど移動量が小さくなる。

Such a calculation is performed for all points P (excluding the reference point O) included in the change target part, and as shown in FIG. 21, all points P (excluding the reference point O) included in the change target part. ) Is moved to point P ′. Also in the present embodiment, the amount of movement increases as the distance P to the reference point O decreases, and the amount of movement decreases as the distance P to the reference point O increases.

  When the point P is moved to the point P ′ in this way, the point P ′ is further moved to the point P ″ on the virtual surface TS as shown in FIG. 22. The point P ″ is changed from the point P ′. A straight line extending in the horizontal or vertical direction intersects the virtual plane TS at the shortest. As a result, as shown in FIG. 23, the virtual surface TS is in a state where the point P ″ where the point P included in the change target portion is moved and the point P which is not included in the change target portion exists. Point P ″ represents the coordinates after the change of the vertex included in the change target portion.

  By moving the point P to the point P ″ in this way, the coordinates of the vertices included in the change target portion are changed. By such a change in the coordinates of the vertices of the virtual surface TS, the points attached to the virtual surface TS are pasted. The size of the data (subject image) is also changed, so that, in the present embodiment, the subject near the center of the target region TR is greatly enlarged, while the subject near the periphery of the target region TR is compressed. Thus, a partially enlarged image CPa shown in FIG.

  In the present embodiment, the distance only in the horizontal direction is considered as the distance R for deriving the enlargement increase rate W without considering the position of the virtual viewpoint VP. For this reason, it is possible to reduce the amount of calculation for generating the partially enlarged image CPa.

<3. Third Embodiment>
Next, a third embodiment will be described. Since the configuration and operation of the image display system 10 of the third embodiment are substantially the same as those of the first embodiment, the following description will be focused on differences from the first embodiment.

  The image display system 10 according to the third embodiment is different from the first embodiment only in a method in which the image generation unit 22 changes the coordinates of the vertices of the virtual surface TS. In the first embodiment, the vertex coordinates are changed in all directions of the three-dimensional coordinate system. On the other hand, in the third embodiment, the amount of calculation is reduced by changing the coordinates of the vertex only in the horizontal direction.

  Also in the present embodiment, the coordinates before the change of the vertex of the virtual surface TS are arranged along the virtual surface TS as shown by a point P shown in FIG. 24 and 25 are diagrams for explaining a method of changing the coordinates of the vertex of the virtual surface TS in the present embodiment. 24 and 25 show a cross section of the virtual surface TS viewed from the horizontal direction.

  As shown in FIG. 24, it is assumed that a part from the end point Pa to the end point Pb in the virtual surface TS is designated as the change target part based on the enlargement parameter 27c, and r is designated as the enlargement ratio.

  Also in the present embodiment, first, the reference point O (Xo, Yo) is obtained. In this case, only the horizontal coordinate, that is, the X coordinate and the Y coordinate are considered, and the center point of the end point Pa and the end point Pb is derived. A point on the virtual plane TS having the X and Y coordinates of the central point is set as a reference point O. In the present embodiment, the horizontal distance from the reference point O to the end points Pa and Pb is the radius d of the portion to be changed.

  Next, as shown in FIG. 24, the point P included in the change target portion is moved in the horizontal direction to be a point P ′. In the present embodiment, since the point P is moved to the point P ′ only by changing the horizontal coordinate (X coordinate and Y coordinate), the Z coordinate is excluded from the calculation target. The amount of movement from the point P to the point P ′ takes into account the horizontal distance from the point P to the reference point O.

  Assuming that the horizontal distance from the point P to the reference point O is R, the distance R is the horizontal coordinate (x, y) of the point P and the horizontal coordinate (Xo, Yo) of the reference point O. And is obtained by the above-mentioned formula 9.

  The enlargement increase rate W with respect to the point P is obtained by the above-described Expression 5 using the distance R, the radius d of the change target portion, and the enlargement rate r. Therefore, also in the present embodiment, the enlargement increase rate W increases as the distance R decreases. However, when the distance R is 0 (when the point P is the reference point O), the enlargement increase rate W is 0.

  Further, assuming that the horizontal distance from the point P ′ to which the point P is moved to the reference point O is R ′, the distance R ′ can be obtained by the above-described equation (6). Then, the coordinates (x ′, y ′) of the point P ′ to which the point P is moved are expressed by the following equation 11 using the coordinates (Xo, Yo) of the reference point O, the distance R, and the distance R ′. Is required.

このような演算は、変更対象部分に含まれる全ての点Ｐ（基準点Ｏを除く）に関して行われ、図２５に示すように、変更対象部分に含まれる全ての点Ｐ（基準点Ｏを除く）が点Ｐ’に移動される。本実施の形態でも、基準点Ｏまでの距離Ｒが小さい点Ｐほど移動量が大きくなり、基準点Ｏまでの距離Ｒが大きな点Ｐほど移動量が小さくなる。本実施の形態では、この点Ｐ’が変更対象部分に含まれる頂点の変更後の座標を表している。

Such a calculation is performed for all points P (excluding the reference point O) included in the change target part, and as shown in FIG. 25, all the points P (excluding the reference point O) included in the change target part. ) Is moved to point P ′. Also in the present embodiment, the amount of movement increases as the distance P to the reference point O decreases, and the amount of movement decreases as the distance P to the reference point O increases. In the present embodiment, this point P ′ represents the coordinates after the change of the vertex included in the change target portion.

  Thus, by moving the point P to the point P ′, the coordinates of the vertices included in the change target portion are changed. In the present embodiment, the shape of the virtual surface TS is also changed with the change of the coordinates of the vertex of the virtual surface TS. By changing the coordinates of the vertex of the virtual surface TS, the size of the data (subject image) attached to the virtual surface TS is also changed. As a result, also in the present embodiment, a partially enlarged image CPa is generated in which the subject near the center of the target region TR is greatly enlarged and the subject near the periphery of the target region TR is compressed. become.

  In the present embodiment, since the coordinates of the vertex of the virtual surface TS are changed only in the horizontal direction along the ground plane of the vehicle 9, the amount of calculation for generating the partially enlarged image CPa can be greatly reduced.

<4. Fourth Embodiment>
Next, a fourth embodiment will be described. Since the configuration and operation of the image display system 10 of the fourth embodiment are substantially the same as those of the first embodiment, the following description will be focused on differences from the first embodiment. The parameter setting unit 20b according to the first embodiment sets the target region TR in accordance with a user operation. On the other hand, the parameter setting unit 20b of the fourth embodiment has a function of detecting an object existing around the vehicle 9, and sets the target region TR according to the detection result of the object.

  FIG. 26 is a diagram illustrating a configuration of the image display system 10 according to the fourth embodiment. The image display system 10 of the fourth embodiment further includes an object detection device 7 in addition to the configuration of the first embodiment shown in FIG. The object detection device 7 detects an object existing around the vehicle 9. A result signal indicating the result of detecting the object by the object detection device 7 is input to the image generation device 2. The signal reception unit 26 of the image generation device 2 receives this result signal and inputs it to the control unit 20.

  FIG. 27 is a diagram mainly illustrating a configuration of the object detection device 7. As shown in the figure, the object detection device 7 includes a sonar control unit 70 that controls the entire device, a plurality of clearance sonars 72, and a buzzer 71 that emits a warning sound in the passenger compartment.

  The clearance sonar 72 detects an object existing around the vehicle 9 by transmitting an ultrasonic wave and receiving a reflected wave reflected by the object. Moreover, the clearance sonar 72 can detect the distance of the object based on the time from when the ultrasonic wave is transmitted until it returns. The detection result of each clearance sonar 72 is input to the sonar control unit 70. The sonar control unit 70 controls the buzzer 71 and causes the buzzer 71 to output a warning sound corresponding to the distance to the object. Thereby, the user can grasp that an object exists around the vehicle 9.

  FIG. 28 is a diagram illustrating positions where a plurality of clearance sonars 72 are arranged in the vehicle 9. The plurality of clearance sonars 72 are respectively provided at the left and right end portions in front of the vehicle 9 and the left and right end portions in the rear of the vehicle 9.

  Each clearance sonar 72 transmits an ultrasonic wave toward some peripheral areas A11 to A14 around the vehicle 9. Specifically, the clearance sonar 72 provided at the left end portion in front of the vehicle 9 transmits ultrasonic waves toward the peripheral area A11 set on the left side in front of the vehicle 9. Further, the clearance sonar 72 provided at the right end portion in front of the vehicle 9 transmits ultrasonic waves toward the peripheral area A12 set on the right front side of the vehicle 9. In addition, the clearance sonar 72 provided at the left end on the rear side of the vehicle 9 transmits an ultrasonic wave toward the peripheral area A <b> 13 set on the rear left side of the vehicle 9. Further, the clearance sonar 72 provided at the right end portion on the rear side of the vehicle 9 transmits an ultrasonic wave toward the peripheral area A <b> 14 set on the rear right side of the vehicle 9.

  These four peripheral areas A11 to A14 are fixed relatively to the vehicle 9 and set in advance. By such an arrangement of the clearance sonar 72, the object detection device 7 can detect an object existing in any of the four peripheral regions A11 to A14. Based on the position of the clearance sonar 72 that has detected the object, the object detection device 7 can grasp which of the four peripheral regions A11 to A14 is the position of the detected object. Furthermore, the object detection device 7 can grasp the distance of the detected object.

  The result signal indicating the detection result of the object detection device 7 includes the position of the object and the distance of the object. The sonar control unit 70 outputs the result signal to the image generation device 2. Thereby, the parameter setting unit 20b of the image generation device 2 can use the detection result of the object detection device 7.

  When the object detection device 7 detects an object, the parameter setting unit 20b makes the position of the object detected by the object detection device 7 the target region TR so that the user can easily confirm the object in detail. The enlargement parameter 27c is set.

  For example, when the object detection device 7 detects an object existing in the peripheral area A11 on the left front side of the vehicle 9, the parameter setting unit 20b sets the area on the left front side of the vehicle 9 as the target area TR. As a result, a partially enlarged image CPa showing an enlarged subject present in the left front area of the vehicle 9 is generated and displayed on the display device 3. A user who visually recognizes this partially enlarged image CPa can confirm in detail an object existing in the vicinity of the vehicle 9.

<5. Modification>
Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications are possible. Below, such a modification is demonstrated. All the forms including the above-described embodiment and the form described below can be appropriately combined.

  In the above embodiment, the partially enlarged image CPa shows the subject in the vicinity of the inside of the periphery of the target region TR in a compressed manner. On the other hand, the partially enlarged image CPa may be shown by compressing a subject near the outside of the periphery of the target region TR. Even in this case, continuity between the subject included in the partially enlarged image CPa and the subject not included in the partially enlarged image CPa is maintained.

  In the above embodiment, the image generation unit 22 generates the partially enlarged image CPa by changing some coordinates of the virtual surface TS. On the other hand, after enlarging a part of the four photographed images SF, SB, SL, and SR, and pasting the data of these photographed images on the virtual surface TS, a partially enlarged image. May be generated. For example, if the data of four captured images SF, SB, SL, and SR are pasted on the virtual surface TS after enlarging the image of the subject only with respect to the captured image SB of the rear camera 5B, only the subject existing behind the vehicle 9 is displayed. It is possible to generate a partially enlarged image showing the enlarged image.

  In the above embodiment, the enlargement parameter 27c is set in response to the pinch-out operation. However, the enlargement parameter 27c is set using a setting screen that can specify the target area TR and the enlargement ratio r. You may do it.

  In the first and second embodiments, when the coordinates of the vertex of the virtual surface TS are changed, the point P is moved to the point P ′, and the point P ′ is further moved to the point P ″ on the virtual surface TS. On the other hand, the point P may only be moved to the point P′.In this case, the shape of the virtual surface TS also changes as the coordinates of the vertex of the virtual surface TS change. Slightly changed.

  Further, the image display system 10 of the fourth embodiment detects an object existing around the vehicle 9 using the clearance sonar 72. On the other hand, for example, an object existing around the vehicle 9 may be detected by another method such as object detection by a radar device or object recognition based on a captured image.

  In addition, the function described as one block in the above embodiment is not necessarily realized by a single physical element, and may be realized by distributed physical elements. Further, the functions described as a plurality of blocks in the above embodiments may be realized by a single physical element. In addition, even if the device in the vehicle and the device outside the vehicle share processing related to any one function and exchange information by communication between these devices, the one function can be realized as a whole. Good.

  In addition, it has been described that all or part of the functions described as being realized by software by executing the program in the above embodiment may be realized by an electrical hardware circuit or by a hardware circuit. All or part of the functions may be realized by software. Further, the function described as one block in the above-described embodiment may be realized by cooperation of software and hardware.

2 Image generation device 3 Display device 7 Object detection device 10 Image display system 20b Parameter setting unit 22 Image generation unit 31 Touch panel 90 Vehicle image CP Composite image CPa Partially enlarged image

Claims (10)

An image generation device used in a vehicle,
Obtaining means for obtaining a plurality of photographed images obtained by a plurality of cameras for photographing the periphery of the vehicle;
Generating means for generating a composite image showing the periphery of the vehicle viewed from a virtual viewpoint using the plurality of captured images;
With
The generating means enlarges a subject included in a part of a target area of the whole area while showing an entire area around the vehicle, and shows the subject not included in the target area without expanding. An image generating apparatus that generates a partially enlarged image that is a composite image. The image generation apparatus according to claim 1,
The generating means includes
Affixing the data of the plurality of captured images to a virtual surface corresponding to the periphery of the vehicle, and generating the composite image using the data affixed to the virtual surface;
An image generating apparatus, wherein the partial enlarged image is generated by changing coordinates of a portion corresponding to the target area on the virtual plane. The image generation apparatus according to claim 2,
The image generation apparatus, wherein the virtual surface is a curved surface set in a virtual three-dimensional space. The image generation apparatus according to claim 3.
The image generation apparatus according to claim 1, wherein the generation unit changes the coordinates only in a direction along a ground contact surface of the vehicle. In the image generation device according to claim 3 or 4,
The image generation apparatus characterized in that the generation means can set the virtual viewpoint at an arbitrary position in the three-dimensional space. In the image generation device according to any one of claims 1 to 5,
The partially enlarged image is an image generating apparatus characterized by compressing and showing a subject near the periphery of the target area. The image generation device according to any one of claims 1 to 6,
Setting means for setting the target area according to a user operation;
An image generation apparatus further comprising: The image generation device according to any one of claims 1 to 6,
Setting means for setting the target area according to a detection result of a detection means for detecting a position of an object existing in the vicinity of the vehicle;
An image generation apparatus further comprising: An image display system used in a vehicle,
An image generation apparatus according to any one of claims 1 to 8,
A display device for displaying the partially enlarged image generated by the image generation device;
An image display system comprising: An image generation method used in a vehicle,
(A) acquiring a plurality of captured images obtained by a plurality of cameras that capture the periphery of the vehicle;
(B) using the plurality of captured images, generating a composite image showing the periphery of the vehicle viewed from a virtual viewpoint;
With
The step (b) shows an entire area around the vehicle while enlarging a subject included in a part of the target area, and showing an object not included in the target area without expanding. An image generation method characterized by generating a partially enlarged image that is one of the composite images.

Images