JP2014069629A - Image processor, and image processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technology capable of easily grasping at which position, from which direction and within which range an image is seen even in a case of continuously displaying images around a vehicle.SOLUTION: An image processor includes: image acquisition means for acquiring a photographed images of a camera mounted on an own vehicle; generation means for generating a peripheral image showing a periphery of the own vehicle seen from a virtual viewpoint, and an auxiliary image showing an area of the peripheral image on the basis of the photographed images; and output means for outputting the peripheral image and the auxiliary image to a display device and displaying the peripheral image and the auxiliary image on the display device.

Description

  The present invention relates to a technique for processing an image of a camera mounted on a vehicle.

  2. Description of the Related Art Conventionally, an image display system that acquires an image showing the periphery of a vehicle such as an automobile and displays the image on a display device provided in the vehicle is known. By using such an image display system, a user (typically a driver) can check the situation around the vehicle.

  For example, if an image display system that displays images showing respective front, rear, left, and right regions of the vehicle is used, the user can grasp the situation around the vehicle by displaying each image continuously. As a technique for continuously displaying images around a vehicle, for example, Patent Document 1 is available.

JP 2011-66763 A

  However, when images around the vehicle are continuously displayed, the visual direction moves, so it may be difficult to grasp which position is viewed from which direction and in which range. Further, although it is desired to set the viewpoint according to the user's preference, a method that can be easily set has not been realized.

  The present invention has been made in view of the above problems, and even when continuously displaying images around a vehicle, it is easy to grasp which position is viewed from which direction and in which range. It is an object of the present invention to provide a technique that can be easily set according to user preferences.

  In order to solve the above-mentioned problem, the invention of claim 1 is an image processing device for processing an image, based on image acquisition means for acquiring a captured image of a camera mounted on the own vehicle, and the captured image, Generation means for generating a peripheral image showing the periphery of the host vehicle viewed from a virtual viewpoint and an auxiliary image indicating a region of the peripheral image, and the peripheral image and the auxiliary image are output to a display device and displayed on the display device Output means.

  The invention according to claim 2 is the image processing apparatus according to claim 1, wherein the output means outputs the peripheral image to a first display device and causes the first display device to display the auxiliary image. The image is output to the second display device and displayed on the second display device.

  According to a third aspect of the present invention, in the image processing apparatus according to the second aspect, the second display device has a touch panel, and the viewpoint position and the line-of-sight direction in the virtual viewpoint are set based on an operation of the touch panel. Setting means is further provided.

  According to a fourth aspect of the present invention, in the image processing apparatus according to the third aspect, the setting unit sets the line-of-sight direction after setting the viewpoint position.

  According to a fifth aspect of the present invention, in the image processing apparatus according to the third or fourth aspect, when the operation of the touch panel is an operation of moving without releasing a finger from the touch panel, the setting unit includes: The movement of the viewpoint position and the line-of-sight direction is also set.

  Further, in the image processing apparatus according to claim 5, the setting unit synchronizes the timing of movement of the viewpoint position and the timing of movement of the line-of-sight direction to synchronize the viewpoint position. And the line-of-sight direction.

  According to a seventh aspect of the present invention, in the image processing apparatus according to any one of the third to sixth aspects, the setting means sets a viewpoint position and a line-of-sight direction in a three-dimensional space.

  According to an eighth aspect of the present invention, in the image processing apparatus according to the seventh aspect, the setting means sets the vertical viewpoint position and line-of-sight direction after setting the viewpoint position and line-of-sight direction on a horizontal plane.

  According to a ninth aspect of the present invention, in the image processing device according to any one of the fifth to eighth aspects, the setting means also sets the viewpoint position and the movement speed in the line-of-sight direction.

  According to a tenth aspect of the present invention, in the image processing apparatus according to the ninth aspect, the setting means sets the viewpoint position and the movement speed in the line-of-sight direction according to the movement speed of the operation position on the touch panel.

  The invention according to claim 11 is the image processing apparatus according to claim 9 or 10, wherein the setting means divides the periphery of the host vehicle into a plurality of areas, and movements in a viewpoint position and a line-of-sight direction for each area. Set the speed.

  The invention according to claim 12 is the image processing device according to any one of claims 3 to 11, further comprising storage means for storing setting information set by the setting means, wherein the storage means Setting information is stored in association with buttons displayed on the first display device or the second display device.

  A thirteenth aspect of the present invention is an image processing system including the image processing apparatus according to any one of the first to twelfth aspects and a display device that displays an image.

  According to the first to thirteenth aspects of the present invention, since the auxiliary image indicating the area of the peripheral image is generated and displayed, it is easy to grasp from which virtual viewpoint the peripheral image is the peripheral area of the vehicle. be able to.

  In particular, according to the invention of claim 2, since the auxiliary image and the peripheral image are displayed on the second display device that is different from the first display device, the peripheral image can be displayed only by looking at the second display device. The area can be grasped.

  In particular, according to the third and fourth aspects of the present invention, by operating the touch panel, the viewpoint position and the line-of-sight direction in the virtual viewpoint can be set, so that it is possible to display a peripheral image according to the user's preference. Become.

  According to the invention of claim 5 in particular, when the operation is performed without moving the finger from the touch panel, the movement of the viewpoint position and the line-of-sight direction is also set. It is possible to display the surrounding image while dynamically changing it, making it easier to grasp the surroundings of the vehicle.

  In particular, according to the invention of claim 6, the viewpoint position and the line-of-sight direction set by different operations can be synchronized. In particular, according to the seventh and eighth aspects of the invention, the viewpoint position and the line-of-sight direction can be set three-dimensionally.

  In particular, according to the ninth to eleventh aspects of the present invention, the display speed of the peripheral image can be easily set to be a speed according to the user's desire.

  In particular, according to the invention of claim 12, since the setting information is associated with the button displayed on the first or second display device, the stored setting information is read only by operating the button. Is possible.

FIG. 1 is a diagram showing an overview of an image display system. FIG. 2 is a diagram illustrating directions in which four cameras capture images. FIG. 3 is a diagram illustrating a method in which the image composition unit generates a composite image. FIG. 4 is a diagram illustrating an example of a peripheral image and an auxiliary image. FIG. 5 is a diagram illustrating an example of a peripheral image and an auxiliary image. FIG. 6 is a diagram showing an example of continuously displayed peripheral images. FIG. 7 is a diagram showing an example of continuously displayed peripheral images. FIG. 8 is a diagram showing an example of continuously displayed peripheral images. FIG. 9 is a diagram illustrating an example of an auxiliary image. FIG. 10 is a diagram illustrating an example in which an auxiliary image and an operation image are displayed. FIG. 11 is a diagram illustrating an example of a peripheral image and an auxiliary image. FIG. 12 is a diagram illustrating an example of a peripheral image and an auxiliary image. FIG. 13 is a diagram for explaining the three-dimensional coordinates. FIG. 14 shows a viewpoint data setting screen. FIG. 15 is a flowchart showing the flow of processing of the image display system. FIG. 16 is a diagram illustrating an example of a viewpoint position setting screen. FIG. 17 is a diagram for explaining the association of coordinate values. FIG. 18 is a diagram illustrating an example of a screen for setting the line-of-sight direction. FIG. 19 is a diagram for explaining the association of coordinate values. FIG. 20 is a diagram for explaining the association of coordinate values. FIG. 21 is a diagram illustrating an example of a moving speed setting screen. FIG. 22 is a diagram illustrating an example of a display range setting screen. FIG. 23 is a flowchart showing the flow of viewpoint data setting processing. FIG. 24 is a flowchart showing the flow of viewpoint data setting processing. FIG. 25 is a diagram illustrating an example of a peripheral image and an auxiliary image.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  <1. First Embodiment>

    <1-1. Configuration>

  FIG. 1 is a diagram illustrating a configuration of an image display system 10 according to the first embodiment. This image display system 10 is used in a vehicle (in this embodiment, an automobile), and has a function of generating an image showing an area around the vehicle and displaying it in the vehicle interior. By using this image display system 10, a user (typically a driver) of the image display system 10 can grasp the state around the vehicle in almost real time. Hereinafter, a vehicle on which the image display system 10 is mounted is referred to as “own vehicle”.

  As shown in FIG. 1, the image display system 10 mainly includes a plurality of cameras 5, an image generation device 2, a main display 3, a sub display 6, and operation buttons 4. Each of the plurality of cameras 5 captures a captured image by capturing the periphery of the host vehicle, and inputs the acquired captured image to the image generation device 2. The image generation device 2 generates a peripheral image to be displayed on the main display 3 using a captured image indicating the periphery of the host vehicle, and generates an auxiliary image to be displayed on the sub display 6. The main display 3 displays the peripheral image generated by the image generation device 2, and the sub display 6 displays the auxiliary image generated by the image generation device 2. The operation button 4 receives a user operation.

  Each of the plurality of cameras 5 includes a lens and an image sensor, and electronically obtains a photographed image showing the periphery of the host vehicle. The multiple cameras 5 include a front camera 5F, a rear camera 5B, a left side camera 5L, and a right side camera 5R. These four cameras 5F, 5B, 5L, 5R are arranged at different positions in the host vehicle 9, and photograph different directions around the host vehicle.

  FIG. 2 is a diagram illustrating directions in which the four cameras 5F, 5B, 5L, and 5R are respectively photographed. The front camera 5F is provided at the front end of the host vehicle 9, and the optical axis 5Fa is directed in the straight traveling direction of the host vehicle 9. The rear camera 5 </ b> B is provided at the rear end of the host vehicle 9, and the optical axis 5 </ b> Ba is directed in the direction opposite to the straight traveling direction of the host vehicle 9. The left side camera 5L is provided on the left side mirror 93L on the left side, and its optical axis 5La is directed to the left side of the host vehicle 9 (a direction orthogonal to the straight traveling direction). Further, the right side camera 5R is provided on the right side mirror 93R on the right side, and its optical axis 5Ra is directed to the right side of the host vehicle 9 (a direction orthogonal to the straight traveling direction).

  A wide-angle lens such as a fisheye lens is adopted as the lens of these cameras 5F, 5B, 5L, 5R, and each camera 5F, 5B, 5L, 5R has an angle of view θ of 180 degrees or more. For this reason, it is possible to photograph the entire periphery of the host vehicle 9 by using the four cameras 5F, 5B, 5L, and 5R.

  Returning to FIG. 1, the main display 3 and the sub display 6 include a thin display panel such as a liquid crystal display, for example, and displays various types of information and images. The main display 3 and the sub display 6 are arranged on an instrument panel or the like of the host vehicle 9 so that the user can visually recognize the screen. The main display 3 may be disposed in the same housing as the image generation device 2 and integrated with the image generation device 2, or may be a separate device from the image generation device 2. The sub display is located at a position different from the main display and is more easily visible than the main display even when the user is driving the host vehicle. For example, the sub-display 6 is provided in or on the meter panel in front of the driver's seat where the user (driver) is seated. The main display 3 and the sub display 6 are provided with touch panels 31 and 61 overlaid on the display panel, and can accept user operations.

  The operation button 4 is an operation member that receives a user operation. The operation button 4 is provided, for example, on the steering wheel of the host vehicle 9 and mainly receives an operation from the driver. The user can perform various operations on the image display system 10 via the operation buttons 4 and the touch panels 31 and 61 of the main display 3 and the sub display 6. When a user operation is performed on either the operation button 4 or the touch panel 31 or 61, an operation signal indicating the content of the operation is input to the image generation device 2.

  The image generation device 2 is an electronic device that can perform various types of image processing, and includes an image acquisition unit 21, an image synthesis unit 22, an image adjustment unit 23, and an image output unit 24.

  The image acquisition unit 21 acquires captured images respectively obtained by the four cameras 5F, 5B, 5L, and 5R. The image acquisition unit 21 has an image processing function such as a function of converting an analog captured image into a digital captured image. The image acquisition unit 21 performs predetermined image processing on the acquired captured image, and inputs the processed captured image to the image composition unit 22 and the image adjustment unit 23.

  The image composition unit 22 is a hardware circuit that performs image processing for generating a composite image. The image composition unit 22 composes a plurality of captured images acquired by the plurality of cameras 5 and generates a composite image showing the periphery of the host vehicle 9 viewed from the virtual viewpoint. The image composition unit 22 projects data of a plurality of photographed images on a virtual projection plane corresponding to the periphery of the host vehicle 9, and generates a composite image using the data on the projection surface. Details of the method for generating the composite image will be described later.

  The image adjusting unit 23 generates a peripheral image to be displayed on the main display 3 and an auxiliary image to be displayed on the sub display 6. The image adjustment unit 23 generates a peripheral image to be displayed on the main display using the composite image generated by the image composition unit 22. Further, the image adjustment unit 23 creates an auxiliary image corresponding to the peripheral image displayed on the main display 3 as an image to be displayed on the sub display 6.

  The image output unit 24 outputs the peripheral image generated by the image adjustment unit 23 to the main display 3 and causes the main display 3 to display the peripheral image. Further, the image output unit 24 outputs the auxiliary image generated by the image adjustment unit 23 to the sub display 6 and causes the sub display 6 to display the auxiliary image. As a result, a peripheral image indicating the periphery of the host vehicle 9 as viewed from the virtual viewpoint is displayed on the main display 3, and an auxiliary image indicating a region of the peripheral image displayed on the main display 3 is displayed on the sub display 6.

  The image generation device 2 further includes a control unit 20, an operation reception unit 25, a signal reception unit 26, and a storage unit 27. The control unit 20 is a microcomputer including a CPU, a RAM, a ROM, and the like, for example, and comprehensively controls the entire image generation apparatus 2.

  The operation receiving unit 25 receives operation signals output from the operation buttons 4 and the touch panels 31 and 61 when the user performs an operation. Thereby, the operation reception part 25 receives a user's operation. The operation reception unit 25 inputs the received operation signal to the control unit 20.

  The signal receiving unit 26 receives a signal from another device provided in the host vehicle 9 separately from the image generating device 2 and inputs the signal to the control unit 20. For example, the signal receiving unit 26 receives a signal indicating a shift position, which is the position of the shift lever of the transmission of the host vehicle 9, from the shift sensor 91. Based on this signal, the control unit 20 can determine whether the traveling direction of the host vehicle 9 is forward or backward.

  The storage unit 27 is, for example, a nonvolatile memory such as a flash memory, and stores various types of information. The storage unit 27 stores a program 27a as firmware, various data used by the image composition unit 22 to generate a composite image, and various data used by the image adjustment unit 23 to generate a peripheral image and an auxiliary image. Remember. For example, viewpoint data 27b including a virtual viewpoint is included as data used for generating such peripheral images and auxiliary images.

  Various functions of the control unit 20 are realized by the CPU performing arithmetic processing according to the program 27 a stored in the storage unit 27. The image control unit 20a, the viewpoint data setting unit 20b, and the registration unit 20c shown in the drawing are a part of functional units realized by the CPU performing arithmetic processing according to the program 27a.

  The image control unit 20a controls an image composition unit 22 that generates a composite image and an image adjustment unit 23 that generates a peripheral image and an auxiliary image. The image control unit 20a controls the image composition unit 22 and the image adjustment unit 23 to generate a composite image, a peripheral image, and an auxiliary image according to the traveling state of the host vehicle 9 and the user's operation. The viewpoint data setting unit 20b sets viewpoint data used when generating a composite image, a peripheral image, and an auxiliary image. The viewpoint data setting unit 20b sets a virtual viewpoint when the user operates the operation buttons 4 and the touch panels 31 and 61 to set favorite viewpoint data. The registration unit 20 c registers the set viewpoint data in association with the preset button, and stores it in the storage unit 27.

    <1-2. Generation of composite image>

  Next, a method will be described in which the image composition unit 22 generates a composite image that shows the surroundings of the host vehicle 9 viewed from a virtual viewpoint. FIG. 3 is a diagram illustrating a method in which the image composition unit 22 generates a composite image.

  When shooting is performed with each of the front camera 5F, the rear camera 5B, the left side camera 5L, and the right side camera 5R, four captured images SF that respectively indicate the front, rear, left side, and right side of the host vehicle 9 are displayed. SB, SL, and SR are acquired. These four captured images SF, SB, SL, and SR include data around the entire vehicle 9.

  First, the image composition unit 22 projects data (values of each pixel) included in these four captured images SF, SB, SL, and SR onto a projection plane TS in a virtual three-dimensional space. The projection surface TS is a virtual three-dimensional surface corresponding to the area around the host vehicle 9. The central area of the projection plane TS is defined as a vehicle area R0 that is the position of the host vehicle 9.

  On the projection surface TS, the captured image data is not projected on the vehicle area (position of the host vehicle 9) R0, and the captured image data is projected on the area outside the vehicle area R0. Hereinafter, a region on the projection surface TS onto which the captured image data is projected (a region outside the vehicle region R0) is referred to as a “projection target region”.

  The vehicle region R0 is virtually provided with a vehicle image PG that is a polygon model indicating the three-dimensional shape of the host vehicle 9. The configured vehicle image PG is arranged at a substantially hemispherical center portion defined as the position of the host vehicle 9 in the three-dimensional space where the projection plane TS is set.

  Each position in the projection target area of the projection surface TS is associated with one of the four captured images SF, SB, SL, and SR by correspondence information such as table data. The image composition unit 22 projects the data of the four captured images SF, SB, SL, and SR on the corresponding portions of the projection target area.

  The image composition unit 22 projects the data of the captured image SF of the front camera 5F onto a portion PF corresponding to the front of the host vehicle 9 in the projection target area. In addition, the image composition unit 22 projects the data of the captured image SB of the rear camera 5B onto a portion PB corresponding to the rear of the host vehicle 9 in the projection target area. Furthermore, the image composition unit 22 projects the data of the captured image SL of the left side camera 5L on the portion PL corresponding to the left side of the host vehicle 9 in the projection target area, and corresponds to the right side of the host vehicle 9 in the projection target area. Data of the captured image SR of the right side camera 5R is projected onto the part PR to be performed.

  When the captured image data is projected onto each portion of the projection target area of the projection surface TS in this way, the image composition unit 22 virtually constructs a polygon model indicating the three-dimensional shape of the host vehicle 9. The model of the host vehicle 9 is arranged in the vehicle region R0 that is the position of the host vehicle 9 in the three-dimensional space where the projection plane TS is set.

  Next, the image composition unit 22 sets a virtual viewpoint VP for the three-dimensional space under the control of the image control unit 20a. The image composition unit 22 can set the virtual viewpoint VP at an arbitrary viewpoint position in the three-dimensional space in an arbitrary line-of-sight direction. Then, the image composition unit 22 cuts out, as an image, data projected on an area included in a predetermined viewing angle from the set virtual viewpoint VP in the projection surface TS. Further, the image composition unit 22 performs rendering on the model of the host vehicle 9 according to the set virtual viewpoint VP, and superimposes the two-dimensional vehicle image 90 as a result on the cut-out image. Thereby, the image composition unit 22 generates the composite image CP showing the host vehicle 9 and the periphery of the host vehicle 9 viewed from the virtual viewpoint VP.

  For example, as shown in FIG. 3, when a virtual viewpoint VPa is set with the viewpoint position directly above the host vehicle 9 and the line-of-sight direction immediately below, a composite image (overview of the host vehicle 9 and the surrounding area of the host vehicle 9 ( Overhead image) CPa is generated. When the virtual viewpoint VPb is set with the viewpoint position inside the host vehicle 9 and the line-of-sight direction set to the left front, a composite image CPb that displays the left front periphery of the host vehicle 9 is generated. Similarly, when the virtual viewpoint VPc with the viewpoint position on the left side of the host vehicle 9 and the line-of-sight direction on the left is set, a composite image CPc that displays the periphery on the left side of the host vehicle 9 is generated.

  In the present embodiment, a user sets a virtual viewpoint in advance, and viewpoint data 27b including the set virtual viewpoint is registered in the storage unit 27 as preset data. When the user selects any viewpoint data 27b from the registered preset data, the image composition unit 22 displays the host vehicle 9 and the surroundings of the host vehicle 9 based on the selected viewpoint data 27b. A composite image to be generated is generated.

  For example, when the user selects the viewpoint data 27b including the virtual viewpoint in which the vicinity of the viewpoint of the driver inside the host vehicle 9 is the viewpoint position and the left front of the host vehicle 9 is the line-of-sight direction, the virtual viewpoint VPb is set. The image composition unit 22 creates a composite image CPb in which the left front is displayed from the driver viewpoint by cutting out the data projected on the corresponding area as an image.

  Similarly, when the user selects the viewpoint data 27b including the virtual viewpoint in which the left side surface of the host vehicle 9 is the viewpoint position and the left direction is the line-of-sight direction, the virtual viewpoint VPc is set, and the image composition unit 22 By cutting out the data projected on the corresponding area as an image, a composite image CPc displaying the left direction from the left side surface of the vehicle is created. When the viewpoint data 27b includes a plurality of viewpoint positions and line-of-sight directions, the image composition unit 22 creates a plurality of composite images corresponding to them. Details of the viewpoint data setting process by the user will be described later.

    <1-3. Peripheral image and auxiliary image>

  Next, a process in which the image adjustment unit 23 generates a peripheral image and an auxiliary image, and a process in which the image output unit 24 displays the peripheral image and the auxiliary image on the main display 3 and the sub display 6 will be described. FIG. 4 is a diagram illustrating an example of a peripheral image and an auxiliary image generated by the image adjustment unit 23. FIG. 4A shows an example of a peripheral image displayed on the main display 3, and FIG. 4B shows an example of an auxiliary image displayed on the sub display 6.

  The peripheral image shown in FIG. 4A is a peripheral image when the user selects viewpoint data 27b including a virtual viewpoint in which the vicinity of the viewpoint of the driver of the host vehicle 9 is the viewpoint position and the left front of the host vehicle is the line-of-sight direction. It is. The peripheral image is an image in which a transmission image inside the vehicle is superimposed and displayed on the composite image created by the image composition unit 22. This is because when the inside of the vehicle is set as the viewpoint position, the image is close to an actual scene seen in the line-of-sight direction, and the positional relationship between the surrounding object and the host vehicle 9 is easily understood. However, the image inside the vehicle is made a transparent image so that objects around the host vehicle 9 are not hidden by the image inside the vehicle. The transmission image inside the vehicle can be created using an image created in advance by CG. In FIG. 4A, the image adjustment unit 23 combines the transmission image using CG corresponding to the line-of-sight direction. A peripheral image is generated by superimposing and displaying the image.

  The auxiliary image shown in FIG. 4B is an image showing the area of the peripheral image displayed on the main display 3. That is, the auxiliary image and the peripheral image are associated images. In the auxiliary image shown in FIG. 4B, the area of the peripheral image displayed on the main display 3 is highlighted with respect to the image of the image obtained by looking down the host vehicle 9 from above. In this way, on the sub-display 6, a scope image indicating in which direction the peripheral image displayed on the main display 3 hits as viewed from the host vehicle 9 is displayed as an auxiliary image.

  When the image adjustment unit 23 creates a peripheral image to be displayed on the main display 3, the image adjustment unit 23 creates an auxiliary image based on the viewpoint data. Specifically, the viewpoint data includes data relating to the virtual viewpoint (that is, the viewpoint position and the line-of-sight direction), and the image adjustment unit 23 assists by highlighting a region corresponding to the line-of-sight direction. Create an image.

  In the case of FIG. 4, a peripheral image is displayed on the main display 3 with the vicinity of the viewpoint of the driver of the host vehicle 9 as the viewpoint position and the front left direction as the line-of-sight direction. An auxiliary image indicating that this is the case (that is, the area of the peripheral image is the front left of the vehicle) is displayed. In this way, the user can easily grasp which direction the image displayed on the main display 3 is viewed from only by looking at the sub-display 6.

  In FIG. 4B, a fan-shaped scope image indicating a peripheral image region is displayed, but the auxiliary image is not limited to this. For example, an arrow extending from the viewpoint position in the line-of-sight direction (the central axis direction of the peripheral image) can be displayed. It is also possible to display two points: a viewpoint position and a position corresponding to the center of the area of the peripheral image. Further, when the viewpoint position is fixed, only the point at the position corresponding to the center of the area of the peripheral image may be displayed.

  In the present embodiment, it is possible not only to display a peripheral image by setting the virtual viewpoint to one point, but also to display a peripheral image by setting a plurality of virtual viewpoints. For example, when the viewpoint near the driver's viewpoint is set as the virtual viewpoint and the four directions of the front, rear, right side, and left side of the host vehicle 9 are set as the line-of-sight directions, the peripheral images viewed from each virtual viewpoint Is created.

  FIG. 5 is a diagram illustrating an example of a peripheral image and an auxiliary image in this case. FIG. 5A is a peripheral image when the front direction of the host vehicle 9 is set as the line-of-sight direction from the viewpoint position, and FIG. 5B is a peripheral image when the left side of the host vehicle 9 is set as the line-of-sight direction. FIG. 5C is a peripheral image when the rear direction of the host vehicle 9 is set as the line-of-sight direction, and FIG. 5D is a peripheral image when the right side of the host vehicle 9 is set as the line-of-sight direction. Each of these peripheral images is created by the image adjustment unit 23 using the composite image and the transmission image by the method described above. FIGS. 5E to 5H are auxiliary images corresponding to FIGS. 5A to 5D, respectively. Also in each of these auxiliary images, the image adjusting unit 23 creates them by the method described above.

  When these peripheral images are displayed on the main display 3, the peripheral images can be switched in order and displayed. For example, the front peripheral image, the left peripheral image, the rear peripheral image, and the right peripheral image are switched and displayed in this order. At this time, the auxiliary display corresponding to each peripheral image is sequentially switched and displayed on the sub display 6 at the same timing. That is, the image output unit 24 displays the peripheral image and the auxiliary image on the main display 3 and the sub display 6 at the same timing. Thus, the peripheral image displayed on the main display 3 and the auxiliary image displayed on the sub display 6 can be displayed in association with each other, and the image displaying the periphery of the host vehicle 9 can be viewed from which direction. It is possible to easily grasp whether the image is an image. Note that the display order is arbitrary and can be set as appropriate.

  In addition, if the number of virtual viewpoints is set to be larger and peripheral images corresponding to each virtual viewpoint are displayed in order, peripheral images slightly different in the line-of-sight direction can be continuously displayed, and a moving image-like display can be obtained. It becomes possible. With such a display, the user can continuously check the entire area around the host vehicle, instead of checking a plurality of locations around the host vehicle in pieces. Corresponding to this, if the auxiliary image is also continuously displayed, it is possible to continuously grasp the area of the peripheral image.

  6 to 8 are diagrams illustrating an example in which peripheral images are continuously displayed on the main display 3. 6 to 8B, the vicinity of the driver's viewpoint is set as the viewpoint position. 6A to 6C are peripheral images in which the front direction, the left front direction, and the left side of the host vehicle 9 are line-of-sight directions. FIGS. 7A to 7C are peripheral images in which the left rear, the rear, and the right rear of the host vehicle 9 are line-of-sight directions. FIGS. 8A and 8B are peripheral images in which the right side and the right front of the host vehicle 9 are line-of-sight directions, respectively, and FIG. 8C is a view position directly above the host vehicle and the line-of-sight direction directly below the host vehicle. Is a peripheral image. 6 to 8 are given as representative examples of peripheral images, but a plurality of peripheral images (not shown) are generated between these peripheral images.

  These peripheral images and the peripheral images generated between the peripheral images are shown in FIGS. 6A, 6B, 7C, 7A, 7B, 7C, and 8A. When the images are displayed in the order of b) and (c), the line-of-sight direction can be rotated from the front of the host vehicle 9 to the left, and finally the image of the host vehicle 9 viewed from above can be continuously displayed. For this reason, the user can check the situation around the host vehicle 9 as a moving image. In the present embodiment, since the peripheral image viewed from the top of the host vehicle 9 is displayed lastly, an image of the surroundings viewed from the driver's viewpoint, and an image of the entire region surrounding the host vehicle 9 are displayed. Both of these can be confirmed, and it becomes easier to grasp the situation around the vehicle.

  FIG. 9 is a diagram illustrating an example of an auxiliary image displayed on the sub-display 6 when the peripheral images of FIGS. 6 to 8 are displayed on the main display 3. The auxiliary images shown in FIGS. 9A to 9I correspond to the peripheral images shown in FIGS. 6A to 8C. That is, when the peripheral images shown in FIGS. 6A to 8C are displayed on the main display 3, the auxiliary images shown in FIGS. 9A to 9I are displayed on the sub display 6 with the corresponding peripheral images. It is displayed continuously at the same timing. Therefore, even in this case, the area of the peripheral image displayed on the main display 3 can be easily grasped.

  It is also possible for the user to freely change the display of the peripheral image by operating the operation button 4 or the touch panel 61. In this case, the sub display 6 displays an image as shown in FIG. In the image shown in FIG. 10, an operation image is displayed in addition to the auxiliary image. The operation image is an image on which a button for stopping and reproducing the continuous display of peripheral images, a button for rotating in the line of sight direction, and the like are displayed.

  When the user operates the touch panel 61 corresponding to the operation image displayed on the sub-display 6, the display of the peripheral image can be changed. For example, when the stop button is operated while the peripheral image is being continuously displayed, the continuous display of the peripheral image is stopped in the state of the displayed image. Then, when the “viewpoint right rotation” button is operated, a peripheral image in which the line-of-sight direction is moved to the right from the line-of-sight direction of the currently displayed peripheral image is displayed. Further, when the “viewpoint left rotation” button is operated in a state where the display of the peripheral image is stopped, the peripheral image in which the line-of-sight direction is moved to the left from the line-of-sight direction of the currently displayed peripheral image is displayed.

  For example, when the “viewpoint right rotation” button is operated when the peripheral image and the auxiliary image shown in FIG. 11A are displayed, the peripheral image and the auxiliary image whose gaze direction is moved to the right are continuously displayed. Is displayed. Thereby, for example, the image shown in FIG. 11B is displayed. Similarly, when the “viewpoint left rotation” button is operated while the peripheral image and the auxiliary image shown in FIG. 11A are displayed, each peripheral image and each auxiliary image whose gaze direction is moved to the left are displayed. Images are displayed continuously. Thereby, for example, the image shown in FIG. 11C is displayed.

  It should be noted that the moving speed of the peripheral image can be appropriately changed according to the operation method of the touch panel 61 (the length of time for operation, the magnitude of force, etc.). Further, when the playback button is operated, continuous display with a preset moving direction and moving speed starts from the state of the peripheral image displayed at the time of the operation. For example, when the operation button 4 is provided on the steering wheel and the operation button 4 is associated with the operation image, the same operation can be performed by operating the operation button 4 of the steering wheel. You may be able to do that.

  Further, when the viewpoint position of the virtual viewpoint is outside the vehicle, the transmitted image is not displayed in the peripheral image, and only the composite image is displayed. Examples of peripheral images in this case are shown in FIGS. FIG. 12A is a peripheral image viewed from a virtual viewpoint with the left side outside the host vehicle 9 as the viewpoint position and the left side of the host vehicle 9 as the viewing direction. FIG. 12B is a peripheral image viewed from a virtual viewpoint in which the rear of the host vehicle 9 is the viewpoint position and the rear of the host vehicle 9 is the viewing direction.

  As an auxiliary image when the viewpoint position is outside the host vehicle 9, a scope image having the viewpoint position outside the host vehicle 9 can be used in addition to the auxiliary image described above. For example, the auxiliary images shown in FIGS. FIG. 12C is an auxiliary image corresponding to FIG. 12A, and FIG. 12C is an auxiliary image corresponding to FIG. In both cases, since the viewpoint position is outside the own vehicle 9, a scope image with the viewpoint position outside the vehicle is used.

    <1-4. Setting viewpoint data>

  Next, processing for setting viewpoint data by the user will be described. The viewpoint data can be arbitrarily set and registered as setting information by the user in addition to the case where general data is registered as initial settings. The viewpoint data includes data relating to the virtual viewpoint, the moving direction, the moving speed, and the display range. The virtual viewpoint is data regarding the viewpoint position and the line-of-sight direction. The moving direction is data relating to the direction in which the virtual viewpoint is moved. The moving speed is data relating to the speed when moving the virtual viewpoint. The display range is data regarding the viewing angle to be displayed.

  In this embodiment, when the viewpoint position and the line-of-sight direction are set, the virtual viewpoint is moved by making the line-of-sight direction go around the vehicle and the surrounding image of the vehicle is continuously displayed. For this reason, only one virtual viewpoint is set, but the moving direction and moving speed are also set.

  Since the virtual viewpoint is arranged in a three-dimensional space, it is represented using three-dimensional coordinates with an arbitrary point as the origin. For example, as shown in FIG. 13, XYZ Cartesian coordinates are used in which the center of the host vehicle is the origin, the left-right direction of the vehicle on the horizontal plane is the X axis, the front-rear direction is the Y axis, and the vertical direction is the Z axis. Can do.

  Here, it demonstrates using FIG. 14A is a viewpoint data selection screen, and FIGS. 14B and 14C are viewpoint data setting screens. Setting of the viewpoint data is performed by the user operating the setting screen. The setting screen is displayed by selecting a setting button displayed on the viewpoint data selection screen or the like.

  When the user selects the setting button on the selection screen shown in FIG. 14A, the screen transitions to the setting screen shown in FIG. In the selection screen shown in FIG. 14B, the viewpoint position and the line-of-sight direction are set. Since the viewpoint position and the line-of-sight direction are represented by coordinate values (X, Y, Z) in terms of data, it is possible to set a new viewpoint position and line-of-sight direction by changing the coordinate value for each coordinate axis. Become. Note that the first coordinate value on the setting screen may be the origin (0, 0, 0) or the previously selected coordinate value, and can be set as appropriate. In addition, the point set on the line-of-sight direction setting screen is an arbitrary point on a straight line extending from the viewpoint position toward the line-of-sight direction (hereinafter referred to as “point on line of sight”). Therefore, the line-of-sight direction is set by setting the coordinate value of the point on the line of sight.

  In the setting screen shown in FIG. 14B, when setting the viewpoint position, each coordinate value can be changed by pressing the right button or the left button displayed corresponding to each coordinate axis of XYZ. . For example, when the right button on the X axis is pressed, the X coordinate moves to the + side, and when the left button is pressed, the X coordinate moves to the-side. The same applies to the Y axis and the Z axis. The same applies when the line-of-sight direction (point on the line of sight) is set.

  A numerical value corresponding to the coordinate value is displayed beside each coordinate setting button of XYZ. When setting the viewpoint position and the line-of-sight direction, the numerical value corresponding to the coordinate value changes in accordance with the operation of the right button or the left button of XYZ. For example, a numerical value increases when the right button is pressed, and a numerical value decreases when the left button is pressed. Thereby, since the user can recognize the coordinate value as a numerical value, the setting becomes easy. This numerical value may be any numerical value as long as it corresponds to the coordinate value. Moreover, it is good also as a display which displays the point of (X, Y, Z) on the pseudo | simulation image of a vehicle, and moves a point according to operation of a button. In this case, a place such as a viewpoint position with respect to the vehicle can be recognized, and setting becomes easy.

  Note that when the viewpoint position and the line-of-sight direction are determined, the movement path when the line-of-sight direction goes around the vehicle is determined. Since the line-of-sight direction is a direction from the viewpoint position toward a point on the line of sight, when the point on the line of sight moves, the line-of-sight direction moves corresponding to the movement. In the present embodiment, the point on the line of sight moves on the XY plane on which it is present in a circular shape centered on the point where the XY point at the viewpoint position is projected onto the XY plane. In other words, the direction of the line of sight is the direction of the line of sight toward the circle with the viewpoint position as the base point, and the line of sight moves in a circle.

  Specifically, as shown in FIG. 14D, when the coordinate values of the viewpoint position are (X1, Y1, Z1) and the coordinate values of the points on the line of sight are (X2, Y2, Z2), The point of is the radius of the distance between (X1, Y1, Z2) and (X2, Y2, Z2) around (X1, Y1, Z2) on the XY plane where (X2, Y2, Z2) exists. Move in a circle. The line-of-sight direction moves from the viewpoint position (X1, Y1, Z1) toward each point on the circle in accordance with the movement of the point on the line of sight.

  When the setting of the viewpoint position and the line-of-sight direction is completed, the moving direction, moving speed, and display range are set next. FIG. 14C is a setting screen for the moving direction, moving speed, and display range. The moving direction sets a direction in which the virtual viewpoint is moved when the peripheral images are continuously displayed, and the “right” direction or the “left” direction can be selected. When “right” is selected, the surrounding image is displayed clockwise from the virtual viewpoint as the starting point, and when “left” is selected, the surrounding image is displayed counterclockwise. Thus, the direction (right or left) selected by the user is set as the movement direction.

  The moving speed sets the moving speed of the virtual viewpoint when continuously displaying the peripheral images, and three stages of “fast”, “normal”, and “slow” can be selected. That is, when “fast” is selected, the peripheral images are continuously displayed fast, and when “slow” is selected, they are continuously displayed slowly. Thus, the speed selected by the user is set as the moving speed.

  In the setting screen for the moving direction and moving speed, the default setting or the currently selected setting is highlighted. The setting screen shown in FIG. 14C shows a state in which the moving direction is set to “left” and the moving speed is set to “normal”. The “left” button, the “normal” button, Is highlighted. For example, when the user selects “right” as the movement direction, the highlighting is changed from “left” to “right”.

  The display range is a viewing angle as described above, and is set according to the angle in the present embodiment. A button for changing the angle is displayed on the setting screen, and the desired angle is set by operating this button. Thus, the angle set by the user is set as the display range.

  The display range can be set as a range that equally spreads left and right around a straight line connecting the viewpoint position and the point on the line of sight. For example, when the display range is set to 20 degrees, the angle is set to 10 degrees on the right side and 10 degrees on the left side with respect to the straight line connecting the viewpoint position and the point on the line of sight. The upper and lower display ranges are set based on the aspect ratio of the display screen to be displayed and the set left and right display ranges.

  Note that a preview for confirming the set viewpoint data may be displayed during the setting of each data or after all the data is set. For example, the viewpoint position and line-of-sight direction set on the pseudo image of the vehicle are displayed, and the virtual viewpoint is moved in accordance with the set movement direction and movement speed. In order to display a preview from the setting screen for the viewpoint position and the line-of-sight direction, a “preview” button displayed on the setting screen may be pressed. If a preview is to be displayed after all data has been set, a “preview” button displayed on the screen when setting the moving direction or the like may be pressed.

  The preview display may be configured to actually create and display a peripheral image according to the set viewpoint data. These previews may be displayed on the sub display 6 or may be displayed on the main display 3. Thereby, since the user can confirm the viewpoint data set by himself / herself, if the user does not meet his / her wishes, it can be reset. If desired, the set data is registered as viewpoint data in association with the preset button by operating the registration button RB and the preset button PB. That is, when the registration button RB is operated, the screen changes to a screen on which the preset button PB is displayed, and the viewpoint data and the selected preset number are associated with each other by operating the desired preset number from the preset button PB. be registered. This viewpoint data is stored in a memory area associated with the selected preset button PB (selected preset number). In addition, the registered viewpoint data can be read and set from the corresponding memory area just by pressing the preset button and display a desired peripheral image after the next time.

    <1-5. Processing of image display system>

  Next, processing of the image display system 10 will be described. FIG. 15 is a flowchart showing a process flow of the image display system 10.

  The processing of the image display system 10 is started, for example, by displaying a screen for confirming whether or not to display a peripheral image when the system is activated. First, the user selects whether or not to display a peripheral image (step S1501). If the peripheral image is not displayed (No in step S1501), the process ends. On the other hand, when displaying a peripheral image (Yes in step S1501), the image composition unit 22 selects the viewpoint data 27b (step S1502). Specifically, the user selects the viewpoint data 27b when selecting the display of the peripheral image. Therefore, the image composition unit 22 reads the viewpoint data 27b selected by the user by operating the preset button PB or the like from the storage unit, and uses the viewpoint data 27b as data used to create a peripheral image or the like.

  Next, a peripheral image is created using the read viewpoint data 27b (step S1503). That is, the image composition unit 22 generates a composite image based on the virtual viewpoint and the display range, and the image adjustment unit 23 creates a peripheral image in which the transmission image is superimposed on the composite image.

  Specifically, the image composition unit 22 creates a plurality of composite images according to the display interval when the peripheral images are continuously displayed. For example, when the display interval of continuous display is 10 msec, the image composition unit 22 samples every 10 msec while moving the virtual viewpoint according to the set moving speed, and uses the sampled point as the virtual viewpoint. Create a composite image. Then, the image adjustment unit 23 creates a peripheral image in which a transparent image is superimposed on each created composite image.

  Next, the image adjustment unit 23 creates an auxiliary image (step S1504). When the image adjustment unit 23 creates a peripheral image, the image adjustment unit 23 creates an auxiliary image corresponding thereto. Specifically, scope images in the same direction and the same range are created based on the line-of-sight direction of the surrounding image and the display range. An auxiliary image is created for each peripheral image. The auxiliary image may be created after the peripheral image is created, or may be created simultaneously with the creation of the peripheral image.

  Next, the image output unit 24 displays the created peripheral image and auxiliary image (step S1505). That is, the image output unit 24 displays the peripheral image on the main display 3 and displays the auxiliary image on the sub display 6. Each of these images is displayed by synchronizing both corresponding images. The image output unit 24 continuously displays surrounding images based on the moving direction and moving speed included in the viewpoint data 27b. That is, the image output unit 24 displays the peripheral images continuously in the order of the movement direction according to the movement speed. Thereby, the user can confirm the circumference | surroundings of the own vehicle 9 continuously with the moving direction and moving speed which he set.

  Next, the image output unit 24 determines whether or not the display of all surrounding images has been completed (step S1506). Whether or not the display of all the surrounding images has ended is whether or not the display of the peripheral images from the virtual viewpoint as the starting point to the virtual viewpoint as the ending point has ended (whether or not the display of the entire circumference of the vehicle has ended) It is. If the display of all the peripheral images has not been completed (No in step S1506), there are still peripheral images to be displayed, so the image output unit 24 performs control to display the remaining peripheral images (step S1506). S1505). On the other hand, when the display of all the peripheral images is completed (Yes in step S1506), the peripheral image and auxiliary image data are erased from the memory (step S1507), and the process ends.

  <2. Second Embodiment>

  Next, a second embodiment will be described. In the second embodiment, a method for setting viewpoint data by a method different from that in the first embodiment will be described. Since the system configuration and the like are the same as those in the first embodiment, the viewpoint data setting method will be described below.

    <2-1. Setting viewpoint data>

  In the second embodiment, the user operates the touch panel 61 of the sub display 6 to set the viewpoint data. On the viewpoint data setting screen, a pseudo image of the host vehicle viewed from above and a pseudo image of the vehicle viewed from the side are displayed on the sub display 6, and the viewpoint position, the line-of-sight direction, and the like are individually set. The points on the touch panel 61 of the sub display 6 are associated with the coordinate values of the three-dimensional coordinates described in the first embodiment, and the viewpoint data is set by associating the points operated by the user with the coordinate values. .

  A method for setting the viewpoint position will be described. In order to set the viewpoint position, first, the XY coordinate value of the viewpoint position is derived. On the viewpoint position setting screen, first, as shown in FIG. 16A, an image of the host vehicle viewed from above is displayed. When the user drags the touch panel 61 of the sub-display 6 (an operation for moving the touch panel without releasing the finger from the touch panel), the dragged line becomes the viewpoint position. That is, in the case of FIG. 16A, the viewpoint position starts from the start point 161, travels around the rear toward the rear of the vehicle, and ends at the end point 162.

  Then, the XY coordinate value of the viewpoint position is derived based on the dragged line. A point on the touch panel 61 of the sub display 6 is associated with a coordinate value on the XY plane of three-dimensional coordinates, and a line dragged by the user is converted into an XY coordinate value. Further, the point between the start point 161 and the end point 162 is extracted by sampling between the start point 161 and the end point 162, for example, every 10 msec, and a coordinate value corresponding to the extracted point is derived. Thereby, the XY coordinate values of the viewpoint positions from the start point 161 to the end point 162 are derived.

  Next, the Z coordinate value of the viewpoint position is derived. The sub display 6 displays an image of the host vehicle viewed from the side. When the user drags the touch panel 61 of the sub-display 6, the dragged line becomes the viewpoint position. That is, in the case of FIG. 16B, the viewpoint position starts from the start point 163 and ends at the end point 164.

  Then, the Z coordinate value of the viewpoint position is derived based on the dragged line. A point on the touch panel 61 of the sub-display 6 is associated with a coordinate value on the Z axis of the three-dimensional coordinate, and a line dragged by the user is converted into a Z coordinate value. Further, the point between the start point 163 and the end point 164 is extracted by sampling between the start point 163 and the end point 164, for example, every 10 msec, and a coordinate value corresponding to the extracted point is derived. Thereby, the Z coordinate value of the viewpoint position from the start point 163 to the end point 164 is derived.

  In addition, although the solid line which shows the movement locus | trajectory of the finger | toe currently displayed in FIG. 16, and the point which shows the viewpoints 161 * 163 and the end points 162 * 164 are displayed on the screen for convenience of explanation, in the actual setting screen, , It may or may not be displayed. The same applies to other similar operations.

  The XYZ coordinate values related to the viewpoint position are derived by the above operation. However, since the XY coordinate value and the Z coordinate value are derived from different operations, it is necessary to associate them. Therefore, a method of associating will be described with reference to FIG.

  FIG. 17 is a diagram illustrating a method of associating XY coordinate values and Z coordinate values derived for setting the viewpoint position. FIG. 17A shows each data of the derived XY coordinate values and Z coordinate values. The XY data indicates the XY coordinate value at each sampling point, and the Z data indicates the Z coordinate value at each sampling point. In the case of XY data, the coordinate value corresponding to the start point 161 is a1, and the coordinate value corresponding to the end point 162 is an. Further, a2 to an-1 are coordinate values of points sampled every 10 msec. Similarly, in the case of Z data, the coordinate value corresponding to the start point 163 is b1, the coordinate value corresponding to the end point 164 is bm, and the coordinate values of points sampled between them are b2 to bm−. 1.

  Since the time when the user operates the touch panel 61 when deriving XY data is different from the time when the touch panel 61 is operated when deriving Z data, the number of coordinate value data is different. For this reason, the operation time of the Z data is changed to the same length as that of the XY data, and the timing in the data of each coordinate value is changed. Then, the XY data is associated with the changed Z data. An example of this is shown in FIG.

  For example, when the time operated when deriving XY data is n and the time operated when deriving Z data is m, the time of Z data is multiplied by n / m. Then, the coordinate value b1 of the start point and the coordinate value bm of the end point are fixed at the first and last timings, and each coordinate value between them is changed so as to be the timing at the same time ratio as the timing before the change. Therefore, the time interval between b1 and b2 is n / m times.

  Then, the XY data and the Z data having the same operation time are compared, and the coordinate values of the Z data at the same time as the time of each coordinate value of the XY data are associated. That is, the coordinate value corresponding to a1 is b1, and the coordinate value corresponding to a2 is between b1 and b2 and is the coordinate value at the same time as the time when a2 was sampled. Since the Z data is a Z coordinate value, a coordinate value at a corresponding position on a straight line connecting the coordinate value b1 and the coordinate value b2 may be selected. Thereby, the XYZ coordinate values (c1 to cl) are determined, and the viewpoint position is set.

  When setting the viewpoint position, Z data may not be derived, but only XY data. In this case, a predetermined value may be adopted as the Z coordinate, for example, a coordinate value that is ½ the height of the host vehicle. Therefore, in this case, the process of matching the XY data and the Z data is not performed, and the viewpoint position is simply set by adding a predetermined Z coordinate value to the XY coordinate value. Become.

  Next, a method for setting the line-of-sight direction will be described. The line-of-sight direction is set by deriving a point on the line of sight and using the point on the line of sight and the viewpoint position. First, derivation of points on the line of sight will be described. The derivation of the point on the line of sight can be performed by the same method as the setting of the viewpoint position. As shown in FIG. 18A, XY data is derived based on a line dragged by the user with respect to an image of the host vehicle viewed from above. And as shown in FIG.18 (b), Z data is derived | led-out based on the line which the user dragged with respect to the image which looked at the own vehicle from the side.

  And the process which matches XY data and Z data is performed. FIG. 19A shows the derived XY data and Z data, and FIG. 19B shows the points on the line of sight derived by associating them. Also in this association processing, as in the case of the viewpoint position setting processing, the XY data and the Z data are changed so as to have the same time, and a coordinate value obtained by combining the timings of the respective data may be derived. . Thereby, XYZ coordinate values (f1 to fi) are determined, and points on the line of sight are derived.

  When a point on the line of sight is derived, the line-of-sight direction is set using the viewpoint position. Since the line-of-sight direction is a direction from the viewpoint position toward a point on the line of sight, it is necessary to derive a point on the line of sight corresponding to the viewpoint position. Again, since the time when the user operated the touch panel 61 when setting the viewpoint position and the time when the user operated the touch panel 61 when deriving a point on the line of sight are different, the above-described viewpoint position and line of sight Similar to the point derivation method, the corresponding coordinate values may be derived by combining the operation times. FIG. 20 is a diagram showing this example. The direction from the coordinate value of the associated viewpoint position toward the coordinate value of the point on the line of sight is the line-of-sight direction. As described above, the viewpoint position and the line-of-sight direction are set, and the virtual viewpoint is set. Note that when deriving points on the line of sight, Z data may not be derived and only XY data may be derived, as with the viewpoint position.

  Next, the moving direction is set. The moving direction may be set based on the direction operated when the user sets the viewpoint position or the line-of-sight direction. For example, when the user drags leftward from the left front of the vehicle, the moving direction is also leftward.

  Next, the moving speed is set. The moving speed can also be set based on the drag speed when the user sets the viewpoint position or the line-of-sight direction. For example, the speed may be the same as the speed dragged by the user, or may be a speed derived by performing predetermined arithmetic processing on the speed dragged by the user. In addition, the speed desired by the user can also be set, not limited to the case based on the drag speed of the user. For example, the periphery of the host vehicle may be divided into a plurality of areas, and the speed desired by the user may be set for each area. Specifically, as shown in FIG. 21, the host vehicle is divided into four regions, the forward moving speed is increased, the left side and the right side are made normal, and the rear side is made slower.

  Next, the display range is set. For setting the display range, first, as shown in FIG. 22A, XY data is derived based on a line dragged by the user with respect to an image of the host vehicle viewed from above. This can be performed by the same method as the setting of the viewpoint position and the line-of-sight direction described above. Since the aspect ratio of the main display 3 is determined, the vertical range is derived based on this aspect ratio and the derived XY data (that is, the horizontal range). Thereby, the display range is set.

  In this way, when the display range is derived based on the aspect ratio of the main display 3, it is not necessary to derive the Z data, but when displaying a peripheral image having an aspect ratio different from the aspect ratio of the main display 3. The Z data is derived to set the display range. For example, after the XY data is derived by the method shown in FIG. 22A, the Z data is obtained based on the line dragged by the user with respect to the image obtained by viewing the host vehicle from the side as shown in FIG. 22B. To derive. Then, the display range is set by associating the XY data with the Z data in the same manner as the setting of the viewpoint position and the like described above.

  In this case, since the aspect ratio of the main display 3 and the aspect ratio of the peripheral image are different, portions other than the peripheral image displayed on the main display 3 are not displayed (for example, black). Further, the display range set by deriving the XY data and the Z data may be changed by enlargement / reduction or the like so as to have the same aspect ratio as that of the main display 3.

  As described above, each data of the virtual viewpoint, moving direction, moving speed, and display range is set, and each of these data is used as viewpoint data, and this viewpoint data and preset buttons are registered in association with each other and stored in the storage unit. The setting of viewpoint data is completed. Note that when the user sets the viewpoint data by an operation such as dragging, the present invention is not limited to the case of operating the touch panel 61 of the sub display 6, and the same processing may be performed by operating the touch panel 31 of the main display 3. Good. Furthermore, the operation is not limited to dragging, and may be another operation as long as the same processing can be performed.

    <2-2. Viewpoint data setting processing>

  Next, viewpoint data setting processing according to the second embodiment will be described with reference to a flowchart. 23 and 24 are flowcharts showing viewpoint data setting processing.

  The viewpoint data setting process starts when the user selects a setting button. When the viewpoint data setting process is started, a viewpoint position setting screen is first displayed on the sub-display 6 (step S2301). Then, the viewpoint data setting unit 20b determines whether there is a drag operation by the user for setting the viewpoint position (step S2302). This determination is made based on whether or not the touch panel 61 of the sub display 6 has been operated.

  If there is no drag operation by the user (No in step S2302), it is determined whether a predetermined time has elapsed since the setting screen was displayed (step S2303). If the predetermined time has elapsed (Yes in step S2303), the user determines that the viewpoint data setting process is not performed and ends the process (A in FIG. 23). On the other hand, if the predetermined time has not elapsed (Yes in step S2303), a process of determining the presence / absence of an operation is executed again (step S2302).

  If it is determined in step S2302 that the presence or absence of the drag operation has been performed (Yes in step S2302), the viewpoint position set by the operation is stored in the storage unit 27 (step S2304). . This viewpoint position setting process can be performed by the method described above. In this case, steps S2302 and S2303 are similarly performed for both the XY data derivation process and the Z data derivation process.

  Next, a line-of-sight setting screen is displayed on the sub-display 6 (step S2305). Then, the viewpoint data setting unit 20b determines whether or not there is a drag operation by the user for setting the line-of-sight direction (step S2306). This determination is also made based on whether or not the touch panel 61 of the sub display 6 has been operated.

  If there is no drag operation by the user (No in step S2306), it is determined whether a predetermined time has elapsed since the setting screen was displayed (step S2307). If the predetermined time has not elapsed (No in step S2307), the process for determining the presence / absence of the operation is executed again (step S2305).

  On the other hand, if the predetermined time has elapsed (Yes in step S2307), the predetermined value is stored in the storage unit 27 as the line-of-sight direction (step S2308). As described above, when a predetermined time has elapsed without the user performing the line-of-sight direction setting process, a predetermined value is set as the line-of-sight direction.

  If it is determined in step S2306 that determines whether or not there is a drag operation (Yes in step S2306), the line-of-sight direction set by the operation is stored in the storage unit 27 (step S2309). . This line-of-sight direction setting process can also be performed by the method described above. In this case, steps S2306 and S2307 are performed in the same manner for both the XY data derivation process and the Z data derivation process.

  Next, a display range setting screen is displayed on the sub-display 6 (step S2310), and the process proceeds to the next process (B in FIG. 23). Then, the viewpoint data setting unit 20b determines whether there is a drag operation by the user for setting the display range (step S2401). This determination is also made based on whether or not the touch panel 61 of the sub display 6 has been operated.

  If there is no drag operation by the user (No in step S2401), it is determined whether a predetermined time has elapsed since the setting screen was displayed (step S2402). If the predetermined time has not elapsed (No in step S2402), processing for determining the presence / absence of an operation is executed again (step S2401).

  On the other hand, if the predetermined time has elapsed (Yes in step S2402), the predetermined value is stored in the storage unit 27 as a display range (step S2403). Also in the display range setting process, when a predetermined time elapses without the user performing the setting process, a predetermined value is set as the display range.

  If it is determined in step S2401 that determines whether or not there is a drag operation (Yes in step S2401), the display range set by the operation is stored in the storage unit 27 (step S2404). . Then, the moving direction and moving speed are stored in the storage unit 27 (step S2405). The moving direction and moving speed are set by the method described above, and the contents are stored.

  Next, it is determined whether or not there is a preview display instruction (step S2406). This is because when the setting of each data of the viewpoint data is completed, it may be desired to confirm the peripheral image displayed based on each data set by the user. If there is a preview display instruction (Yes in step S2406), the image generation apparatus 2 displays a preview based on each set data (step S2407). This display can be performed by displaying a peripheral image created based on each set data on the main display 3. However, other methods may be used. For example, a moving path of a virtual viewpoint based on each set data may be displayed on the pseudo image of the vehicle displayed on the sub display 6.

  Then, after displaying the preview, it is determined whether there is no problem with the contents of the preview (step S2408). If there is a problem with the contents of the preview (No in step S2408), it is necessary to change the contents of each data. Therefore, in order to redo the viewpoint data setting process from the beginning, the process returns to the process of displaying a screen for setting the viewpoint position again (FIG. 24C).

  On the other hand, if there is no problem with the contents of the preview (Yes in step S2408), each set data is registered and set as viewpoint data (step S2409). This registration is performed using a preset button or the like, and is one of the viewpoint data options for displaying the next and subsequent peripheral images. Then, the setting screen is deleted (step S2410), and the viewpoint data setting process is terminated.

  <3. Modification>

  In each of the above embodiments, when an auxiliary image is displayed on the sub-display 6, if there is an object to be watched such as an obstacle around the vehicle, a configuration for notifying the fact is added. Also good.

  For example, as shown in FIG. 25 (a), when an obstacle is present on the right front side of the vehicle, a display 251 for notifying the presence of the obstacle is added to a corresponding portion of the auxiliary image displayed on the sub display 6. . Thereby, the user can grasp that there is an obstacle around the own vehicle. Further, an arrow 252 or the like may be displayed in a corresponding direction on the peripheral image displayed on the main display 3. In this case, the direction in which the obstacle actually exists can be grasped as the direction from the passenger compartment.

  Further, as shown in FIGS. 25C and 25D, an auxiliary image to be displayed on the sub display 6 may be displayed on a part of the main display 3. In this case, since the display content of the sub-display 6 can be grasped only by looking at the main display 3, it is possible to easily recognize the presence of an obstacle. Further, in addition to the display of the auxiliary image, a display indicating a direction such as an arrow may be displayed on the main display 3.

  As mentioned above, although embodiment of this invention has been described, this invention is not limited to the said embodiment. Moreover, the said embodiment and each modification can be combined suitably.

  Further, in each of the above embodiments, it has been described that various functions are realized in software by the arithmetic processing of the CPU according to the program. However, some of these functions are realized by an electrical hardware circuit. May be. Conversely, some of the functions realized by the hardware circuit may be realized by software.

2 Image generation device 3 Main display 4 Operation buttons 5 Camera 6 Sub display 10 Image display system

Claims (13)

An image processing apparatus for processing an image,
Image acquisition means for acquiring a captured image of a camera mounted on the host vehicle;
Based on the captured image, generating means for generating a peripheral image showing the periphery of the host vehicle viewed from a virtual viewpoint, and an auxiliary image showing a region of the peripheral image,
Output means for outputting the peripheral image and the auxiliary image to a display device and causing the display device to display them;
An image processing apparatus comprising: The image processing apparatus according to claim 1.
The output means outputs the peripheral image to the first display device to be displayed on the first display device, and outputs the auxiliary image to the second display device to be displayed on the second display device. An image processing apparatus. The image processing apparatus according to claim 2,
The image processing apparatus, wherein the second display device includes a touch panel, and further includes setting means for setting a viewpoint position and a line-of-sight direction at the virtual viewpoint based on an operation of the touch panel. The image processing apparatus according to claim 3.
The image processing apparatus according to claim 1, wherein the setting unit sets a line-of-sight direction after setting the viewpoint position. The image processing apparatus according to claim 3 or 4,
When the operation of the touch panel is an operation to move without releasing a finger from the touch panel,
The image processing apparatus according to claim 1, wherein the setting unit also sets movement of a viewpoint position and a line-of-sight direction. The image processing apparatus according to claim 5.
The image processing apparatus, wherein the setting unit sets the viewpoint position and the line-of-sight direction by synchronizing the timing of movement of the viewpoint position and the timing of movement of the line-of-sight direction. The image processing apparatus according to any one of claims 3 to 6,
The image processing apparatus, wherein the setting unit sets a viewpoint position and a line-of-sight direction in a three-dimensional space. The image processing apparatus according to claim 7.
The setting means sets the vertical viewpoint position and line-of-sight direction after setting the viewpoint position and line-of-sight direction on a horizontal plane. The image processing apparatus according to any one of claims 5 to 8,
The image processing apparatus, wherein the setting means also sets the viewpoint position and the moving speed in the line-of-sight direction. The image processing apparatus according to claim 9.
The image processing apparatus characterized in that the setting means sets the moving speed in the viewpoint position and the line-of-sight direction according to the moving speed of the operation position on the touch panel. The image processing apparatus according to claim 9 or 10,
The setting unit divides the periphery of the host vehicle into a plurality of regions, and sets the viewpoint position and the movement speed in the line-of-sight direction for each region. The image processing apparatus according to any one of claims 3 to 11,
A storage means for storing the setting information set by the setting means;
The storage means stores the setting information in association with buttons displayed on the first display device or the second display device. An image processing system,
The image processing apparatus according to any one of claims 1 to 12,
A display device for displaying an image;
An image processing system comprising:

Images