JP2013225903A - Image processing device, image processing system and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique enabling a user to know at a glance which area in the periphery of a vehicle is displayed on a display device as a synthesis image, according to a viewpoint position of a virtual viewpoint by changing a visual field range in a model chart.SOLUTION: A synthesis image viewed from an arbitrary virtual viewpoint is generated on the basis of a plurality of images obtained by photographing the vehicle periphery using a plurality of cameras. Then, a model chart indicating the viewpoint position of the virtual viewpoint to the side face of the vehicle is output to the display device. Also, a model chart relative to the vehicle side face after changing the viewpoint position according to the operation of a vehicle user is output. Thus, the user can confirm the viewpoint position while easily imaging a positional relationship of the virtual viewpoint relative to the vehicle viewed from the side face.

Description

  The present invention relates to a technique for displaying an image on a display device mounted on a vehicle.

  In general, there is an apparatus that monitors an area around a vehicle by acquiring an image around the vehicle using a plurality of cameras mounted on the vehicle and displaying the image on a display device automatically or by a user operation. Further, there is disclosed a technique of a vehicle periphery monitoring device provided with a switch button for adjusting an angle of a virtual viewpoint so that an image taken by a camera is an image viewed from a plurality of virtual viewpoints different from the camera viewpoint ( For example, see Patent Document 1.)

JP 2004-32464 A

  However, the technique described in Patent Document 1 confirms whether the composite image desired by the user is displayed by displaying the composite image screen on the display device after adjusting the viewpoint position of the virtual viewpoint using the switch button. To do. When the composite image of the display range desired by the user is not displayed, there is a problem that it becomes a cumbersome operation of changing the viewpoint position of the virtual viewpoint by the switching button again from the setting screen and performing adjustment again.

  The present invention has been made in view of the above problems, and provides a technology that allows a user to know at a glance which area around the vehicle is displayed on a display device as a composite image according to the viewpoint position of a virtual viewpoint. The purpose is to do.

  In order to solve the above-mentioned problem, the invention of claim 1 is a composite image generating means for generating a composite image viewed from an arbitrary virtual viewpoint based on a plurality of images obtained by photographing the periphery of the vehicle with a plurality of cameras, An output unit that outputs to a display device a model diagram indicating a viewpoint position of the virtual viewpoint with respect to a side surface of the vehicle, wherein the output unit changes the viewpoint position according to an operation of a user of the vehicle. Is output.

  According to a second aspect of the present invention, in the image processing apparatus according to the first aspect, the output unit generates the synthesis generated in accordance with the changed viewpoint position together with the model diagram in which the viewpoint position is changed. Output an image.

  According to a third aspect of the present invention, in the image processing apparatus according to the first or second aspect, the viewpoint position is changed by the user's operation using two viewpoint position change icons.

Further, the invention of claim 4 is a composite image generating means for generating a composite image viewed from an arbitrary virtual viewpoint based on a plurality of images obtained by photographing the periphery of the vehicle with a plurality of cameras.
An output means for outputting to the display device a model diagram showing the visual field range of the virtual viewpoint with respect to the side surface of the vehicle, wherein the output means changes the visual field range according to an operation of a user of the vehicle. Is output.

  According to a fifth aspect of the present invention, in the image processing apparatus according to the fourth aspect, the output unit generates the composite generated according to the changed visual field range together with the model diagram in which the visual field range is changed. Output an image.

  According to the first to fifth aspects of the invention, by outputting a model diagram for the side surface of the vehicle whose viewpoint position has been changed in accordance with the operation of the user of the vehicle, the user can easily position the virtual viewpoint relative to the vehicle as viewed from the side surface. You can check the viewpoint position while imagining.

  Further, according to the invention of claim 2, by outputting a composite image generated in accordance with the changed viewpoint position together with the model diagram in which the viewpoint position is changed, the user can select which viewpoint of the generated composite image. You can check whether it is a composite image as seen from the position.

  In particular, according to the invention of claim 3, the viewpoint position is changed by a user operation using two viewpoint position change icons, so that the user can change the viewpoint position while confirming the change of the viewpoint position in the model diagram. Can be changed.

  Further, according to the invention of claim 4 in particular, by outputting a model diagram for the side surface of the vehicle in which the visual field range is changed according to the operation of the user of the vehicle, the user can set the visual range of the virtual viewpoint for the vehicle as viewed from the side surface. I can confirm.

  Further, according to the invention of claim 5, in particular, by outputting a composite image generated according to the changed visual field range together with a model diagram in which the visual field range is changed, the user can determine which visual field the generated composite image is. You can check whether it is a composite image from the range.

FIG. 1 is a diagram illustrating a configuration of an image processing system. FIG. 2 is a diagram illustrating a position where the in-vehicle camera is arranged in the vehicle. FIG. 3 is a diagram for explaining a method of generating a composite image. FIG. 4 is a diagram illustrating transition of operation modes of the image processing system. FIG. 5 is a diagram showing a first example of a model diagram. FIG. 6 is a diagram showing a second example of a model diagram. FIG. 7 is a diagram showing a first example in which both a model diagram and a composite image are displayed. FIG. 8 is a diagram showing a second example in which both the model diagram and the composite image are displayed. FIG. 9 is a flowchart of the viewpoint position setting process based on the model diagram.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<1-1. System configuration>
FIG. 1 is a block diagram showing a configuration of the image processing system 120. The image processing system 120 is mounted on a vehicle (in this embodiment, an automobile), generates an image by photographing the periphery of the vehicle, and uses the generated image as a navigation device 20 in the vehicle interior. It has a function of outputting to a display device. By using this image processing system 120, a user (typically a driver) of the image processing system 120 can grasp the state around the vehicle in almost real time.

  As shown in FIG. 1, an image processing system 120 includes an image processing device 100 that generates a peripheral image indicating the periphery of the vehicle and outputs image information to a display device such as a navigation device 20, and a camera that captures the periphery of the vehicle. And a photographing unit 5 having the above.

  The navigation device 20 provides navigation guidance to the user, and includes a display 21 such as a liquid crystal provided with a touch panel function, an operation unit 22 that is operated by the user, and a control unit 23 that controls the entire device. Yes. The navigation device 20 is installed on an instrument panel or the like of the vehicle so that the screen of the display 21 is visible from the user. Various instructions from the user are received by the operation unit 22 and the display 21 as a touch panel. The control part 23 is comprised as a computer provided with CPU, RAM, ROM, etc., and various functions including a navigation function are implement | achieved when CPU performs arithmetic processing according to a predetermined program.

  The navigation apparatus 20 is communicably connected to the image processing apparatus 100, and can transmit and receive various control signals to and from the image processing apparatus 100 and receive peripheral images generated by the image processing apparatus 100. . An image based on the function of the navigation device 20 alone is normally displayed on the display 21 under the control of the control unit 23, but the periphery showing the state of the periphery of the vehicle generated by the image processing device 100 under a predetermined condition An image is displayed. Thereby, the navigation device 20 also functions as a display device that receives and displays a peripheral image generated by the image processing device 100.

  The image processing apparatus 100 is configured as an ECU (Electronic Control Unit) whose main body 10 has a function of generating a peripheral image, and is arranged at a predetermined position of the vehicle. The image processing apparatus 100 includes an imaging unit 5 that captures the periphery of the vehicle. The image generation unit 5 generates a composite image viewed from a virtual viewpoint based on a captured image obtained by capturing the periphery of the vehicle. Functions as a device. The plurality of in-vehicle cameras 51, 52, and 53 provided in these photographing units 5 are arranged at appropriate positions on a vehicle different from the main body unit 10, but will be described in detail later.

  The main body 10 of the image processing apparatus 100 includes a control unit 1 that controls the entire apparatus, an image generation unit 3 that generates a peripheral image for display by processing a captured image acquired by the imaging unit 5, and a navigation device 20. And a navigation communication unit 42 that communicates with each other.

  Various instructions from the user received by the operation unit 22 or the display 21 of the navigation device 20 are received by the navigation communication unit 42 as control signals and input to the control unit 1. In addition, the image processing apparatus 100 includes a changeover switch 43 that receives an instruction to switch display contents from a user. A signal indicating a user instruction is also input to the control unit 1 from the changeover switch 43. As a result, the image processing apparatus 100 can operate in response to both a user operation on the navigation apparatus 20 and a user operation on the changeover switch 43. The changeover switch 43 is arranged at an appropriate position of the vehicle separately from the main body unit 10 so that the user can easily operate.

  The image generation unit 3 is configured as a hardware circuit capable of various image processing, and includes a composite image generation unit 31 as a main function.

  The composite image generation unit 31 generates a composite image viewed from an arbitrary virtual viewpoint around the vehicle based on the plurality of captured images acquired by the plurality of in-vehicle cameras 51, 52, and 53 of the photographing unit 5. A method in which the composite image generation unit 31 generates a composite image viewed from a virtual viewpoint will be described later.

  The navigation communication unit 42 outputs the composite image generated by the image generation unit 3 and the model diagram indicating the visual field range from the virtual viewpoint of the composite image to the navigation device 20 and the touch panel function of the navigation device 20. And a receiving unit 42b that receives information operated by the user from the display 21 and the operation unit 22 provided. Here, the model diagram indicates a plurality of candidate viewpoint positions of a virtual viewpoint that can be selected by the user for an image of a model vehicle imitating an actual vehicle, and the user uses the viewpoint position change icon for the viewpoint position. It is an image for changing. The viewpoint position can be changed to the viewpoint position of an arbitrary virtual viewpoint by operating the display 21 or the operation unit 22 having the touch panel function of the navigation device. Hereinafter, the combined image and the model diagram may be collectively referred to as image information.

  The output of the image information from the output unit 42a is performed based on the image information output instruction signal of the control unit 1. In response to this signal, for example, a model diagram showing a visual field range from a virtual viewpoint with respect to the vehicle that can be expressed by a composite image is output. Thereby, the user can confirm the model figure which shows the visual field range from the virtual viewpoint of the image displayed on the navigation apparatus 20 as a synthesized image.

  The output unit 42a can also output a composite image to the navigation device 20 together with the above model diagram. Thereby, while confirming the model figure which shows the visual field range from the virtual viewpoint with respect to the vehicle and the synthesized image generated based on the viewpoint position of the virtual viewpoint selected by the user on one screen of the navigation device 20, the viewpoint position can be changed. Can be set.

  The receiving unit 42b accepts a change in the viewpoint position of the virtual viewpoint described later from the user in a state where the model diagram is displayed on the navigation device 20, and as a result, the model diagram in which the viewpoint position of the virtual viewpoint is changed is output from the output unit 42a. Is output.

  The control unit 1 is configured as a computer including a CPU, a RAM, a ROM, and the like, and various control functions are realized by the CPU performing arithmetic processing according to a predetermined program. An image control unit 11 and a display control unit 12 shown in the figure show some of the functions of the control unit 1 realized in this way.

  The image control unit 11 controls image processing executed by the image generation unit 3. For example, the image control unit 11 instructs various parameters necessary for generating a composite image generated by the composite image generation unit 31.

  The display control unit 12 mainly performs control when displaying the image information processed by the image processing device 100 on the navigation device 20. For example, output control of the composite image information generated by the composite image generation unit 31 to the navigation device 20 and output control of the model diagram to the navigation device 20 are performed.

  The main body 10 of the image processing apparatus 100 further includes a nonvolatile memory 40, a card reading unit 44, and a signal input unit 41, which are connected to the control unit 1.

  The nonvolatile memory 40 is configured by a flash memory or the like that can maintain stored contents even when the power is turned off. The nonvolatile memory 40 mainly stores vehicle type data 4a and model diagram data 4b.

  The vehicle type data 4a is data corresponding to the type of vehicle required when the composite image generation unit 31 generates a composite image.

  The model diagram data 4b is data mainly including a vehicle and a plurality of virtual viewpoint viewpoint position candidates, and viewpoint position candidate change icons for changing the viewpoint position by a user operation. is there. The model diagram data 4b is output to the navigation device 20 via the output unit 42a in response to an image information output instruction signal from the display control unit 12 of the control unit 1.

  The card reading unit 44 reads a memory card MK that is a portable recording medium. The card reading unit 44 includes a card slot in which the memory card MK can be attached and detached, and reads data recorded on the memory card MK attached to the card slot. Data read by the card reading unit 44 is input to the control unit 1.

  The memory card MK is composed of a flash memory or the like that can store various data, and the image processing apparatus 100 can use various data stored in the memory card MK. For example, it is possible to update a program (firmware) that realizes the function of the control unit 1 by storing the program in the memory card MK and reading the program. Further, by storing vehicle type data corresponding to a vehicle of a type different from the vehicle type data 4a stored in the non-volatile memory 40 in the memory card MK, and reading this out and storing it in the non-volatile memory 40, image processing It is also possible for the system 120 to correspond to different types of vehicles.

  Moreover, the signal input part 41 inputs the signal from the various apparatuses provided in the vehicle. A signal from the outside of the image display system 120 is input to the control unit 1 via the signal input unit 41. Specifically, signals indicating various information are input to the control unit 1 from the shift sensor 81 and the vehicle speed sensor 82.

  From the shift sensor 81, the operation position of the shift lever of the transmission of the vehicle 9, ie, “P (parking)”, “D (forward)”, “N (neutral)”, “R (reverse)”, etc. The shift position is input. From the vehicle speed sensor 82, the traveling speed (km / h) of the vehicle 9 at that time is input.

<1-2. Shooting Department>
Next, the photographing unit 5 of the image processing apparatus 100 will be described in detail. The photographing unit 5 is electrically connected to the control unit 1 and operates based on a signal from the control unit 1.

  The photographing unit 5 includes a front camera 51, a back camera 52, and a side camera 53 that are in-vehicle cameras. Each of these on-vehicle cameras 51, 52, and 53 includes an image sensor such as a CCD or a CMOS and electronically acquires an image.

  FIG. 2 is a diagram illustrating positions where the in-vehicle cameras 51, 52, 53 are arranged on the vehicle 9. In the following description, the three-dimensional XYZ orthogonal coordinates shown in the figure are used as appropriate when indicating the direction and direction. The XYZ axes are fixed relative to the vehicle 9. Here, the X-axis direction is along the left-right direction of the vehicle 9, the Y-axis direction is along the straight traveling direction (front-rear direction) of the vehicle 9, and the Z-axis direction is along the vertical direction. For convenience, the + X side is the right side of the vehicle 9, the + Y side is the rear side of the vehicle 9, and the + Z side is the upper side.

  The front camera 51 is provided in the vicinity of the license plate mounting position at the front end of the vehicle 9, and its optical axis 51 a is directed in the straight traveling direction of the vehicle 9 (−Y side in the Y-axis direction in plan view). The back camera 52 is provided in the vicinity of the license plate mounting position at the rear end of the vehicle 9, and its optical axis 52 a is directed in the reverse direction of the vehicle 9 in the straight traveling direction (+ Y side in the Y-axis direction in plan view). Yes. The side cameras 53 are provided on the left and right door mirrors 93, respectively, and the optical axis 53a is directed to the outside along the left-right direction of the vehicle 9 (X-axis direction in plan view). Note that the mounting position of the front camera 51 and the back camera 52 is preferably approximately the center in the left and right, but may be slightly shifted in the left and right directions from the center in the left and right.

  As the lenses of these in-vehicle cameras 51, 52, and 53, fish-eye lenses are employed, and the in-vehicle cameras 51, 52, and 53 have an angle of view α of 180 degrees or more. For this reason, it is possible to shoot the entire periphery of the vehicle 9 by using the four in-vehicle cameras 51, 52, and 53.

<1-3. Image conversion processing>
Next, a method in which the composite image generation unit 31 of the image generation unit 3 generates a composite image that shows how the periphery of the vehicle 9 is viewed from an arbitrary virtual viewpoint based on a plurality of captured images obtained by the imaging unit 5. explain. When generating the composite image, the vehicle type data 4a stored in advance in the nonvolatile memory 40 is used. FIG. 3 is a diagram for explaining a method of generating a composite image.

  When the front camera 51, the back camera 52, and the side camera 53 of the photographing unit 5 are simultaneously photographed, four photographed images P1 to P4 that respectively indicate the front, rear, left side, and right side of the vehicle 9 are acquired. The That is, the four captured images P1 to P4 acquired by the imaging unit 5 include information indicating the entire periphery of the vehicle 9 at the time of shooting.

  Next, each pixel of the four captured images P1 to P4 is projected onto the three-dimensional curved surface SP2 in the virtual three-dimensional space. The three-dimensional curved surface SP2 has, for example, a substantially hemispherical shape (a bowl shape), and a center portion (a bottom portion of the bowl) is determined as a position where the vehicle 9 exists. A correspondence relationship is determined in advance between the positions of the pixels included in the captured images P1 to P4 and the positions of the pixels of the solid curved surface SP2. For this reason, the value of each pixel of the solid curved surface SP2 can be determined based on this correspondence and the value of each pixel included in the captured images P1 to P4.

  Note that, as described above, the in-vehicle cameras 51, 52, and 53 use the wide-angle camera having the angle of view α of 180 degrees or more for capturing the captured images P1 to P4. When shooting with a wide-angle camera in this way, part of the image is obstructed by obstacles such as the camera hood and filter frame, and the amount of light in the surrounding area is reduced. May occur. This phenomenon of shadowing is generally called “vignetting”.

  The solid curved surface SP1 shown in FIG. 3 shows a state in which a certain amount of vignetting of the captured images P1 to P4 causes a shadow due to a decrease in the amount of light in a predetermined area around the solid curved surface SP1 on which these images are projected. Yes. When the three-dimensional curved surface that is shaded in this way is displayed on the navigation device 20 as it is, the synthesized image viewed from a predetermined virtual viewpoint does not become a substantially hemispherical shape (a bowl shape).

  For this reason, a composite image corresponding to an arbitrary virtual viewpoint is used by using the solid curved surface SP2 which is a substantially hemispherical (bowl-shaped) region in the center excluding the peripheral region where the light amount is reduced due to vignetting in the solid curved surface SP1. Generate. For example, as illustrated in FIG. 4, the solid curved surface SP <b> 2 is formed by removing the peripheral region where the light amount is reduced due to vignetting with the broken line portion of the solid curved surface SP <b> 1 as a boundary. As a result, an image of a substantially hemispherical (bowl-shaped) subject can be formed, and an image that can grasp the positional relationship between the vehicle and the obstacle displayed in a three-dimensional manner as if the user looked into the bowl from above can be provided. .

  In the above description, the processing in the case where the amount of light is reduced due to vignetting has been described as an example.

  The correspondence between the positions of the pixels of the captured images P1 to P4 and the positions of the pixels of the three-dimensional curved surface SP is determined by the arrangement of the four on-vehicle cameras 51, 52, and 53 in the vehicle 9 (the distance between each other, the ground height, Depending on the optical axis angle, etc.). For this reason, the table data indicating this correspondence is included in the vehicle type data 4 a stored in the nonvolatile memory 40.

  Further, polygon data indicating the shape and size of the vehicle body included in the vehicle type data 4a is used, and a vehicle image that is a polygon model indicating the three-dimensional shape of the vehicle 9 is virtually configured. The configured vehicle image is arranged in a substantially hemispherical central portion determined as the position of the vehicle 9 in the three-dimensional space where the three-dimensional curved surface SP is set.

  Further, the virtual viewpoint VP is set by the control unit 1 for the three-dimensional space where the solid curved surface SP exists. The virtual viewpoint VP is defined by the viewpoint position and the visual field direction, and is set to an arbitrary visual field position corresponding to the periphery of the vehicle 9 in this three-dimensional space toward an arbitrary visual field direction.

  Then, according to the set virtual viewpoint VP, a necessary area on the three-dimensional curved surface SP2 is cut out as an image as described above. The relationship between the virtual viewpoint VP and a necessary area in the three-dimensional curved surface SP is determined in advance, and is stored in advance in the nonvolatile memory 40 or the like as table data. On the other hand, rendering is performed on the vehicle image composed of polygons according to the set virtual viewpoint VP, and the resulting two-dimensional vehicle image is superimposed on the cut out image. Thereby, the composite image which shows a mode that the vehicle 9 and the periphery of the vehicle 9 were seen from arbitrary virtual viewpoints will be produced | generated.

  For example, when the virtual viewpoint VP11 in which the viewpoint position is a position just above the center of the position of the vehicle 9 and the visual field direction is a direction immediately below the vehicle 9 is set, the vehicle 9 ( Actually, a composite image CP1 showing the vehicle image) and the surroundings of the vehicle 9 is generated. Also, as shown in the figure, when the virtual viewpoint VP12 is set with the viewpoint position at the left rear of the position of the vehicle 9 and the visual field direction substantially in front of the vehicle 9, the entire periphery from the left rear of the vehicle 9 is set. As seen, a composite image CP <b> 2 is generated that shows the vehicle 9 (actually a vehicle image) and the surroundings of the vehicle 9.

  In the case of actually generating a composite image, it is not necessary to determine the values of all the pixels of the three-dimensional curved surface SP2, and only the values of the pixels in the area necessary corresponding to the set virtual viewpoint VP are photographed. By determining based on the images P1 to P4, the processing speed can be improved.

<1-4. Operation mode>
Next, the operation mode of the image processing system 120 will be described. FIG. 4 is a diagram illustrating transition of operation modes of the image processing system 120. The image processing system 120 has four operation modes: a navigation mode M0, a surrounding confirmation mode M1, a front mode M2, and a back mode M3. These operation modes can be switched by the control of the control unit 1 in accordance with the operation of the driver and the traveling state of the vehicle 9.

  The navigation mode M0 is an operation mode in which a map image for navigation guidance is displayed on the display 21 by the function of the navigation device 20. In the navigation mode M0, the functions of the image processing apparatus 100 are not used, and various displays are performed using the functions of the navigation apparatus 20 alone. For this reason, when the navigation device 20 has a function of receiving and displaying television broadcast radio waves, a television broadcast screen may be displayed instead of the map image for navigation guidance.

  On the other hand, in the surrounding confirmation mode M1, the front mode M2, and the back mode M3, an operation for displaying on the display 21 a display image that shows the situation around the vehicle 9 in real time by using the function of the image processing apparatus 100. Mode.

  The surrounding confirmation mode M <b> 1 is an operation mode in which an animation is expressed such that the vehicle 9 circulates around the vehicle 9 while looking down at the vehicle 9. The front mode M2 is an operation mode for displaying a display image that mainly indicates the front or side of the vehicle 9 that is required when moving forward. The back mode M3 is an operation mode for displaying a display image that mainly indicates the rear of the vehicle 9 that is required when the vehicle is moving backward.

  When the image processing system 120 is activated, it first enters the surrounding confirmation mode M1. In the case of the surrounding confirmation mode M1, when a predetermined time (for example, 6 seconds) elapses after the animation expression that goes around the vehicle 9 is made, the mode is automatically switched to the front mode M2. Further, in the case of the front mode M2, for example, when the changeover switch 43 is continuously pressed for a predetermined time or more while the traveling speed is 0 km / h (stopped state), the surrounding confirmation mode M1 is switched. Note that the surrounding confirmation mode M1 may be switched to the front mode M2 in accordance with a predetermined instruction from the driver.

  Further, in the case of the front mode M2, when the traveling speed becomes, for example, 10 km / h or more, the mode is switched to the navigation mode M0. Conversely, when the traveling speed input from the vehicle speed sensor 82 is less than 10 km / h in the navigation mode M0, the mode is switched to the front mode M2.

  When the traveling speed of the vehicle 9 is relatively high, the front mode M2 is canceled in order to concentrate the driver on traveling. On the other hand, when the traveling speed of the vehicle 9 is relatively low, the driver drives in consideration of the situation around the vehicle 9, more specifically, approaching an intersection with poor visibility, changing direction, or shifting the width. There are a lot of scenes that do. For this reason, when the traveling speed is relatively low, the navigation mode M0 is switched to the front mode M2. When switching from the navigation mode M0 to the front mode M2, a condition that there is an explicit operation instruction from the driver may be added to the condition that the traveling speed is less than 10 km / h.

  In the case of the navigation mode M0, when the changeover switch 43 is continuously pressed for a predetermined time or more while the traveling speed is, for example, 0 km / h (stopped state), the surrounding confirmation mode M1 is switched. Then, when a predetermined time (for example, 6 seconds) elapses after the animation expression is made to circulate around the vehicle 9, the mode is automatically switched to the navigation mode M2 which is the original mode.

  In the navigation mode M0 or the front mode M2, when the position of the shift lever input from the shift sensor 81 is “R (reverse)”, the mode is switched to the back mode M3. That is, when the transmission of the vehicle 9 is operated to the “R (reverse)” position, since the vehicle 9 is in a reverse state, the vehicle 9 is switched to the back mode M3 mainly showing the rear of the vehicle 9.

On the other hand, in the case of the back mode M3, when the position of the shift lever is other than “R (reverse)”, the mode is switched to the navigation mode M0 or the front mode M2 based on the traveling speed at that time. That is, if the traveling speed is 10 km / h or more, the mode is switched to the navigation mode M0, and if the traveling speed is less than 10 km / h, the mode is switched to the front mode M2.
<1-5. Model diagram>
Next, a model diagram output from the output unit 42a of the navigation communication unit 42 included in the image processing apparatus 100 and displayed on the navigation device 20 will be described with reference to FIG. 5 showing a first example of the model diagram. To do.

  In the model diagram MD1 shown in FIG. 5, candidate positions VP1 to VP5 of a plurality of virtual viewpoints are provided for the model vehicle MC. The user changes these candidates to arbitrary viewpoint positions by operating the viewpoint position change icon 61. This operation is performed using the display 21 or the operation unit 22 having the touch panel function of the navigation device 20.

  In the example shown in FIG. 5, the viewpoint position VP1 located behind the model vehicle MC is selected, and the visual field range FE1 from the viewpoint position VP1 is displayed with diagonal lines. The visual field range FE1 corresponds to the display range of the composite image displayed on the navigation device 20 in the back mode or the front mode, for example.

  The return button 71 is selected by the user when returning to a previous setting screen (not shown). The completion button 72 is selected by the user when the change information of the position of the virtual viewpoint is stored in the nonvolatile memory 40 and the screen returns to the setting menu screen (not shown) which is the previous screen. Then, based on the set viewpoint position of the virtual viewpoint, the composite image in the back mode or the front mode is displayed.

  Next, a model diagram showing a visual field range from another viewpoint position will be described as a second example with reference to FIG. 6 differs from the model diagram shown in FIG. 5 in that the viewpoint position of the virtual viewpoint is changed from the viewpoint position VP1 behind the model vehicle MC to the viewpoint position VP3 immediately above (directly above) the model vehicle MC. is there. This change is performed based on the operation of the display 21 or the operation unit 22 having a touch panel function of the user.

  With the change of the viewpoint position, the visual field range is also changed from the visual field range of FE1 corresponding to the visual point position VP1 to the visual field range FE3 corresponding to the visual point position VP3. Thereby, the change of the viewpoint position of the virtual viewpoint can be accepted while outputting the model diagram indicating the visual field range from the virtual viewpoint to the vehicle. In addition, by outputting a model diagram in which the visual field range is changed according to the change in the viewpoint position of the virtual viewpoint, any region around the vehicle is displayed as a composite image on the display device according to the viewpoint position of the virtual viewpoint. The user knows at a glance whether or not For this reason, if the composite image desired by the user is not shown as a result of displaying and confirming the composite image after setting the position of the virtual viewpoint, the troublesome work of redoing the setting can be eliminated.

  Further, the display mode of the selected viewpoint position is made different from the display modes of other viewpoint position candidates. Thereby, the user can confirm at a glance the viewpoint position selected by viewing the display device.

  As a method of making the display mode of the viewpoint position selected by the user different from the display mode of the viewpoint position not selected in this way, the brightness of each display can be changed. For example, the selected viewpoint position is a bright color (color that is illuminated by light such as yellow or red), and the unselected viewpoint position is a dark color (black or brown light is blocked) Color).

  Note that when the viewpoint position is changed from VP1 to VP3, the vicinity of the roof spoiler at the upper end of the rear glass at the rear of the vehicle (the blind spot area FS3 in black in FIG. 6) is the viewpoint position VP3 of the virtual viewpoint. It is displayed as a range that becomes a blind spot. Thus, the range which becomes a blind spot from the viewpoint position selected by the user is set to a display mode different from the display mode of the visual field range FE3. Thereby, it is possible to confirm at a glance a portion that becomes a blind spot from the virtual viewpoint set by the user and is not displayed by the composite image.

  As a method of making the display mode of the visual field range different from the display mode of the range that becomes the blind spot, the brightness of both is changed. For example, the field of view is a bright color (color that is illuminated by light such as yellow or red), and the range that becomes a blind spot is dark (color that is blocked by light such as black or brown). .

  Further, as another example, a first example in which a model diagram and a composite image are combined and displayed on the navigation device 20 is shown in FIG. 7, and a second example is shown in FIG. In FIG. 7, the composite image CI1 displayed corresponding to the display range FE1 is displayed on one screen of the navigation device 20 together with the model diagram described in FIG. As a result, the user can confirm the viewpoint position while actually confirming the model diagram showing the visual field range from the virtual viewpoint with respect to the vehicle and the composite image generated based on the viewpoint position of the virtual viewpoint selected by the user on one screen. Can be set.

  FIG. 8 displays the composite image CI3 displayed corresponding to the display range FE3 on one screen of the navigation device 20 together with the model diagram described in FIG. Then, the model diagram and the composite image are changed from FIG. 7 to FIG. 8 by the user operating the viewpoint position change icon 61. Thus, the user can change the viewpoint position while confirming the viewpoint position of the virtual viewpoint and the display range of the composite image displayed on the navigation device 20 corresponding to the viewpoint position.

  In the above-described embodiment, the viewpoint position is changed from VP1 to VP3. However, the viewpoint position can be changed to other viewpoint positions, and the five viewpoint positions described in the embodiment can be changed. There is no limitation, and viewpoint positions may be provided at positions other than these five positions. Further, the number of viewpoint positions may be reduced from five.

  Further, when both the model diagram and the composite image are displayed, they may be displayed as separate screens in addition to being displayed on one screen.

<2. Operation>
Next, the viewpoint position setting process based on the model diagram will be described with reference to the flowchart shown in FIG. The image processing apparatus 100 receives a setting button pressing signal indicating that the user has operated the display 21 or the operation unit 22 having a touch function in order to set the viewpoint position of the virtual viewpoint of the composite image (Yes in step S101). Then, information on a setting screen (not shown) stored in the nonvolatile memory 40 is output to the navigation device 20 (step S102), and the process proceeds to step S103, which is the next process. If the setting button pressing signal has not been received (No in step S101), the process ends.

  Next, in step S103, when a signal indicating that the viewpoint position setting button of the virtual viewpoint in the setting screen is pressed is received (step S103 is Yes), a model diagram showing a visual field range corresponding to the currently set viewpoint position. Is read from the model diagram data 4b of the nonvolatile memory 40 and output to the navigation device 20 via the output unit 42a (step S104). If the receiving unit 42b of the image processing apparatus 100 has not received a pressing signal for the virtual viewpoint viewpoint position setting button (No in step S103), the process ends.

  When the viewpoint position is changed to a predetermined position by the user operating the viewpoint position change icon 61 of the model diagram displayed on the navigation device 20, the reception unit 42b receives the signal (Yes in step S105). ) And the changed viewpoint position information is stored in the nonvolatile memory 40 (step S106).

  If the reception unit 42b has not received a signal associated with a change in viewpoint position, the model diagram of the same viewpoint position is displayed as it is on the navigation device 20, and the process proceeds to step S108, which is the next process.

  After the information on the viewpoint position is stored in step S106, model diagram information indicating the visual field range corresponding to the viewpoint position at the changed viewpoint position is read from the model diagram data 4b and sent from the output unit 42a to the navigation device 20. Output (step S107). Thereby, the change of the viewpoint position of the virtual viewpoint is accepted while outputting the model diagram showing the visual field range from the virtual viewpoint to the vehicle.

  Then, by outputting a model diagram in which the visual field range is changed according to the change of the viewpoint position of the virtual viewpoint, any region around the vehicle is displayed as a composite image on the display device according to the viewpoint position of the virtual viewpoint. At a glance. For this reason, if the composite image desired by the user is not shown as a result of displaying and confirming the composite image after setting the position of the virtual viewpoint, the troublesome work of redoing the setting can be eliminated.

  Next, when a signal indicating that the completion button 72 in the model diagram has been pressed is received (Yes in step S108), the process ends with the setting of the changed viewpoint position, and a screen before the setting button is pressed (for example, Return to the navigation mode screen. If the signal indicating that the completion button has been pressed is not received (No in step S108), the model diagram is displayed as it is on the navigation device 20. Alternatively, the screen returns to the navigation mode screen after a predetermined time has elapsed.

Note that the model data 4b stored in the non-volatile memory 40 described in the above processing, setting screen data (not shown), and the like may be stored in a memory (not shown) provided in the navigation device 20.
<3. Modification>
Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications can be made. Below, such a modification is demonstrated. All forms including those described in the above embodiment and those described below can be combined as appropriate.

  In the above embodiment, the image processing apparatus 100 and the navigation apparatus 20 are described as separate apparatuses. However, the image processing apparatus 100 and the navigation apparatus 20 are arranged in the same casing and configured as an integrated apparatus. May be.

  In the above embodiment, the display device that displays the image generated by the image processing device 100 is described as the navigation device 20, but a general display device that does not have a special function such as a navigation function. It may be.

  In the above embodiment, some of the functions described as being realized by the control unit 1 of the image processing apparatus 100 may be realized by the control unit 23 of the navigation apparatus 20.

  In the above embodiment, part or all of the signals described as being input to the control unit 1 of the image processing apparatus 100 via the signal input unit 41 are input to the navigation apparatus 20. Also good. In this case, the signal may be input to the control unit 1 of the image processing apparatus 100 via the navigation communication unit 42.

  Further, in the above-described embodiment, it has been described that various functions are realized in software by the arithmetic processing of the CPU according to the program. However, some of these functions are realized by an electrical hardware circuit. Also good. Conversely, some of the functions realized by the hardware circuit may be realized by software.

1 Control unit
3 Image generator 5 Imager 20 Navigation device 40 Non-volatile memory 41 Signal input unit 42 Navigation communication unit 81 Shift sensor 82 Vehicle speed sensor

Claims (5)

Composite image generation means for generating a composite image viewed from an arbitrary virtual viewpoint based on a plurality of images obtained by photographing the periphery of the vehicle with a plurality of cameras;
An output means for outputting a model diagram showing a viewpoint position of the virtual viewpoint with respect to a side surface of the vehicle to a display device;
The output means outputs the model diagram in which the viewpoint position is changed according to an operation of a user of the vehicle;
An image processing apparatus.
The image processing apparatus according to claim 1.
The output means outputs the composite image generated according to the changed viewpoint position together with the model diagram in which the viewpoint position has been changed.
An image processing apparatus.
The image processing apparatus according to claim 1 or 2,
The viewpoint position is changed by the user's operation using two viewpoint position change icons;
An image processing apparatus.
Composite image generation means for generating a composite image viewed from an arbitrary virtual viewpoint based on a plurality of images obtained by photographing the periphery of the vehicle with a plurality of cameras;
Output means for outputting a model diagram showing a visual field range of the virtual viewpoint with respect to a side surface of the vehicle to a display device;
The output means outputs the model diagram in which the visual field range is changed according to an operation of a user of the vehicle;
An image processing apparatus.
The image processing apparatus according to claim 4.
The output means outputs the composite image generated according to the changed visual field range together with the model diagram in which the visual field range is changed,
An image processing apparatus.

Images