JP5643028B2 - Image display system, image processing apparatus, and image processing method - Google Patents

Description

  The present invention relates to an image display technique in a vehicle.

  Conventionally, when a vehicle is parked, the size of the parking space is determined by the user's visual observation. And as a result of starting parking in the said space, there was a case where the parking space had to be abandoned because the parking space was narrower than the size of the vehicle. Further, when the size of the vehicle and the size of the parking space are substantially the same, there is a possibility that a user unaccustomed to driving may collide the vehicle body with an obstacle outside the parking space when parking.

  On the other hand, Patent Document 1 discloses a technique for determining whether or not parking is possible in a parking space. In this technique, an image of a side of the vehicle is taken by a camera provided in front of the vehicle and displayed on a monitor. Based on the displayed image, the user selects one end of the parking space as a start position. Then, after the start point position is determined, the vehicle is advanced, and the user selects the other end of the parking space as the end point position, thereby measuring the distance in the traveling direction of the vehicle in the parking space, and calculating the measured distance and the length of the vehicle body. In comparison, it is determined whether or not parking is possible.

JP 2007-91046 A

  However, in the technique disclosed in Patent Document 1, since the user selects the start point position and the end point position based on an image in which the vehicle of the host vehicle from the side of the vehicle is not displayed, The positional relationship between the start point position and end point position of the parking space is not clear. For this reason, it is difficult to accurately set the positions of the start point and end point of the parking space and accurately measure the distance of the parking space along the traveling direction of the vehicle. Moreover, when a user wanted to measure the distance from arbitrary positions other than a parking space, it was not able to measure.

  An object of the present invention is to accurately measure the distance in correspondence with the vehicle position of the own vehicle.

In order to solve the above-mentioned problems, an invention according to claim 1 is an image display system mounted on a vehicle, and overlooks the vehicle and a surrounding area of the vehicle from a captured image obtained by a camera arranged in the vehicle. Generating means for generating the overhead image, display means for displaying the overhead image, a first reference position specified in accordance with a user operation, and a second reference position based on the specific part of the vehicle, Setting means for setting during the display of the overhead image, and derivation means for deriving a first distance that is a distance between the first reference position and the second reference position, and the display means includes the first reference position A linear line image that connects a position and the second reference position, and changes in length according to a change in the first distance accompanying the movement of the vehicle, and the movement of the vehicle causes the first distance to be Less than the second distance required for parking the vehicle When a, I change the display mode of the line image.

According to a second aspect of the present invention, in the image display system according to the first aspect, the derivation means is configured to move the vehicle after the first reference position and the second reference position are the same position. Derivation of the first distance is started when the first reference position and the second reference position are different .

According to a third aspect of the present invention, in the image display system according to the first or second aspect, when the first reference position is set , the display means extends from the front end of the vehicle in the left-right direction of the vehicle. The point on the extended line is displayed as an index .

According to a fourth aspect of the present invention, in the image display system according to any one of the first to third aspects, the specific part of the vehicle is one of a rear end portion of the vehicle and a front end portion of the vehicle .

Further, the invention of claim 5 is an image processing device for generating an image to be displayed on a display device mounted on a vehicle, and from the captured image obtained by a camera disposed in the vehicle, the vehicle and the vehicle Generation means for generating an overhead image overlooking the surrounding area, output means for outputting and displaying the overhead image on the display device, a first reference position specified according to a user operation, and specification of the vehicle Setting means for setting a second reference position based on a part during display of the overhead image, and derivation means for deriving a first distance that is a distance between the first reference position and the second reference position. The output means connects the first reference position and the second reference position, and outputs a linear line image whose length changes according to a change in the first distance due to the movement of the vehicle; As the vehicle moves, the first distance There when it becomes the second distance or more necessary parking of the vehicle, and displays and outputs the line image display mode is changed.

The invention of claim 6 is an image display method of an image processing device for generating an image to be displayed on a display device mounted on a vehicle, and (a) from a photographed image obtained by a camera arranged in the vehicle. , said vehicle and generating a bird's-eye view image looking down the peripheral region of the vehicle, a step of displaying outputs a (b) the overhead image on the display device, is specified in response to the operation of (c) user A step of setting a first reference position and a second reference position based on a specific part of the vehicle during display of the overhead image, and (d) a distance between the first reference position and the second reference position A step of deriving a first distance, wherein the step (b) connects the first reference position and the second reference position and responds to a change in the first distance as the vehicle moves. Output a linear line image of varying length , When the first distance becomes greater than or equal to the second distance necessary for parking the vehicle due to the movement of the vehicle, the line image in which the display mode is changed is output and displayed .

  According to the first to tenth aspects of the present invention, the information about the first distance that changes due to the movement of the vehicle is notified to the user, so that the user of the vehicle can accurately measure the distance from any position. be able to.

  According to the invention of claim 2, the positional relationship between the position of the vehicle and the parking space is clarified by notifying the user when the first distance is equal to or greater than the second distance necessary for parking the vehicle. Thus, the distance of the parking space can be accurately notified to the user.

  In particular, according to the invention of claim 3, when the display means sets the first reference position, the user can display the point on the line extending from the front end of the vehicle in the left-right direction of the vehicle as an index, so that the user can It is possible to grasp the relationship between the parking space that can be visually confirmed from the vehicle and the displayed bird's-eye view image while moving the vehicle, and to reliably set the position visually confirmed by the user.

  In particular, according to the invention of claim 4, the deriving means derives a first distance that is a distance between the second reference position and the first reference position, which is a position based on the rear end of the vehicle, so that the user Can start the parking operation immediately from the position of the vehicle at the distance required for parking.

  In particular, according to the invention of claim 5, the deriving means derives a first distance that is a distance between the second reference position that is a position based on the front end portion of the vehicle and the first reference position, so that the user can While deriving the first distance, the state of the parking space can be visually confirmed as the vehicle moves.

  According to a sixth aspect of the invention, the derivation means derives the first distance when the second reference position that moves as the vehicle moves away from the first reference position after contacting the first reference position. By starting the operation, the user can know the timing of the derivation of the distance, and by eliminating unnecessary steering operation after the start of the derivation of the distance, the accurate distance can be derived.

  In particular, according to the invention of claim 7, the display means displays the user's driving operation information necessary for deriving the first distance in association with the overhead image, so that the user derives the first distance. The necessary driving operation can be confirmed along with the composite image.

FIG. 1 is a block diagram of an image display system. FIG. 2 is a diagram illustrating a position where the in-vehicle camera is arranged in the vehicle. FIG. 3 is a diagram for explaining a method of generating a composite image. FIG. 4 is a diagram illustrating transition of operation modes of the image display system. FIG. 5 is a diagram for explaining the parallel parking mode according to the first embodiment. FIG. 6 is a process flowchart of the image processing apparatus. FIG. 7 is a diagram for explaining the parallel parking mode according to the second embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<First Embodiment>
<1-1. System configuration>
FIG. 1 is a block diagram of the image display system 120. The image display system 120 is mounted on a vehicle (in this embodiment, an automobile), generates an image by photographing the periphery of the vehicle, and uses the generated image as a navigation device 20 in the vehicle interior. It has a function of outputting to a display device. By using this image display system 120, a user (typically a driver) of the image display system 120 can grasp the state around the vehicle in almost real time.

  As shown in FIG. 1, an image display system 120 includes an image processing device 100 that generates a peripheral image indicating the periphery of a vehicle and outputs image information to a display device such as a navigation device 20, and a camera that captures the periphery of the vehicle. And a photographing unit 5 having the above.

  The navigation device 20 provides navigation guidance to the user, and includes a display 21 such as a liquid crystal provided with a touch panel function, an operation unit 22 that is operated by the user, and a control unit 23 that controls the entire device. Yes. The navigation device 20 is installed on an instrument panel or the like of the vehicle so that the screen of the display 21 is visible from the user. Various instructions from the user are received by the operation unit 22 and the display 21 as a touch panel. The control unit 23 is a computer including a CPU, a RAM, a ROM, and the like, and various functions including a navigation function are realized by the CPU performing arithmetic processing according to a predetermined program.

  The navigation apparatus 20 is communicably connected to the image processing apparatus 100, and can transmit and receive various control signals to and from the image processing apparatus 100 and receive peripheral images generated by the image processing apparatus 100. . An image based on the function of the navigation device 20 alone is normally displayed on the display 21 under the control of the control unit 23, but the periphery showing the state of the periphery of the vehicle generated by the image processing device 100 under a predetermined condition An image and a bird's-eye view of the vehicle 9 and the surrounding area of the vehicle 9 are displayed. Moreover, the bird's-eye view image displayed on the display 21 shows the peripheral area almost in real time. Thereby, the navigation device 20 also functions as a display device that receives and displays a peripheral image generated by the image processing device 100.

  The image processing apparatus 100 is an ECU (Electronic Control Unit) whose main body 10 has a function of generating a peripheral image, and is arranged at a predetermined position of the vehicle. The image processing apparatus 100 includes an imaging unit 5 that captures the periphery of the vehicle. The image generation unit 5 generates a composite image viewed from a virtual viewpoint based on a captured image obtained by capturing the periphery of the vehicle. Functions as a device. The plurality of in-vehicle cameras 51, 52, and 53 provided in these photographing units 5 are arranged at appropriate positions on a vehicle different from the main body unit 10, but will be described in detail later.

  The main body 10 of the image processing apparatus 100 includes a control unit 1 that controls the entire apparatus, an image generation unit 3 that generates a peripheral image for display by processing a captured image acquired by the imaging unit 5, and a navigation device 20. And a navigation communication unit 42 that communicates with each other.

  Various instructions from the user received by the operation unit 22 or the display 21 of the navigation device 20 are received by the navigation communication unit 42 as control signals and input to the control unit 1. As a result, the image processing apparatus 100 can operate in response to a user operation on the navigation apparatus 20.

  Further, the image processing apparatus 100 includes a parking assistance button 201. When the parking assist button 201 is operated by the user, a signal indicating the user's instruction is input to the control unit 1, and an image of a front mode M2 (shown in FIG. 4) described later is displayed. In addition, this parking assistance button 45 may be provided in the image processing apparatus 100 or the navigation apparatus 20 as a hard button. The parking assistance button 45 may be displayed on the display 21 of the navigation device 20 as a touch panel button that is operated by a user touching the display 21 with a finger or the like.

  The image generation unit 3 is configured as a hardware circuit capable of various image processing, and includes a composite image generation unit 31, an image range selection unit 32, and an image information output unit 33 as main functions.

  The composite image generation unit 31 generates a composite image viewed from an arbitrary virtual viewpoint around the vehicle based on the plurality of captured images acquired by the plurality of in-vehicle cameras 51, 52, and 53 of the photographing unit 5. A method in which the composite image generation unit 31 generates a composite image viewed from a virtual viewpoint will be described later.

  Further, the image information output unit 33 outputs the composite image information generated by the composite image generation unit 31 to the navigation device 20. As a result, a peripheral image showing the periphery of the vehicle is displayed on the display 21 of the navigation device 20.

  The control unit 1 is a computer including a CPU, a RAM, a ROM, and the like, and various control functions are realized by the CPU performing arithmetic processing according to a predetermined program. The image control unit 11, the position setting control unit 12, the distance derivation control unit 13, and the notification control unit 14 shown in the drawing show some of the functions of the control unit 1 realized in this way. Yes.

  The image control unit 11 controls image processing executed by the image generation unit 3. For example, the image control unit 11 instructs various parameters necessary for generating a composite image generated by the composite image generation unit 31. In addition, the image range selection unit 32 issues an instruction for selecting a predetermined range of an image captured by the side camera 53 based on parameter information for each vehicle type.

  The position setting control unit 12 and an index MA (shown in FIG. 5) displayed on the front end portion of the vehicle 9 and other vehicles or others in the parallel parking mode selected when the user described later parks the vehicle 9 in parallel. The measurement start position DA (shown in FIG. 5) is set by the user's selection based on the positional relationship with the position (for example, one point in the parking space area). That is, the index MA is fixed at the point selected by the user and becomes the measurement start position DA.

  The position setting control unit 12 also includes a measurement start position DA (first reference mark) that is relatively fixed to a peripheral area of the vehicle 9 shown in the overhead image (for example, the image M41 shown in FIG. 5) and the vehicle shown in the overhead image. 9 is performed so that the measurement position BA (second reference mark) relatively fixed to 9 is superimposed on the overhead image.

  The measurement start position DA is set by a user operation using the touch panel or hard button of the display 21.

  The distance derivation control unit 13 determines a measurement position BA based on a specific part of the vehicle 9 described later (for example, a front end portion of the vehicle 9 or a rear end portion of the vehicle 9) and a measurement start position DA set by the user's selection. Deriving the distance between. The measurement position BA is displayed on the display 21 based on a specific part of the vehicle 9 according to the setting of the measurement start position DA by the user, and moves in the traveling direction of the vehicle 9 as the vehicle 9 moves.

  When the vehicle 9 moves, the notification control unit 14 causes the distance between the measurement start position DA and the measurement position BA to be equal to or longer than the distance necessary for parking the vehicle 9 (for example, about 1.5 times the total length of the vehicle). To the user. Examples of notifications when the distance is longer than necessary for parking include changing the color and size of an image displayed on the display 21 and outputting sound from the navigation device 20.

  The main body 10 of the image processing apparatus 100 further includes a nonvolatile memory 40, a card reading unit 44, and a signal input unit 41, which are connected to the control unit 1.

  The nonvolatile memory 40 is configured by a flash memory or the like that can maintain stored contents even when the power is turned off. The nonvolatile memory 40 stores vehicle type data 4a. The vehicle type data 4a is data corresponding to the type of vehicle required when the composite image generation unit 31 generates a composite image. In addition, data such as a distance necessary for parking the vehicle is also recorded.

  The card reading unit 44 reads a memory card MC that is a portable recording medium. The card reading unit 44 includes a card slot in which the memory card MC can be attached and detached, and reads data recorded on the memory card MC installed in the card slot. Data read by the card reading unit 44 is input to the control unit 1.

  The memory card MC is configured by a flash memory or the like capable of storing various data, and the image processing apparatus 100 can use various data stored in the memory card MC. For example, a program (firmware) that realizes the function of the control unit 1 can be updated by storing a program in the memory card MC and reading the program. Further, the vehicle type data corresponding to the vehicle of a type different from the vehicle type data 4a stored in the nonvolatile memory 40 is stored in the memory card MC, and this is read out and stored in the nonvolatile memory 40, thereby displaying an image. It is also possible for the system 120 to correspond to different types of vehicles.

  Moreover, the signal input part 41 inputs the signal from the various apparatuses provided in the vehicle. A signal from the outside of the image display system 120 is input to the control unit 1 via the signal input unit 41. Specifically, signals indicating various information are input to the control unit 1 from the shift sensor 81, the vehicle speed sensor 82, the direction indicator 83, and the like.

  From the shift sensor 81, the operation position of the shift lever of the transmission of the vehicle 9, ie, “P (parking)”, “D (forward)”, “N (neutral)”, “R (reverse)”, etc. The shift position is input. From the vehicle speed sensor 82, the traveling speed (km / h) of the vehicle 9 at that time is input.

  From the direction indicator 83, a turn signal indicating a direction instruction based on the operation of the turn signal switch, that is, a direction instruction intended by the driver of the vehicle is input. When the turn signal switch is operated, a turn signal is generated, and the turn signal indicates the operated direction (left direction or right direction). When the turn signal switch is in the neutral position, the turn signal is turned off.

<1-2. Shooting Department>
Next, the photographing unit 5 of the image processing apparatus 100 will be described in detail. The photographing unit 5 is electrically connected to the control unit 1 and operates based on a signal from the control unit 1.

  The photographing unit 5 includes a front camera 51, a back camera 52, and a side camera 53 that are in-vehicle cameras. Each of these on-vehicle cameras 51, 52, and 53 includes an image sensor such as a CCD or a CMOS and electronically acquires an image.

  FIG. 2 is a diagram illustrating positions where the in-vehicle cameras 51, 52, 53 are arranged on the vehicle 9. In the following description, the three-dimensional XYZ orthogonal coordinates shown in the figure are used as appropriate when indicating the direction and direction. The XYZ axes are fixed relative to the vehicle 9. Here, the X-axis direction is along the left-right direction of the vehicle 9, the Y-axis direction is along the straight traveling direction (front-rear direction) of the vehicle 9, and the Z-axis direction is along the vertical direction. For convenience, the + X side is the right side of the vehicle 9, the + Y side is the rear side of the vehicle 9, and the + Z side is the upper side.

  The front camera 51 is provided in the vicinity of the license plate mounting position at the front end of the vehicle 9, and its optical axis 51 a is directed in the straight traveling direction of the vehicle 9 (−Y side in the Y-axis direction in plan view). The back camera 52 is provided in the vicinity of the license plate mounting position at the rear end of the vehicle 9, and its optical axis 52 a is directed in the reverse direction of the vehicle 9 in the straight traveling direction (+ Y side in the Y-axis direction in plan view). Yes. The side cameras 53 are provided on the left and right door mirrors 93, respectively, and the optical axis 53a is directed to the outside along the left-right direction of the vehicle 9 (X-axis direction in plan view). Note that the mounting position of the front camera 51 and the back camera 52 is preferably approximately the center in the left and right, but may be slightly shifted in the left and right directions from the center in the left and right.

  As the lenses of these in-vehicle cameras 51, 52, and 53, fish-eye lenses are employed, and the in-vehicle cameras 51, 52, and 53 have an angle of view α of 180 degrees or more. For this reason, it is possible to shoot the entire periphery of the vehicle 9 by using the four in-vehicle cameras 51, 52, and 53.

<1-3. Image conversion processing>
Next, a method in which the composite image generation unit 31 of the image generation unit 3 generates a composite image that shows how the periphery of the vehicle 9 is viewed from an arbitrary virtual viewpoint based on a plurality of captured images obtained by the imaging unit 5. explain. When generating the composite image, the vehicle type data 4a stored in advance in the nonvolatile memory 40 is used. FIG. 3 is a diagram for explaining a method of generating a composite image.

  When the front camera 51, the back camera 52, and the side camera 53 of the photographing unit 5 are simultaneously photographed, four photographed images P1 to P4 that respectively indicate the front, rear, left side, and right side of the vehicle 9 are acquired. The That is, the four captured images P1 to P4 acquired by the imaging unit 5 include information indicating the entire periphery of the vehicle 9 at the time of shooting.

  Next, each pixel of the four captured images P1 to P4 is projected onto a three-dimensional curved surface SP in a virtual three-dimensional space. The three-dimensional curved surface SP has, for example, a substantially hemispherical shape (a bowl shape), and a center portion (a bottom portion of the bowl) is determined as a position where the vehicle 9 exists. The correspondence between the positions of the pixels included in the captured images P1 to P4 and the positions of the pixels of the solid curved surface SP is determined in advance. For this reason, the value of each pixel of the solid curved surface SP can be determined based on this correspondence and the value of each pixel included in the captured images P1 to P4.

  The correspondence between the positions of the pixels of the captured images P1 to P4 and the positions of the pixels of the three-dimensional curved surface SP is determined by the arrangement of the four in-vehicle cameras 51, 52, and 53 in the vehicle 9 (the distance between each other, the height above the ground, the optical axis). Angle). For this reason, the table data indicating this correspondence is included in the vehicle type data 4 a stored in the nonvolatile memory 40.

  Further, polygon data indicating the shape and size of the vehicle body included in the vehicle type data 4a is used, and a vehicle image that is a polygon model indicating the three-dimensional shape of the vehicle 9 is virtually provided. The configured vehicle image is arranged in a substantially hemispherical central portion determined as the position of the vehicle 9 in the three-dimensional space where the three-dimensional curved surface SP is set.

  Further, the virtual viewpoint VP is set by the control unit 1 for the three-dimensional space where the solid curved surface SP exists. The virtual viewpoint VP is defined by the viewpoint position and the visual field direction, and is set to an arbitrary visual field position corresponding to the periphery of the vehicle 9 in this three-dimensional space toward an arbitrary visual field direction.

  Then, according to the set virtual viewpoint VP, a necessary area on the three-dimensional curved surface SP is cut out as an image. The relationship between the virtual viewpoint VP and a necessary area in the three-dimensional curved surface SP is determined in advance, and is stored in advance in the nonvolatile memory 40 or the like as table data. On the other hand, rendering is performed on the vehicle image composed of polygons according to the set virtual viewpoint VP, and the resulting two-dimensional vehicle image is superimposed on the cut out image. Thereby, the composite image which shows a mode that the vehicle 9 and the periphery of the vehicle 9 were seen from arbitrary virtual viewpoints will be produced | generated.

  For example, when the virtual viewpoint VP1 in which the viewpoint position is a position just above the center of the position of the vehicle 9 and the visual field direction is a direction immediately below the vehicle 9 is set, the vehicle 9 ( Actually, a composite image CP1 showing the vehicle image) and the surroundings of the vehicle 9 is generated. Further, as shown in the figure, when the virtual viewpoint VP2 in which the viewpoint position is the left rear of the position of the vehicle 9 and the visual field direction is substantially in front of the vehicle 9 is set, the entire periphery from the left rear of the vehicle 9 is set. As seen, a composite image CP <b> 2 is generated that shows the vehicle 9 (actually a vehicle image) and the surroundings of the vehicle 9.

  In the case of actually generating a composite image, it is not necessary to determine the values of all the pixels of the three-dimensional curved surface SP, and only the values of the pixels in the area necessary corresponding to the set virtual viewpoint VP are photographed. By determining based on the images P1 to P4, the processing speed can be improved.

<1-4. Operation mode>
Next, the operation mode of the image display system 120 will be described. FIG. 4 is a diagram illustrating transition of operation modes of the image display system 120. The image display system 120 has four operation modes: a navigation mode M1, a front mode M2, a back mode M3, and a parallel parking mode M4. These operation modes can be switched by the control of the control unit 1 according to the user's operation and the running state of the vehicle 9.

  The navigation mode M <b> 1 is an operation mode in which a map image NP for navigation guidance is mainly displayed on the display 21 by the function of the navigation device 20. In the navigation mode M1, the functions of the image processing apparatus 100 are not used, and various displays are performed with the functions of the navigation apparatus 20 alone.

  The front mode M2, the back mode M3, and the parallel parking mode M4 use the functions of the image processing device 100 to display the captured image PP, the composite images CP and CQ0, the driving support image PH0, and the like on the display 21. This is an operation mode in which the situation around the vehicle 9 is shown to the user almost in real time.

  Further, the front mode M2 is an operation mode for displaying an image mainly showing the front and side of the vehicle 9 that is required when moving forward. For example, as shown in the figure, a composite image (overhead image) CP showing a state of the vehicle 9 and the surroundings of the vehicle 9 as viewed from the virtual viewpoint directly above the vehicle 9, that is, a virtually overhead image, and the front camera 51 The photographed image PP obtained by photographing with is displayed on the display 21 at the same time. The position of the virtual viewpoint of the composite image CP is not limited to the position directly above the vehicle 9, and may be switched to the rear side of the vehicle 9 or the like according to a user operation. Further, the front mode M2 is a mode in which a range of 2 to 3 m around the center of the host vehicle is displayed as an overhead image.

  Further, the back mode M3 is an operation mode for displaying an image mainly showing the rear of the vehicle 9 that is required when reversing. For example, as shown in the figure, a composite image (overhead image) CP showing a state of the vehicle 9 and the surroundings of the vehicle 9 viewed from a virtual viewpoint directly above the vehicle 9, and a captured image PP obtained by capturing with the back camera 52 Are simultaneously displayed on the display 21. The position of the virtual viewpoint of the composite image CP is not limited to the position immediately above the vehicle 9, and may be switched to the front side of the vehicle 9 or the like according to a user operation.

  Furthermore, the parallel parking mode M4 is a mode that supports the user's driving operation when the user parks the vehicle 9 in parallel. For example, as shown in the figure, the vehicle 9 viewed from the virtual viewpoint directly above the vehicle 9, an indicator MA that displays points on a line extending from the front end of the vehicle 9 in the left-right direction of the vehicle 9, An image showing the situation is displayed as a composite image overhead image) CQ0.

  Specifically, the parallel parking mode M4 is started when the user operates the parallel parking button 202 in the front mode M2. In the parallel parking mode M4, the composite image overhead view image CP showing the vehicle 9 and the surroundings of the vehicle 9 displayed in the front mode M2 is displayed on a line extending in the left-right direction of the vehicle 9 from the front end portion of the vehicle 9. An index MA for displaying the points is added and displayed as a composite image CQ.

  In the following, the index on the right side of the vehicle 9 in the figure is referred to as an index MR, the index on the left side of the vehicle 9 is referred to as an index ML, and these two indices are collectively referred to as an index MA.

  In addition, the driving support image PH0 that displays the information of the driving operation of the user in association with the display of the composite image CQ0 is also displayed on the display simultaneously with the composite image CQ0. That is, when the user operates the parallel parking button 202 in the front mode M2, the captured image PP displayed in the front mode M2 is switched to the driving support image PH0.

  The transition of each mode is performed in the front mode by a predetermined operation such as when the user operates the parking support button 201 displayed on the display 21 of the navigation device 20 or when the user operates the operation unit 22 in the navigation mode M1. Switch to M2. Further, in the front mode M2, the navigation mode M1 is switched by a predetermined operation such as an operation by the operation unit 22 of the user. The navigation mode M1 and the front mode M2 may be switched according to the traveling speed input from the vehicle speed sensor 82.

  In the navigation mode M1 or the front mode M2, when the shift position input from the shift sensor 81 is “R (reverse)”, the mode is switched to the back mode M3. That is, when the shift position is “R (reverse)”, the vehicle 9 is in a reverse state, and therefore, the back mode M3 that mainly indicates the rear of the vehicle 9 is switched. Further, in the case of the back mode M3, when the shift position is other than “R (reverse)”, the operation mode immediately before switching to the back mode M3 is returned. In the back mode M3, when the user operates the parking assistance button 201 displayed on the display 21, the front mode M2 is set.

  When the user operates the parallel parking button 202 displayed on the display 21 in the front mode M2, an image in the parallel parking mode M4 is output from the image processing apparatus 100 and displayed on the display 21. Further, when the user operates the return button 203 of the display 21 in the parallel parking mode M4, the mode is changed to the front mode M2.

<1-5 parallel parking mode>
Below, the detail of the image of parallel parking mode M4 is demonstrated. FIG. 5 is a diagram for explaining the parallel parking mode M4 according to the first embodiment. When the parallel parking button 202 is selected by the user in the front mode M2, an image in the parallel parking mode M4 (hereinafter also referred to as “image M4”) is output from the image processing apparatus 100 and displayed on the display 21. In addition, each image (M4, M41, M42, and M43) in the parallel parking mode M4 described below is a transition of each image in the parallel parking mode, and these images are changed according to the user's selection and vehicle operation. Is output from the image processing apparatus 100 and displayed on the display 21.

  In the image M4, the image of the vehicle 9 is displayed at the approximate center of the composite image CQ0. An indicator MA of points on a line extending in the left-right direction is displayed at the front end of the vehicle 9, and the other vehicle 19 located at the right front side of the vehicle 9 is displayed.

  The index MA includes an index MR on the right side of the vehicle 9 and an index ML on the left side of the vehicle 9, and moves in the same direction of the vehicle 9 as the vehicle 9 travels. That is, in the composite image CQ0, the index MA index MR) approaches the rear end portion of the other vehicle 19 as the vehicle 9 advances. Thus, the composite image CQ0 displays the vehicle 9 and the surrounding area of the vehicle 9 almost in real time.

  The image M4 displays a driving support image PH0 as a bitmap image. This driving support image PH0 displays information of driving operation necessary for the user of the vehicle 9 in association with the overhead image of the composite image CQ0.

  And when starting the measurement of the distance along the advancing direction of the vehicle 9 of the parking space of the vehicle 9 which the user confirmed visually, in order to set the position which starts the measurement, a user sets the point setting button of the image M4. 204 is selected. When the user selects the return button 203, the mode changes to the front mode M2, which is the previous mode.

  Also referred to as “image M41” below the image M41 in the parallel parking mode M4. ) Shows an image when the vehicle 9 is advanced in the direction of the other vehicle 19 and the user selects the point setting button 204. In the composite image CQ1 of the image M41, the index MR of the vehicle 9 is located at the tip of the other vehicle 19. When the user selects the point setting button 204 at such a position, the index MA (index MR and index ML) is set as the measurement start position DA.

  The measurement start position DA includes the measurement start position DR on the right side of the vehicle 9 and the measurement start position DL on the left side of the vehicle 9. The measurement start position DA includes the index MR and the measurement start position DL. The index ML is fixed as the measurement start positions DR and DL. Thereby, while a user can grasp | ascertain the relationship between the parking space which can be visually confirmed from a vehicle, and the displayed bird's-eye view image while moving a vehicle, the position which the user confirmed visually can be set reliably. Further, the user can clearly grasp the position of the vehicle 9 and the positions of the measurement start position DA and the measurement position BA.

  Further, the measurement position BA is set at the rear end of the vehicle 9 with the installation of the measurement start position by the user's selection of the point setting button 204. The measurement position BA includes a measurement position BR on the right side of the vehicle 9 and a measurement position BL on the left side of the vehicle 9. This measurement position BA moves in the same direction as the vehicle 9 as the vehicle 9 travels. In addition, in the image M41, the driving support image PH1 is displayed in association with the overhead image of the composite image CQ1. Note that when the user operates the return button 203, the setting of the measurement start position is canceled and the state transits to the parallel parking mode M4.

  The image M42 is separated from the measurement start position DA after the measurement position BA accompanying the setting of the measurement start position by the user operation shown in the image M41 in the composite image CQ2 comes into contact with the measurement start position DA as the vehicle 9 travels. Shows the state. And when measurement position BA leaves | separated from measurement start position DA, the measurement of the distance along the advancing direction of the vehicle 9 of a parking space is started.

  That is, the measurement position BR on the line extending in the right direction of the rear end portion of the vehicle 9 comes into contact with the measurement start position DR fixed to the position extending in the direction of the front end portion of the other vehicle 19 which is the right direction of the vehicle 9. , Away from the measurement start position DR. At the same time, after the measurement position BL on the line extending in the left direction at the rear end portion of the vehicle 9 comes into contact with the measurement start position DL fixed at the position extending in the left direction at the front end portion of the other vehicle 19, the measurement start position DL starts from the measurement start position DL. is seperated. And immediately after each measurement position contacts and leaves | separates a measurement start position, derivation | leading-out of distance is started.

  As shown in the figure, the image display when the distance is derived is displayed by a distance line LR connecting the measurement start position DR and the measurement position BR, and a distance line LL connecting the measurement start position DL and the measurement position BL. Is done. As a result, the user can know the timing for deriving the distance of the parking space, and it is possible to derive the accurate distance by eliminating an unnecessary operation of the steering after the start of the distance deriving. Further, the user can clearly grasp the positional relationship between the position of the vehicle 9 and the parking space.

  The image M41 displays the driving support image PH1 in association with the overhead image of the composite image CQ2. Note that when the user operates the return button 203, the setting of the measurement start position is canceled and the mode is changed to the parallel parking mode MA.

  The image M43 is the composite image CQ3, and the parking of the vehicle 9 in which the distance between the position of the measurement position BA at the rear end of the vehicle 9 and the position of the measurement start position DA fixed to the position of the front end of the other vehicle 19 is predetermined. This indicates that the required distance is exceeded. As shown in the figure, in order to notify the user that the distance required for parking the vehicle 9 has been reached, the display color of the distance lines LR and LL is greater than the required distance when the distance is being measured. Change with the case. Such control is performed by the notification control unit 14. Thereby, the positional relationship between the position of the vehicle 9 and the parking space can be clarified, and the distance of the parking space can be accurately notified to the user. The user may be notified that the measured distance is greater than or equal to the required distance by image display, audio output, or the like.

  In addition, when the distance along the traveling direction of the vehicle 9 in the parking space becomes the distance necessary for parking the vehicle 9, the user operates the vehicle position determination button 205 displayed in the image M43. The parking space selection and the parking operation are started. Thereby, the user can start a parking operation immediately from the position of the vehicle 9 that has become a distance necessary for parking.

  In addition, in the image M43, the driving support image PH3 is displayed in association with the overhead image of the composite image CQ3. As described above, the image processing apparatus 100 outputs a driving support image in association with each composite image and displays the driving support image on the display 21, thereby performing a driving operation performed by the user together with the composite image when deriving a distance necessary for parking. Can be confirmed. In addition, in the images M4, M41, M42, and M43, driving operation points are displayed in characters in accordance with driving support images displayed in association with the respective composite images, and necessary driving operations are displayed. Note that when the user operates the return button 203, the setting of the measurement start position is canceled and the state transits to the parallel parking mode M4.

<2. Processing>
Next, processing of the image processing apparatus 100 will be described. FIG. 6 is a process flowchart of the image processing apparatus 100. When the image processing apparatus 100 receives an operation signal of the parking assistance button 201 from the user (Yes in step S101), the image processing apparatus 100 outputs an image of the front mode M2 and proceeds to the process of step S103. If the image processing apparatus 100 has not received an operation signal for the parking assistance button 201 from the user (No in step S101), the process ends.

  In step S103, when the image processing apparatus 100 receives an operation signal for the parallel parking button 202 from the user (step S103 is Yes), the image of the parallel parking mode M4 is output (step S104), and the process proceeds to step S105. move on. In addition, when the image processing apparatus 100 has not received the operation signal of the parallel parking button 202 (No in step S103), the image output in the front mode M2 is continued.

  In step S105, when the image processing apparatus 100 has received a setting signal for the measurement start position DA using the index MA by the user's operation (Yes in step S105), the point of the index MA is set as the measurement start position DA. (Step S106), the process proceeds to Step S107.

  When the image processing apparatus 100 has not received the setting signal for the measurement start position DA (No in step S105), the image in the parallel parking mode M4 is continuously displayed. In addition, you may make it switch to other images, such as front mode M2 and parallel parking mode M4, according to a user's operation.

  In step S107, the measurement position BA is output based on the specific part of the vehicle 9 when the measurement start position is set by the user's selection (step S107). The measurement position BA is an index for displaying a point on a line extending from the rear end of the vehicle 9 in the left-right direction of the vehicle 9, for example.

  The image processing apparatus 100 then measures the measurement start position DA after the measurement position BA displayed at the rear end of the vehicle 9 comes into contact with the measurement start position DA fixed at the point selected by the user as the vehicle 9 travels. When it is detected that the user has left (Step S108 is Yes), measurement of the distance between the measurement start position DA and the measurement position BA is started (Step S109).

  When the measurement distance becomes equal to or greater than a predetermined distance (for example, 1.5 times the total length of the vehicle 9) due to further movement of the vehicle 9 (Yes in step S110), the image processing apparatus 100 determines the measurement distance. Is notified to the user by image or sound (step S111), and the process proceeds to step S112.

  In step S108, when it is not detected that the measurement position BA has moved away from the measurement start position DA (step S108), a near real-time image is output in the parallel parking mode M4 without starting the distance measurement. To do.

  Furthermore, in step S110, when the measurement distance is not equal to or greater than the predetermined distance, the distance measurement is continuously performed. In addition, you may make it switch to other images, such as front mode M2 and parallel parking mode M4, according to a user's operation.

  In the process of step S112, when the user confirms that the distance required for parking the vehicle 9 has been reached, and the image processing apparatus 100 receives a vehicle position confirmation signal when the user operates the vehicle position confirmation button 205 (Step S113 is Yes) outputs the parking area setting image which sets the area | region where the measurement of distance by the user's operation was completed as a parking area of the vehicle 9. FIG.

  The parking area is set by a user operating a substantially rectangular area having one side that is about 1.5 times the total length of the vehicle 9 and the other side that is about 1.2 times the vehicle width of the vehicle 9. To set the parking space where the vehicle 9 is parked. In step 112, when the image processing apparatus 100 has not received the vehicle position determination signal (No in step S112), the image is displayed as it is. In addition, you may make it switch to other images, such as front mode M2 and parallel parking mode M4, according to a user's operation.

  When the image processing apparatus 100 receives a signal indicating that the user has set the parking area (Yes in step S114), the image processing apparatus operates such that the user parks the vehicle 9 in the parking space using an image or sound. A parking assistance image for assisting the operation is output (step S115), and the process proceeds to step S116. As an example of this parking assistance image, a position and angle for operating a guide line and steering when the vehicle is moved backward are indicated.

  In step S114, when the image processing apparatus 100 has not received a signal indicating that the parking position has been determined by the user's operation (No in step S114), the image is displayed as it is. In addition, you may make it switch to other images, such as front mode M2 and parallel parking mode M4, according to a user's operation.

  In step S116, when the entire vehicle 9 moves to the parking area set in step S114 due to the user's parking operation, the image processing apparatus 100 receives a parking completion signal based on one information of the vehicle 9 from the navigation device 20 or the like. When the image is received (step S116), the image processing apparatus 100 notifies the user that the parking has been completed by an image or sound (step S117).

  In addition, when the image processing apparatus 100 has not received the parking completion signal (step S116 is No), the output of the parking assistance image is continued. In addition, you may make it switch to other images, such as front mode M2 and parallel parking mode M4, according to a user's operation.

<Second Embodiment>
Next, a second embodiment will be described. In the parallel parking mode M5 of the second embodiment, when the index MA is set as the measurement start position DA by the user's operation, the measurement is located at the rear end portion of the vehicle 9 in the first embodiment. A position BA is provided at the tip of the vehicle 9. The configuration and processing of the image display system 120 of the second embodiment are substantially the same as those of the first embodiment, but are partly different. For this reason, the following description will focus on the differences.

  FIG. 7 is a diagram for explaining the parallel parking mode M4a according to the second embodiment. Hereinafter, the parallel parking modes M4a, M41a, M42a, and M43a are also referred to as an image M4a, an image 41a, an image 42a, and an image 43a.

  The image processing apparatus 100 outputs the image M4a when the signal that the parallel parking button 202 in the front mode M2 is operated by the user is received.

  The image M4a displays the index MA (index MR and ML) in the composite image CQ10 at the front end of the vehicle 9, and the index MA progresses in the direction of the other vehicle 19 as the vehicle 9 advances.

  In the image M41a, the index MA is located at the tip of the other vehicle 19 in the composite image CQ11, and when the user operates the point setting button 204, the index MA is fixed at that point as the measurement start position DA.

  Further, the measurement position BAa is set at the front end of the vehicle 9 in accordance with the setting of the measurement start position by the user's selection of the spot setting button 204. The measurement position BAa includes a measurement position on the right side of the vehicle 9 as a measurement position BRa and a measurement position on the left side of the vehicle 9 as a measurement position BLa. This measurement position BAa moves in the same direction as the vehicle 9 travels.

  The image M42a shows a state in which the measurement position BAa accompanying the setting of the measurement start position DA by the user's operation in the composite image CQ12 is separated from the measurement start position DA after contacting the measurement start position DA as the vehicle 9 advances. ing. And the measurement of the distance along the advancing direction of the vehicle 9 of a parking space is started because measurement position BAa left | separated from measurement start position DA.

  That is, the measurement position BRa on the line extending in the right direction of the front end portion of the vehicle 9 comes into contact with the measurement start position DR fixed to the position extending in the direction of the front end portion of the other vehicle 19 which is the right direction of the vehicle 9. It is away from the measurement start position DR. At the same time, after the measurement position BLa on the line extending in the left direction at the rear end portion of the vehicle 9 comes into contact with the measurement start position DL fixed at the position extending in the left direction at the front end portion of the other vehicle 19, the measurement start position DL starts from the measurement start position DL. is seperated. And immediately after each measurement position contacts and leaves | separates a measurement start position, derivation | leading-out of distance is started.

  As shown in the figure, the image display when the distance is derived is displayed by a distance line LRa connecting the measurement start position DR and the measurement position BR, and a distance line LLa connecting the measurement start position DL and the measurement position BL. Is done. Thereby, the user can visually confirm the state of the parking space as the vehicle 9 moves while deriving the distance.

In the image 43a, as shown in the figure, in order to notify the user that the distance required for parking the vehicle 9 has been reached, the display color of the distance lines LRa and LLa is more than the required distance when the distance is being measured. It has changed in the case of becoming.
<Modification>
Hereinafter, modified examples will be described. In addition, all the forms including the above-described embodiment and the forms described below can be appropriately combined.

  In said embodiment, although the case where the user measured the distance of the space which parks the vehicle 9 was described, the distance of a user's arbitrary area can also be measured besides a parking space. In this case, the position setting control unit 12 sets the first reference position at the position that is the starting point of the user's arbitrary section in the peripheral area of the vehicle 9 during the display displayed on the display 21 of the navigation device 20 by the user's operation. To do.

  Next, with the setting of the first reference position, the position setting control unit 12 sets the second reference position based on the specific part of the vehicle 9. Then, the distance derivation control unit 13 derives a first distance that is a distance between the first reference position and the second reference position.

  Next, the notification control unit 14 notifies the information related to the first distance that changes due to the movement of the vehicle 9 by an image or sound. For example, the notification to the user is a display of the distance on the display 21 by the movement of the vehicle 9. Thereby, when the user of the vehicle 9 wants to measure the distance from an arbitrary position, it can measure accurately.

  In the above embodiment, the processing in the case of parallel parking has been described, but the contents of these embodiments can also be applied in the case of parallel parking.

  In the above embodiment, the index MA, which is an index on the line extending in the left-right direction from the front end portion of the vehicle 9, has been described with respect to the index MR on the right side of the vehicle and the index ML on the left side of the vehicle. When it is determined whether the parking direction of the vehicle 9 is right or left, the index is displayed in the direction selected by the user. That is, the index MR on the right side of the vehicle is displayed on the display 21 when the user's selection is on the right side, and the index ML is displayed on the display 21 when the user's selection is on the left side.

  Furthermore, the measurement start position is set in the same direction as the index displayed when the user selects the direction in which the vehicle 9 is parked, and the measurement position associated with the setting of the measurement start position is also displayed in the same direction. That is, when the right side of the vehicle 9 is selected, the index MR is displayed, and when the user sets the measurement start position of the index MR, the measurement start position DR is displayed on the right side of the vehicle 9 that is the same as the index MR. Then, along with the measurement start position setting, the measurement position BR (or BRa) is displayed on the right side of the vehicle 9 to measure the distance.

  Furthermore, when the left side of the vehicle 9 is selected, the index ML is displayed, and when the user sets the measurement start position of the index ML, the measurement start position DL is displayed on the left side of the vehicle 9 that is the same as the index ML. Then, along with the measurement start position setting, the measurement position BLa is displayed on the left side of the vehicle 9 to measure the distance.

  In addition, the position of the index MA can be moved to a position other than the front end portion of the vehicle 9 by the user's operation to set the measurement start position DA.

  Furthermore, the bird's-eye view image of the above embodiment may be a bird's-eye view image generated by one camera other than a composite image generated by images taken by a plurality of cameras.

DESCRIPTION OF SYMBOLS 1 ... Control part 3 ... Image generation part 5 ... Imaging | photography part 20 ... Navigation apparatus 40 ... Non-volatile memory 41 ... Signal input part 42 ... Navigation communication part 81 ... Shift sensor 82 ... Vehicle speed sensor 83 ... Direction indicator

Claims (6)

An image display system mounted on a vehicle,
Generating means for generating a bird's-eye view image of the vehicle and a surrounding area of the vehicle from a captured image obtained by a camera disposed in the vehicle;
Display means for displaying the overhead image;
Setting means for setting a first reference position specified in accordance with a user operation and a second reference position based on a specific part of the vehicle during display of the overhead image;
Deriving means for deriving a first distance that is a distance between the first reference position and the second reference position;
With
The display means connects the first reference position and the second reference position, displays a linear line image whose length changes according to a change in the first distance accompanying the movement of the vehicle, and the vehicle Changing the display mode of the line image when the first distance is equal to or greater than the second distance required for parking the vehicle due to movement of
An image display system characterized by the above. The image display system according to claim 1,
When the first reference position and the second reference position become the same position after the first reference position and the second reference position become the same position, the derivation means is configured to change the first reference position and the second reference position to a different position. Starting derivation of the first distance ;
An image display system characterized by the above. The image display system according to claim 1 or 2,
The display means, when setting the first reference position, displaying a point on a line extending in the left-right direction of the vehicle from the front end portion of the vehicle as an index ;
An image display system characterized by the above. The image display system according to any one of claims 1 to 3,
The specific part of the vehicle is one of a rear end portion of the vehicle and a front end portion of the vehicle ;
An image display system characterized by the above. An image processing device for generating an image to be displayed on a display device mounted on a vehicle,
From the captured image obtained by the camera arranged in the vehicle, the vehicle and the surrounding area
Generating means for generating a bird's-eye view image overlooking the area;
Output means for outputting and displaying the overhead image on the display device;
Setting means for setting a first reference position specified in accordance with a user operation and a second reference position based on a specific part of the vehicle during display of the overhead image;
Deriving means for deriving a first distance that is a distance between the first reference position and the second reference position;
With
The output means connects the first reference position and the second reference position, outputs a linear line image whose length changes according to a change in the first distance accompanying the movement of the vehicle, and the vehicle When the first distance is equal to or greater than the second distance necessary for parking the vehicle due to movement of the vehicle, the line image in which the display mode is changed is output and displayed .
An image processing apparatus . An image display method of an image processing device for generating an image to be displayed on a display device mounted on a vehicle,
(A) from the captured image obtained by the camera placed on the vehicle, and generating an overhead image obtained by the overhead view of the vehicle and the peripheral area of the vehicle,
A step of displaying (b) outputting the overhead image on the display device,
(C) a step of setting a first reference position specified in accordance with a user operation and a second reference position based on the specific part of the vehicle during display of the overhead image;
And (d) deriving a first distance is a distance between the second reference position and the first reference position,
With
The step (b) connects the first reference position and the second reference position, and outputs a linear line image whose length changes according to a change in the first distance accompanying the movement of the vehicle , When the first distance is equal to or greater than a second distance necessary for parking the vehicle due to the movement of the vehicle, the line image in which the display mode is changed is output and displayed ;
An image display method characterized by the above.

Images