JP2020043418A - Periphery monitoring device - Google Patents

Abstract

To provide a periphery monitoring device capable of displaying images with little distortion so that a driver can always check a target parking area and the position of own vehicle when parking own vehicle in the target parking area.SOLUTION: The periphery monitoring device includes: an image acquisition part that acquires pick-up image data of the periphery of a vehicle; a target acquisition part that acquires a target parking area for guiding the vehicle; and an image generation part that is configured so as to, when guiding the vehicle to the target parking area, generate a virtual viewpoint image in which at least part of the parking area and at least part of the vehicle based on pick-up image data are continuously being included in the display area while continuously moving the position of the virtual viewpoint which is set above the vehicle toward the top according to the moving condition of the vehicle; and an output part that outputs the virtual viewpoint image on the display device.SELECTED DRAWING: Figure 4

Description

  Embodiments of the present invention relate to a periphery monitoring device.

  2. Description of the Related Art Conventionally, a technique of capturing a situation around a vehicle in different directions by a plurality of image capturing units (cameras) provided in a vehicle and performing image processing (for example, viewpoint conversion processing) on the captured images to generate a composite image. It has been known. For example, by changing the viewpoint or the display area according to the running state of the vehicle with respect to the synthesized image generated in this manner, the driver can easily monitor the vicinity of the vehicle. An image display device for providing an image has been proposed.

Japanese Patent No. 3300334 Japanese Patent No. 53212267

  As a scene where the driver wants to check the surroundings by visually recognizing the display device, for example, there is a case where the driver parks the vehicle in a target parking area (parking frame). In such a case, it is desirable that the image to be displayed is, for example, an image in which the target parking area or the position of the own vehicle can be easily confirmed, or an image in which the distortion or the like of the surrounding image is easy to visually recognize. . In this way, if an image with less distortion can be provided so that the target parking area and the position of the own vehicle can always be confirmed, it is possible to improve the accuracy of monitoring the periphery using the image displayed on the display device and to perform monitoring work without stress. It can be done and is meaningful.

  A surroundings monitoring device according to an embodiment of the present invention includes, for example, an image acquisition unit that acquires captured image data of the periphery of a vehicle, a target acquisition unit that acquires a target parking area for guiding the vehicle, When guiding to the target parking area, the target based on the captured image data is moved while the position of the virtual viewpoint set above the vehicle is continuously moved directly upward according to the moving state of the vehicle. An image generation unit that generates a virtual viewpoint image such that both at least a part of the parking area and at least a part of the vehicle continue to be included in the display area, and an output unit that outputs the virtual viewpoint image to a display device, including. According to this configuration, for example, even when the position of the virtual viewpoint changes, it is easy to always recognize the relative positional relationship between the target parking area and the vehicle that the driver wants to pay attention to while driving with parking assistance. Then, when the vehicle finally moves to the target parking area, it becomes possible to display an image that looks down the vehicle from immediately above. Further, since the virtual viewpoint is always directed to the target parking area, an image with less distortion can be provided. As a result, when the target parking area is far away, it is easy to check the surroundings of the target parking area, and when the target parking area approaches, it is easy to check the surroundings of the vehicle. This contributes to improvement and makes it easier to perform low-stress monitoring work.

  The image generation unit of the periphery monitoring device according to the embodiment of the present invention includes, for example, a length of a first virtual line of sight connecting the vehicle center position set immediately above the vehicle and the target parking area, The virtual viewpoint image is generated while moving the position of the virtual viewpoint according to the moving state of the vehicle while maintaining a constant ratio of a length of a second virtual line of sight line connecting the position and the position of the virtual viewpoint. It may be. According to this configuration, for example, as the vehicle approaches the target parking area, a so-called “zoom-in” display in which the vehicle is magnified and displayed in an excellent manner can be easily performed. As a result, it is easy to confirm the surroundings of the vehicle, and the accuracy of surrounding monitoring can be further improved.

  The image generation unit of the periphery monitoring device according to the embodiment of the present invention includes, for example, a virtual curved line that covers the vehicle, a virtual line of sight passing through the target parking area and a vehicle center position set immediately above the vehicle. The virtual viewpoint image may be generated while moving the position of the virtual viewpoint set at a position that intersects the target parking area on the opposite side across the vehicle in accordance with the moving state of the vehicle. According to this configuration, for example, even when the position of the virtual viewpoint moves, the vehicle (own vehicle) is continuously displayed in substantially the same size. As a result, the display content of the virtual viewpoint image is smoothly changed and unnatural display is reduced, an image that is easy to recognize can be provided, and monitoring work with lower stress can be performed more easily. .

  The image generation unit of the periphery monitoring device according to the embodiment of the present invention, for example, the position of the gazing point when facing the target parking area from the position of the virtual viewpoint, with respect to the area center position of the target parking area The position may be moved by a predetermined amount in a direction away from the vehicle. According to this configuration, for example, even when the vehicle overruns the target parking area, the position of the virtual viewpoint may be in the opposite direction from the traveling direction of the vehicle, that is, the virtual viewpoint image is rotated by 180 °. It is possible to reduce the possibility that the image will be displayed, and it is possible to suppress the display of an uncomfortable image at the end of parking.

  The image generation unit of the periphery monitoring device according to the embodiment of the present invention may change a viewing angle of the virtual viewpoint image according to, for example, a moving state of the position of the virtual viewpoint. According to this configuration, for example, a display when emphasis is placed on monitoring of a vehicle at a short distance (around the vehicle) or a display when emphasis is placed on monitoring the entire vehicle including a long distance with respect to the vehicle And the like can be switched, and monitoring variations can be improved.

  The image generation unit of the periphery monitoring device according to the embodiment of the present invention includes, for example, at a position corresponding to the target parking area in the virtual viewpoint image, a parking frame image indicating the target parking area and the target parking area. An index indicating at least one of a virtual vehicle indicating a state in which the vehicle has moved may be superimposed and displayed. According to this configuration, for example, it is possible to easily confirm the position of the target parking area, the degree of approach, and the like.

  The image generation unit of the periphery monitoring device according to the embodiment of the present invention may display, for example, the vehicle and the index in the virtual viewpoint image in different display modes. According to this configuration, for example, it is possible to make it easier to confirm the position of the target parking area, confirm the degree of approach, and the like.

FIG. 1 is an exemplary perspective view showing a state in which a part of a vehicle compartment of a vehicle equipped with the periphery monitoring device of the embodiment is seen through. FIG. 2 is an exemplary plan view of a vehicle equipped with the periphery monitoring device of the embodiment. FIG. 3 is an exemplary block diagram of a configuration of a peripheral monitoring system including the peripheral monitoring device of the embodiment. FIG. 4 is a block diagram exemplarily showing a configuration centered on a peripheral monitoring unit realized by a CPU of the peripheral monitoring system. FIG. 5 is an exemplary schematic diagram illustrating a case where the position of the virtual viewpoint by the periphery monitoring device of the embodiment is moved while maintaining the ratio between the length of the first virtual line of sight and the length of the second virtual line of sight constant. FIG. FIG. 6 is an exemplary schematic diagram illustrating a case where the position of the virtual viewpoint by the periphery monitoring device of the embodiment is moved on a virtual curved surface that covers the vehicle. FIG. 7 is an exemplary schematic diagram illustrating a case where the position of the gazing point in the periphery monitoring device according to the embodiment is moved by a predetermined amount in a direction away from the vehicle. FIG. 8 is an exemplary schematic diagram of a virtual viewpoint image displayed by the periphery monitoring device according to the embodiment, and is a diagram illustrating a case where the position of the virtual viewpoint is a position at the beginning of movement. FIG. 9 is an exemplary schematic diagram of a virtual viewpoint image displayed by the periphery monitoring device according to the embodiment, and is a diagram illustrating a case where the position of the virtual viewpoint is the position of a moving midfield. FIG. 10 is an exemplary schematic diagram of a virtual viewpoint image displayed by the periphery monitoring device according to the embodiment, and is a diagram illustrating a case where the position of the virtual viewpoint is a position at the end of movement. FIG. 11 is a flowchart exemplarily illustrating a flow of a virtual viewpoint image display process performed by the periphery monitoring device according to the embodiment.

  Hereinafter, exemplary embodiments of the present invention will be disclosed. The configurations of the embodiments described below, and the actions, results, and effects provided by the configurations are examples. The present invention can be realized by configurations other than those disclosed in the following embodiments, and can obtain at least one of various effects based on the basic configuration and derivative effects. .

  As illustrated in FIG. 1, in the present embodiment, a vehicle 1 equipped with a peripheral monitoring device (peripheral monitoring system) is, for example, an automobile that uses an internal combustion engine (not shown) as a driving source, that is, an internal combustion engine automobile. Alternatively, the vehicle may be a vehicle using an electric motor (not shown) as a driving source, that is, an electric vehicle, a fuel cell vehicle, or the like. Further, a hybrid vehicle using both of them as drive sources may be used, or a vehicle provided with another drive source may be used. Further, the vehicle 1 can be mounted with various transmissions, and can be mounted with various devices required for driving the internal combustion engine and the electric motor, such as systems and components. In addition, the driving method of the vehicle 1 may be a four-wheel drive vehicle that transmits driving force to all four wheels 3 and uses all four wheels as drive wheels, or may be a front-wheel drive system or a rear-wheel drive system. Good. The type, number, layout, and the like of the devices related to the driving of the wheels 3 can be variously set.

  The vehicle body 2 forms a vehicle compartment 2a in which an occupant (not shown) rides. A steering unit 4, an acceleration operation unit 5, a braking operation unit 6, a shift operation unit 7, and the like are provided in the vehicle interior 2a in a state of facing a driver's seat 2b as an occupant. The steering unit 4 is, for example, a steering wheel protruding from the dashboard 24, the acceleration operation unit 5 is, for example, an accelerator pedal positioned below the driver's feet, and the braking operation unit 6 is, for example, a driver's The shift pedal 7 is, for example, a shift lever projecting from the center console. The steering unit 4, the acceleration operation unit 5, the braking operation unit 6, the shift operation unit 7, and the like are not limited to these.

  Further, a display device 8 as a display output unit and an audio output device 9 as an audio output unit are provided in the passenger compartment 2a. The display device 8 is, for example, an LCD (liquid crystal display), an OELD (organic electroluminescent display), or the like. The audio output device 9 is, for example, a speaker. The display device 8 is covered with, for example, a transparent operation input unit 10 such as a touch panel. The occupant can visually recognize an image displayed on the display screen of the display device 8 via the operation input unit 10. Further, the occupant can execute the operation input by operating the operation input unit 10 by touching, pushing, or moving the operation input unit 10 with a finger or the like at a position corresponding to the image displayed on the display screen of the display device 8. . The display device 8, the audio output device 9, the operation input unit 10, and the like are provided, for example, on the monitor device 11 located at the center of the dashboard 24 in the vehicle width direction, that is, in the left-right direction. The monitor device 11 can include an operation input unit (not shown) such as a switch, a dial, a joystick, and a push button. An audio output device (not shown) can be provided at another position in the passenger compartment 2a different from the monitor device 11, and audio is output from the audio output device 9 of the monitor device 11 and another audio output device. be able to. Note that the monitor device 11 can also be used, for example, as a navigation system or an audio system.

  Further, a display device 12 different from the display device 8 is provided in the passenger compartment 2a, as illustrated in FIG. The display device 12 is provided, for example, on an instrument panel section 25 (see FIG. 1) of the dashboard 24, and can be positioned substantially at the center of the instrument panel section 25 between the speed display section and the rotation speed display section. The size of the screen of the display device 12 may be smaller than the size of the screen of the display device 8. The display device 12 displays an image indicating a control state by an indicator, a mark, character information, or the like as auxiliary information when various functions such as a periphery monitoring of the vehicle 1 and a parking assist function are operating. Can be done. The amount of information displayed on the display device 12 may be smaller than the amount of information displayed on the display device 8. The display device 12 is, for example, an LCD, an OELD, or the like. The information displayed on the display device 12 may be displayed on the display device 8.

  Further, as exemplified in FIGS. 1 and 2, the vehicle 1 is, for example, a four-wheeled vehicle, and has two left and right front wheels 3F and two left and right rear wheels 3R. Each of these four wheels 3 can be configured to be steerable. As illustrated in FIG. 3, the vehicle 1 has a steering system 13 that steers at least two wheels 3. The steering system 13 has an actuator 13a and a torque sensor 13b. The steering system 13 is electrically controlled by an ECU 14 (electronic control unit) or the like to operate the actuator 13a. The steering system 13 is, for example, an electric power steering system, an SBW (steer by wire) system, or the like. Further, the torque sensor 13b detects, for example, a torque given to the steering unit 4 by the driver.

  Further, as illustrated in FIG. 2, the vehicle body 2 is provided with, for example, four imaging units 15 a to 15 d as the plurality of imaging units 15. The imaging unit 15 is, for example, a digital camera having a built-in imaging device such as a charge coupled device (CCD) or a CMOS image sensor (CIS). The imaging unit 15 can output moving image data (captured image data) at a predetermined frame rate. Each of the imaging units 15 has a wide-angle lens or a fisheye lens, and can capture, for example, a range of 140 ° to 220 ° in the horizontal direction. The optical axis of the imaging unit 15 may be set obliquely downward. Therefore, the imaging unit 15 may be a non-solid object such as a road surface on which the vehicle 1 can move, a stop line, a parking frame line, a lane line attached to the road surface, and an object existing around the vehicle 1 (for example, a wall, a tree, or the like). (Three-dimensional objects such as humans, bicycles, and vehicles, approaching objects, and the like may be referred to as “obstacles.”) It is possible to sequentially capture images with attention to them and output them as captured image data.

  The imaging unit 15a is located, for example, at the rear end 2e of the vehicle body 2, and is provided on a wall below the rear window of the door 2h of the rear hatch. The imaging unit 15b is located, for example, at the right end 2f of the vehicle body 2 and is provided on the right door mirror 2g. The imaging unit 15c is located, for example, at the front side of the vehicle body 2, that is, at the front end 2c in the front-rear direction of the vehicle, and is provided on a front bumper, a front grill, and the like. The imaging unit 15d is located, for example, on the left side of the vehicle body 2, that is, on the left end 2d in the vehicle width direction, and is provided on the left side door mirror 2g. The ECU 14 performs arithmetic processing and image processing based on the captured image data obtained by the plurality of imaging units 15 to generate an image with a wider viewing angle, and generates a virtual viewpoint image of the vehicle 1 viewed from above. Or It should be noted that the captured image data (image) captured by each imaging unit 15 is provided with an overlapping area that overlaps each other, so that a missing area does not occur when the images are joined.

  As illustrated in FIGS. 1 and 2, the vehicle body 2 is provided with a plurality of distance measuring units 16 and 17, for example, four distance measuring units 16 a to 16 d and eight distance measuring units 17 a to 17 h. ing. The distance measuring units 16 and 17 are, for example, sonars that emit ultrasonic waves and capture reflected waves. The sonar can also be called a sonar sensor, an ultrasonic detector, or an ultrasonic sonar. In the present embodiment, the distance measuring units 16 and 17 are provided at lower positions in the vehicle height direction of the vehicle 1, for example, at front and rear bumpers. Then, the ECU 14 can measure the presence or absence of an object such as an obstacle located around the vehicle 1 and the distance to the object based on the detection results of the distance measurement units 16 and 17. That is, the distance measurement units 16 and 17 are examples of a detection unit that detects an object. The distance measuring unit 17 can be used, for example, to detect a relatively short distance object, and the distance measuring unit 16 can be used, for example, to detect a relatively long distance object farther than the distance measuring unit 17. The distance measuring unit 17 can be used, for example, for detecting objects in front and behind the vehicle 1, and the distance measuring unit 16 can be used for detecting an object on the side of the vehicle 1.

  As illustrated in FIG. 3, in the peripheral monitoring system 100 (peripheral monitoring device), in addition to the ECU 14, the monitoring device 11, the steering system 13, the distance measuring units 16 and 17, a brake system 18, a steering angle sensor 19, an accelerator sensor 20, a shift sensor 21, a wheel speed sensor 22, a drive system 23 and the like are electrically connected via an in-vehicle network 26 as an electric communication line. The in-vehicle network 26 is configured as, for example, a CAN (controller area network). The ECU 14 can control the steering system 13, the brake system 18, the drive system 23, and the like by sending a control signal through the in-vehicle network 26. Further, the ECU 14 detects detection results of the torque sensor 13b, the brake sensor 18b, the steering angle sensor 19, the distance measuring units 16, 17, the accelerator sensor 20, the shift sensor 21, the wheel speed sensor 22, and the like via the in-vehicle network 26, An operation signal or the like of the operation input unit 10 or the like can be received.

  The ECU 14 includes, for example, a CPU 14a (central processing unit), a ROM 14b (read only memory), a RAM 14c (random access memory), a display control unit 14d, a voice control unit 14e, an SSD 14f (solid state drive, flash memory), and the like. ing. The CPU 14a can execute, for example, arithmetic processing and control of image processing related to images displayed on the display devices 8 and 12. Further, the CPU 14a performs a distortion correction process of performing arithmetic processing and image processing on the captured image data (curved image data) of the wide-angle image obtained by the imaging unit 15 to correct distortion, or the imaging unit 15 Based on the captured image data, a virtual viewpoint image (peripheral image) that displays the own vehicle icon indicating the vehicle 1 at, for example, the center position can be created and displayed on the display device 8. The CPU 14a can generate the virtual viewpoint image to be generated while moving the position of the virtual viewpoint toward right above the vehicle 1. Further, the CPU 14a can execute parking assistance (driving assistance) by identifying lane markings and the like shown on a road surface around the vehicle 1 from the captured image data provided by the imaging unit 15. For example, the CPU 14a detects (extracts) a section (for example, a parking section line) and sets a target area (for example, a target parking area) in which the vehicle 1 can move, or guides the vehicle 1 to the target area. For example, a guidance control (support control) for guiding the vehicle 1 to a target area by fully automatic, semi-automatic, or manual (for example, an operation guide by voice) can be executed.

  The CPU 14a reads a program installed and stored in a non-volatile storage device such as the ROM 14b, and can execute arithmetic processing according to the program. The RAM 14c temporarily stores various data used in the calculation by the CPU 14a. In addition, the display control unit 14d mainly performs, for example, synthesis of image data displayed on the display device 8 in the arithmetic processing performed by the ECU 14. In addition, the voice control unit 14 e mainly performs processing of voice data output from the voice output device 9 in the arithmetic processing in the ECU 14. The SSD 14f is a rewritable nonvolatile storage unit, and can store data even when the power of the ECU 14 is turned off. Note that the CPU 14a, the ROM 14b, the RAM 14c, and the like can be integrated in the same package. Further, the ECU 14 may be configured to use another logical operation processor, a logical circuit, or the like, such as a DSP (digital signal processor), instead of the CPU 14a. Further, a hard disk drive (HDD) may be provided instead of the SSD 14f, and the SSD 14f and the HDD may be provided separately from the ECU 14.

  The brake system 18 includes, for example, an anti-lock brake system (ABS) that suppresses lock of the brake, an electronic stability control (ESC) that suppresses a side slip of the vehicle 1 at the time of cornering, and increases the braking force ( Electric brake system for executing brake assist), BBW (brake by wire) and the like. The brake system 18 applies a braking force to the wheels 3 and thus to the vehicle 1 via the actuator 18a. In addition, the brake system 18 can execute various controls by detecting a lock of a brake, an idling of the wheel 3, a sign of a skid, or the like based on a rotation difference between the left and right wheels 3 and the like. The brake sensor 18b is, for example, a sensor that detects the position of the movable part of the braking operation unit 6. The brake sensor 18b can detect the position of a brake pedal as a movable part. The brake sensor 18b includes a displacement sensor. The CPU 14a can calculate the braking distance from the magnitude of the braking force calculated based on the detection result of the brake sensor 18b and the current vehicle speed of the vehicle 1.

  The steering angle sensor 19 is, for example, a sensor that detects a steering amount of the steering unit 4 such as a steering wheel. The steering angle sensor 19 is configured using, for example, a Hall element. The ECU 14 obtains, from the steering angle sensor 19, the amount of steering of the steering unit 4 by the driver, the amount of steering of each wheel 3 at the time of automatic steering when executing parking assistance, and executes various controls. The steering angle sensor 19 detects a rotation angle of a rotating part included in the steering section 4. The steering angle sensor 19 is an example of an angle sensor.

  The accelerator sensor 20 is, for example, a sensor that detects the position of the movable part of the acceleration operation unit 5. The accelerator sensor 20 can detect the position of an accelerator pedal as a movable part. The accelerator sensor 20 includes a displacement sensor.

  The shift sensor 21 is, for example, a sensor that detects the position of the movable part of the speed change operation unit 7. The shift sensor 21 can detect the position of a lever, arm, button, or the like as a movable unit. The shift sensor 21 may include a displacement sensor or may be configured as a switch.

  The wheel speed sensor 22 is a sensor that detects the amount of rotation of the wheel 3 and the number of rotations per unit time. The wheel speed sensor 22 is disposed on each wheel 3 and outputs a wheel speed pulse number indicating the number of rotations detected by each wheel 3 as a sensor value. The wheel speed sensor 22 can be configured using, for example, a Hall element. The ECU 14 calculates the amount of movement of the vehicle 1 and the like based on the sensor values acquired from the wheel speed sensors 22 and executes various controls. When calculating the vehicle speed of the vehicle 1 based on the sensor value of each wheel speed sensor 22, the CPU 14a determines the vehicle speed of the vehicle 1 based on the speed of the wheel 3 having the smallest sensor value among the four wheels, and executes various controls. I do. In addition, when there is a wheel 3 having a larger sensor value than the other wheels 3 among the four wheels, the CPU 14a determines, for example, that the number of rotations in a unit period (unit time or unit distance) is larger than that of the other wheels 3. When there are more wheels 3 than a predetermined number, the wheels 3 are regarded as being in a slip state (idling state), and various controls are executed. Note that the wheel speed sensor 22 may be provided in the brake system 18 not shown. In that case, the CPU 14a may acquire the detection result of the wheel speed sensor 22 via the brake system 18.

  The drive system 23 is an internal combustion engine (engine) system or a motor system as a drive source. The drive system 23 controls the fuel injection amount and intake air amount of the engine and the output value of the motor according to the required operation amount (for example, the depression amount of an accelerator pedal) of the driver (user) detected by the accelerator sensor 20. . Further, the output values of the engine and the motor can be controlled in cooperation with the control of the steering system 13 and the brake system 18 in accordance with the traveling state of the vehicle 1 irrespective of the operation of the driver.

  The configuration, arrangement, electrical connection form, and the like of the various sensors and actuators described above are merely examples, and can be set (changed) in various ways.

  The CPU 14a according to the present embodiment performs image processing such as viewpoint conversion processing and perspective projection processing on the captured image data (image) captured by the imaging unit 15 so that a virtual viewpoint different from the viewpoint when the imaging unit 15 captures an image. A virtual viewpoint image as viewed from is generated and displayed on the display device 8. For example, when performing parking assistance for guiding the vehicle 1 to the target parking area, an image is provided that makes it easy for the driver (user) to recognize the surrounding situation of the vehicle 1. Specifically, the position of the virtual viewpoint set above the vehicle 1 is continuously moved directly above the vehicle 1 in accordance with the moving state of the vehicle. At this time, both at least a part of the target parking area based on the captured image data and at least a part of the vehicle 1 are kept included in the display area of the virtual viewpoint image.

  FIG. 4 is an exemplary block diagram of a configuration for realizing image processing and parking assistance for moving the vehicle 1 according to the present embodiment in the CPU 14a included in the ECU 14. The configuration of the CPU 14a other than executing the image processing and the parking support of the present embodiment is omitted. The CPU 14a includes various modules for executing the parking support processing as described above and generating a virtual viewpoint image. The various modules are realized by the CPU 14a reading a program installed and stored in a storage device such as the ROM 14b and executing the program. For example, as shown in FIG. 4, the CPU 14a includes a parking support unit 30, a peripheral monitoring unit 32, and the like.

  The parking support unit 30 guides the vehicle 1 to a designated target parking area (parking frame, parking position, target area) by any one of a fully automatic travel, a semi-automatic travel, and a manual travel. The parking support unit 30 includes, for example, a target parking area setting unit 30a, a route acquisition unit 30b, a guidance control unit 30c, and the like as modules for performing parking support.

  The target parking area setting unit 30a sets a target parking area in which the vehicle 1 can move based on the captured image data captured by the imaging unit 15. For example, even when the vehicle 1 is traveling at a low speed (for example, a vehicle speed of 5 km / h or less), the target parking area setting unit 30a captures an image of the periphery of the vehicle 1 from the imaging units 15a to 15d as described above. Distance measurement data on the periphery of the vehicle 1 is sequentially acquired from the captured image data and the distance measurement units 16 and 17. Then, the target parking area setting unit 30a determines a target parking area candidate, which is a space in which the vehicle 1 can be accommodated, around the vehicle 1 based on the acquired captured image data and the ranging data, and the width and length of the vehicle 1. Explore. The target parking area setting unit 30a, when the target parking area candidates can be searched at a plurality of locations, presents them on the display device 8, and allows the driver to select a desired target parking area. If there is only one target parking area candidate, the position is presented on the display device 8. In addition, the target parking area setting unit 30a determines a plurality of conditions based on the surroundings of the vehicle 1 (for example, the distance from the entrance of the parking lot, the surrounding parking situation, and the distance from the entrance of the building if there is a building). A recommended target parking area may be presented from among the target parking area candidates. In this case, the target parking area setting unit 30a may obtain information on the position where the vehicle 1 is currently located (for example, parking lot information) from an external information center and select a recommended target parking area. Good.

  Based on the current position of the vehicle 1 and the target parking area set by the target parking area setting section 30a, the route acquisition section 30b guides the vehicle 1 from the current position to the target parking area with a minimum number of turns, for example. Get the route. The guidance route may be calculated and acquired by the route acquisition unit 30b, or the position of the vehicle 1 and the position of the target parking area are transmitted to an external processing device (for example, a parking lot management device or the like). The calculated guidance route may be acquired by receiving the route in the route acquisition unit 30b. A known technique can be used for calculating the guidance route, and a detailed description thereof will be omitted.

  When the guidance control unit 30c acquires an operation signal requesting the start of guidance of the vehicle 1 via the operation input unit 10 or the like, the guidance control unit 30c guides the vehicle 1 to the target parking area according to the guidance route acquired by the route acquisition unit 30b. For example, when performing guidance by fully automatic traveling, the guidance control unit 30c controls the steering system 13, the brake system 18, the drive system 23, and the like. In addition, for example, in the case of semi-automatic traveling in which only steering is automatically controlled, the operation amount and operation timing of an accelerator pedal, a brake pedal, and the like to be operated by the driver are displayed on the display device 8 and the display device 12, and from the audio output device 9. Output. Similarly, when the vehicle is driven manually, the steering direction and the steering amount of the steering wheel, the operation amount and operation timing of the accelerator pedal and the brake pedal are displayed on the display device 8 and the display device 12, and the audio output device 9 is operated. Or output from

  The periphery monitoring unit 32 generates a virtual viewpoint image as an image displayed on the display device 8 when parking assistance is performed by the parking assistance unit 30. Here, the virtual viewpoint image is a virtual position obtained by performing a viewpoint conversion process or the like on an image captured by the imaging unit 15 and having the viewpoint as the viewpoint, for example, as shown in FIGS. For example, the image is obtained when the vehicle 1 is moved over the vehicle 1 and imaged while looking down at the vehicle 1 from above. In order to generate such a virtual viewpoint image, the periphery monitoring unit 32 includes modules such as an image acquisition unit 34, a target acquisition unit 36, an image generation unit 38, and an output unit 40.

  After the target parking area is set, before and after the guidance of the vehicle 1 is started, the image acquisition unit 34 sequentially acquires captured image data of the periphery of the vehicle 1 from each of the imaging units 15a to 15d.

  The target acquisition unit 36 acquires the target parking area set by the target parking area setting unit 30a and, for example, coordinates of the target parking area in the coordinate system based on the current position of the vehicle 1 (own vehicle) and the vehicle. Obtain the rotation angle for 1 In this case, the current position of the vehicle 1 can be defined by, for example, a center position (reference position) defined by the length of the vehicle 1 in the vehicle width direction and the length in the vehicle length direction. In another embodiment, the center position of the rear wheel axle of the vehicle 1 in the vehicle width direction may be defined as the reference position of the vehicle 1. In addition, the coordinates of the target parking area can be defined by, for example, a center position defined by the width direction length and the depth direction of the target parking area. That is, when the reference position of the vehicle 1 is defined by the length in the vehicle width direction and the length in the vehicle length direction of the vehicle 1, the reference position is moved to the coordinates of the target parking area and the rotation is adjusted. As a result, the vehicle 1 can be guided to the center of the target parking area and housed in a well-balanced manner. Further, when the reference position of the vehicle 1 is defined by the center position of the rear wheel axle, the coordinates of the target parking area are set to positions that match the center position of the rear wheel axle of the vehicle 1 when the vehicle 1 is within the target parking area. It may be. Note that the target acquisition unit 36 sequentially acquires the coordinate position of the target parking area based on the current position (movement position) of the vehicle 1 and the rotation angle with respect to the vehicle 1 while the vehicle 1 is moving during parking assistance. I do. That is, the relative positional relationship between the vehicle 1 and the target parking area is sequentially acquired.

  The image generation unit 38 generates, for example, a gazing point / viewpoint processing unit 38a, a visual field setting unit 38b, a determinant calculation unit 38c, a drawing processing unit 38d, a superimposition processing unit 38e, and an index processing unit 38f in order to generate a virtual viewpoint image. And so on.

  The gazing point / viewpoint processing unit 38a sets the gazing point position 42 and the virtual viewpoint position 44 based on the position of the vehicle 1 and the position of the target parking area P, for example, as illustrated in FIG. In FIG. 5, the gazing point / viewpoint processing unit 38a firstly connects the vehicle center position S set immediately above the vehicle 1 (for example, 2 m above the ground) to the target parking area P (gazing point position 42). The virtual line of sight L1 is set. At this time, the length of the first virtual line of sight L1 is, for example, “1”. Then, the gazing point / viewpoint processing unit 38a multiplies the first virtual line of sight L1 by, for example, “N”, and inverts the vehicle center position S by 180 ° to the opposite side to the target parking area P, thereby changing the vehicle center. A second virtual line of sight L2 passing through the position S is set, and an end of the second virtual line of sight L2 opposite to the vehicle center position S is defined as a virtual viewpoint position 44. The gazing point / viewpoint processing unit 38a always determines the length of the first virtual line-of-sight line L1 connecting the vehicle center position S and the target parking area P, and the vehicle center position S, according to the moving situation of the vehicle 1 with respect to the target parking area P. The virtual viewpoint position 44 is set such that the ratio of the length of the second virtual line of sight L2 connecting the position of the virtual viewpoint position 44 and the position of the virtual viewpoint position 44 is kept constant. For example, the relationship between the gazing point position 42a of the target parking area Pa that is distant from the vehicle 1 and the virtual viewpoint position 44a set across the vehicle center position S is represented by a first virtual line of sight L1a: a second virtual line of sight. Line L2a = 1: N. Similarly, when the vehicle 1 moves and the vehicle 1 approaches the target parking area Pb, the relationship between the gazing point position 42b and the virtual viewpoint position 44b set across the vehicle center position S is the first. Virtual line of sight L1b: Second virtual line of sight L2b = 1: N.

  In this way, by setting the virtual viewpoint position 44 so that the relationship between the first virtual line of sight L1 and the second virtual line of sight L2 is always 1: N, the vehicle 1 approaches the target parking area P. , The virtual viewpoint position 44 gradually approaches the vehicle 1. In other words, it is possible to set the virtual viewpoint position 44 for converting into an image in which the periphery of the vehicle 1 is gradually enlarged (zoomed up). For example, when the target parking area P is far from the vehicle 1, the target parking area P is viewed from a distance across the vehicle 1, and the surrounding area including the distant target parking area P is displayed in a wide area to grasp the overall situation. The virtual viewpoint position 44 can be set so as to generate an easy-to-use image. Further, as the vehicle 1 approaches the target parking area P, the target parking area P is viewed from a position closer to the vehicle 1, and the situation around the vehicle 1 including the approached target parking area P is narrowed. The virtual viewpoint position 44 can be set so as to generate an image (enlarged image) that is displayed regionally and allows the user to easily recognize the sense of distance around the vehicle 1. As a result, it is possible to provide a virtual viewpoint image in which it is easy to confirm the surrounding situation according to the relative positional relationship between the vehicle 1 and the target parking area P, and it is possible to further improve the accuracy of surrounding monitoring. Although FIG. 5 illustrates an example in which the ratio between the first virtual line of sight L1 and the second virtual line of sight L2 is “1: N”, when the vehicle 1 moves, the first virtual line of sight L1 As long as the ratio between and the second virtual line of sight L2 is kept constant, the ratio may be changed as appropriate, and the same effect can be obtained.

  In another embodiment, the gazing point / viewpoint processing unit 38a can set the virtual viewpoint position 44 using a virtual curved surface E that covers the vehicle 1 as shown in FIG. In FIG. 6, the gazing point / viewpoint processing unit 38a first sets a virtual curved surface E having a predetermined radius (for example, radius = 8 m) around the vehicle 1. The gazing point / viewpoint processing unit 38a sets a virtual line of sight L connecting the vehicle center position S and the target parking area P (point of sight 42), and an extension of the virtual line of sight L sandwiches the vehicle 1. A position that intersects the virtual curved surface E on the opposite side of the target parking area P is defined as a virtual viewpoint position 44. Therefore, the virtual viewpoint position 44 is always set on the virtual curved surface E even when the vehicle 1 moves toward the target parking area P and the relative positional relationship changes. For example, a virtual line-of-sight line La connecting the gazing point position 42a of the target parking area Pa distant from the vehicle 1 and the vehicle center position S is farther from the gazing point position 42a across the vehicle center position S (opposite side). A virtual viewpoint position 44a is set at a position intersecting with the virtual curved surface E. Similarly, when the vehicle 1 moves and the vehicle 1 approaches the target parking area Pb, the virtual line-of-sight line Lb connecting the gazing point position 42b of the target parking area Pb and the vehicle center position S becomes the vehicle center position S The virtual viewpoint position 44b is set at a position that intersects the virtual curved surface E on the side (opposite side) far from the gazing point position 42b with respect to.

  Since the virtual viewpoint position 44 is always set on the virtual curved surface E, even when the vehicle 1 approaches the target parking area P, the distance at which the vehicle 1 faces the virtual viewpoint position 44 is kept substantially constant. That is, the size of the vehicle 1 displayed on the virtual viewpoint image is kept substantially constant. As a result, the size of the vehicle 1 does not change much on the virtual viewpoint image, the display content changes smoothly, and it is possible to provide a virtual viewpoint image that is easy to recognize, and it is easier to perform a monitoring operation with lower stress. .

  The gaze point / viewpoint processing unit 38a can select in advance whether to set the virtual viewpoint position 44 in the process shown in FIG. 5 or the virtual viewpoint position 44 in the process shown in FIG. Therefore, the gazing point / viewpoint processing unit 38a may omit processing functions that are not selected in advance.

  When setting the gazing point position 42 of the target parking area P, the gazing point / viewpoint processing unit 38a moves away from the vehicle 1 with respect to the area center position Ps of the target parking area P as shown in FIG. An offset process may be executed to set the position to a position moved by a predetermined amount (for example, 0.5 m). For example, when the gazing point position 42 is set at the area center position Ps, when the vehicle 1 fits in the target parking area P and the parking assistance is completed or almost completed, the target parking area P is set from directly above the vehicle 1. As a result, a display in a narrow range centering on the vehicle 1 is displayed. On the other hand, when the parking assistance is completed or almost completed, the driver may pay attention to a situation in front of the vehicle 1 in the traveling direction. For example, when the vehicle 1 is parked in the target parking area P, there may be a case where the user wants to check an empty space in the traveling direction. Therefore, as described above, by setting the gazing point position 42 to be offset in a direction away from the vehicle 1, the virtual viewpoint position 44 faces the vehicle 1 obliquely from the sky even when the parking assistance is completed. Thus, the traveling direction of the vehicle 1 can be displayed in a wider range. As a result, it is possible to generate a virtual viewpoint image that allows sufficient confirmation of an empty space or the like in the traveling direction.

  When the virtual viewpoint position 44 is set on the opposite side of the target parking area P across the vehicle center position S as in the present embodiment, when the vehicle 1 overruns the target parking area P, the virtual viewpoint position 44 is set on the traveling direction side beyond the vehicle center position S. As a result, when the virtual viewpoint position 44 exceeds the vehicle center position S, the viewing direction is reversed. For example, an unnatural image in which a virtual viewpoint image that faces the front side of the vehicle 1 from behind the vehicle 1 suddenly changes to a virtual viewpoint image that faces the rear side of the vehicle 1 from the front of the vehicle 1. Switching may occur, giving a user such as a driver an uncomfortable feeling. As described above, by setting the gazing point position 42 to be offset in the direction away from the vehicle 1, it is possible to prevent the virtual viewpoint position 44 from exceeding the vehicle center position S even when parking assistance is completed. . As a result, it is possible to prevent such an inconvenience that the image is suddenly switched (reversed). As shown in FIG. 7, the same applies to the case where the vehicle 1 moves forward and approaches the target parking area P, and the case where the vehicle 1 moves backward and approaches the target parking area P. Note that the offset processing performed by the gazing point / viewpoint processing unit 38a changes target coordinates for guiding the reference position of the vehicle 1 in parking assistance by changing the position of the gazing point position 42 in the target parking area P. Not something.

  The visual field setting unit 38b sets a visual field when the gazing point position 42 is viewed from the virtual viewpoint position 44. For example, the field of view can be set regardless of the positional relationship between the vehicle 1 and the target parking area P, or according to the positional relationship. By changing the field of view, it is possible to set a narrow area display or a wide area display of the virtual viewpoint image, and it is possible to smoothly respond to a driver's request for changing the visible range or a request for detailed display.

  The determinant operation unit 38c is used to determine the coordinates of the space when the gazing point position 42 is viewed from the virtual viewpoint position 44 when the captured image data acquired from the imaging unit 15 is drawn as a virtual viewpoint image. (Line-of-sight determinant) is calculated. For example, a line-of-sight determinant for converting coordinates on the captured image data to coordinates on the virtual viewpoint image or converting coordinates of a polygon (own vehicle icon) indicating the vehicle 1 on the virtual viewpoint image is calculated. . In addition, the determinant calculation unit 38c calculates a perspective projection determinant, which is a determinant for converting image data into coordinates in a two-dimensional space that matches the display area of the display device 8 that displays the virtual viewpoint image. When calculating the perspective projection determinant, the determinant may be calculated in consideration of a change in the visual field set by the visual field setting unit 38b. Then, the determinant calculation unit 38c calculates a conversion determinant based on the line-of-sight determinant and the perspective projection determinant. By using the transformation determinant, for example, an image (captured image data) captured by each image capturing unit 15 is converted into a 3D model centered on the vehicle 1 (for example, a bowl-shaped model in which the bottom is a circle and the periphery rises). A virtual viewpoint image projected to the virtual camera can be generated. In addition, the determinant calculation unit 38c determines the display position and display posture of a virtual vehicle (ghost) that images the state in which the vehicle 1 has reached the target parking area P by using an index specified by the index processing unit 38f. A determinant (translation rotation determinant) or the like for changing according to the movement of the position 44 is calculated.

  The drawing processing unit 38d performs coordinate conversion of the image data captured by the imaging unit 15 using the conversion determinant calculated by the gazing point / viewpoint processing unit 38a, and the vehicle viewed from the current virtual viewpoint position 44. A virtual viewpoint image showing the periphery of No. 1 is drawn.

  By the way, as shown in FIG. 2, the imaging unit 15 is arranged, for example, on the outer periphery of the vehicle 1. Therefore, the captured image data captured by the imaging unit 15 includes only a part of the vehicle 1 (a part of a bumper, a part of a door, and the like). Therefore, the image of the vehicle 1 cannot be displayed on the virtual viewpoint image even if the coordinates of the captured image data captured by the imaging unit 15 are converted using the conversion determinant. Therefore, the index processing unit 38f reads the own vehicle icon data indicating the vehicle 1 from a storage unit such as the ROM 14b. Then, the superimposition processing unit 38e applies the transformation determinant calculated by the gazing point / viewpoint processing unit 38a to the own vehicle icon data, and draws the own vehicle icon viewed from the current virtual viewpoint position 44 in the drawing processing unit. 38d is superimposed on the virtual viewpoint image showing the periphery of the vehicle 1 drawn by 38d.

  As described above, the index processing unit 38f includes, in addition to the own vehicle icon displayed on the virtual viewpoint image, an image of a virtual vehicle (ghost) that makes the vehicle 1 reach the target parking area P, the target parking area P Is managed as an index using a parking frame image or the like for specifying the information. The index processing unit 38f includes detailed modules such as an index selecting unit 38f1 for selecting each index and a display mode changing unit 38f2 for changing the display mode of the selected index.

  The index selecting unit 38f1 may select an icon that reproduces the shape of the vehicle 1 as an own-vehicle icon indicating the vehicle 1 by an operation performed by the user using the operation input unit 10 or the like. The own vehicle icon may be selected. When an icon that reproduces the shape of the vehicle 1 is used, a good-looking virtual viewpoint image can be provided as if the vehicle 1 is actually looking down. On the other hand, when the own-vehicle icon having a simple shape is used, it is possible to reduce the calculation load for the display processing of the own-vehicle icon whose display posture changes following the change of the virtual viewpoint position 44. Further, when a user is requested to display an index such as a virtual vehicle (ghost) or a parking frame by an operation using the operation input unit 10 or the like, the index selection unit 38f1 selects the requested index. Do. Also in this case, an index reproducing an actual shape or an index of a simple shape may be made selectable. As the data for displaying the virtual vehicle, data of the own vehicle icon may be used, or individual data may be prepared.

  When a plurality of indices are superimposed and displayed on the virtual viewpoint image, the display mode changing unit 38f2 can change the display mode so that the discriminability of each index is improved. For example, when the own vehicle icon and the virtual vehicle (ghost) are displayed at the same time, their display colors are different so that they can be identified. For example, discrimination can be improved by setting the own vehicle icon to a display color that matches the actual color of the vehicle 1 and changing the display color of the virtual vehicle to the opposite color in the hue circle. In addition, the display mode changing unit 38f2 may change, for example, the depth of the display color of the virtual vehicle according to the distance between the own vehicle icon and the virtual vehicle (the target parking area P). For example, when the vehicle is far away, the display color of the virtual vehicle is made darker, and as the own vehicle icon and the virtual vehicle approach, the display color of the virtual vehicle is made lighter. The display mode changing unit 38f2 may change the transmittance of the virtual vehicle. In addition, the display mode changing unit 38f2 may improve the discriminability by blinking one of the vehicle icon and the virtual vehicle. Note that another display mode may be adopted as long as the display mode allows a plurality of indices to be identified.

  The output unit 40 sequentially outputs the virtual viewpoint images generated by the image generation unit 38 to the display control unit 14d, and causes the display device 8 to display the virtual viewpoint images.

  Note that the module configuration shown in FIG. 4 is an example, and if the same processing can be performed, division or integration of functions can be appropriately performed.

  8 to 10 show an example of the virtual viewpoint image 46 displayed on the display device 8, and are schematic diagrams showing a state in which the vehicle 1 moves while moving backward toward the target parking area P. The virtual viewpoint position 44 shows an example of moving on the virtual curved surface E as shown in FIG. Therefore, the virtual viewpoint position 44 is set on the front side of the vehicle 1 with respect to the target parking area P with the vehicle 1 interposed therebetween, and the virtual viewpoint position 44 is directly above the vehicle center position S (vehicle 1) on the virtual curved surface E. Will move towards.

  FIG. 8 is a virtual viewpoint image 46 showing a case where the vehicle 1 is still far from the target parking area P. In this case, since the own vehicle icon 48a (own vehicle index) indicating the vehicle 1 is far from the parking frame index 48b indicating the target parking area P, a part of the parking frame index 48b is outside the display range of the virtual viewpoint image 46 ( It is located in a portion (black portion on the screen) outside the image data of the imaging unit 15. In this case, the portion of the parking frame index 48b outside the display range may be hidden. In the case of FIG. 8, the position of the target parking area P is further clarified with respect to an actual parking frame line that is curved and displayed at a distance from the vehicle 1 (own vehicle icon 48a) on the virtual viewpoint image 46. Frame index 48b is displayed as a well-formed rectangle. In another embodiment, the parking frame index 48b may be modified using a transformation determinant to be displayed in a state close to the actual display frame line.

  FIG. 9 is a virtual viewpoint image 46 showing a state where the vehicle 1 has entered a part of the target parking area P. In this case, since the front-rear direction of the vehicle 1 approaches the depth direction of the target parking area P, the actual parking frame line and the rectangular parking frame index viewed from the virtual viewpoint position 44 approaching the vehicle center position S. 48b is displayed on the virtual viewpoint image 46 so as to substantially coincide with 48b.

  FIG. 10 is a virtual viewpoint image 46 showing a state in which the vehicle 1 has completely entered the target parking area P and parking assistance has ended. In this case, as described with reference to FIG. 7, the gazing point / viewing point processing unit 38a sets the gazing point position 42 to be offset in a direction away from the vehicle 1 so that the traveling direction of the vehicle 1 (in this case, the vehicle 1 (Behind) is sufficiently displayed in the virtual viewpoint image 46, so that the space behind the vehicle 1 can be more easily confirmed. Further, it is possible to prevent a problem that the virtual viewpoint image 46 is rotated by 180 ° (the direction of the line of sight is reversed) immediately before or immediately after the completion of the parking support.

  In any of FIGS. 8 to 10, at least a part of the target parking area P and at least a part of the vehicle 1 are continuously displayed on the virtual viewpoint image 46. As a result, it is easy to always recognize the relative positional relationship between the target parking area P and the vehicle 1 that the driver wants to pay attention to while traveling with parking assistance. In addition, since the virtual viewpoint position 44 is always directed to the target parking area P, an image with less unnatural distortion can be provided. As a result, when the target parking area P exists far away, it is easy to check the surroundings of the target parking area P, and when the target parking area P approaches, it is possible to check the surroundings of the vehicle and grasp the sense of distance. This can contribute to the improvement of the accuracy of the peripheral monitoring, and also facilitates the monitoring operation with low stress.

  Although not shown in FIGS. 8 to 10, when the virtual viewpoint image 46 is displayed on the display device 8, a user interface such as a message explaining the current situation and switches for performing necessary operations is provided. May be superimposed.

  An example of the flow of the display processing of the virtual viewpoint image 46 by the peripheral monitoring device configured as described above will be described with reference to the flowchart in FIG. In addition, guidance of the vehicle 1 for parking assistance may be performed by fully automatic driving, semiautomatic driving, or manual driving.

  When the power of the vehicle 1 is turned on, the target parking area setting unit 30a and the image acquisition unit 34 always acquire captured image data (peripheral images) from each imaging unit 15 regardless of whether or not the vehicle 1 is traveling (S100). ). In addition, the target parking area setting unit 30a, based on the captured image data acquired by the target parking area setting unit 30a and the distance measurement data acquired by the distance measurement units 16 and 17, obtains information about the peripheral objects around the vehicle 1 (the peripheral object information). The presence / absence and, if there is a surrounding object, the distance to the surrounding object, etc.) are acquired (S102). The peripheral monitoring unit 32 monitors whether a request for peripheral monitoring (parking support) has been performed via the operation input unit 10 or the like (S104). If the request operation has not been performed (No in S104), the peripheral monitoring unit 32 temporarily stops. This flow is terminated, and input of a request operation is waited.

  In S104, when a request operation for periphery monitoring (parking support) is performed (Yes in S104), the target acquisition unit 36 acquires the target parking area P set by the target parking area setting unit 30a (S106). Then, the target acquisition unit 36 acquires the coordinates of the target parking area P in the coordinate system based on the current position of the vehicle 1 (own vehicle) and the rotation angle of the target parking area P with respect to the vehicle 1 (S108). The rotation angle can be used when superimposing and displaying a virtual vehicle, a parking frame index, and the like.

  Subsequently, the gazing point / viewpoint processing unit 38a moves the gazing point position 42 of the target parking area P acquired by the target acquiring unit 36 by a predetermined amount in a direction away from the vehicle 1 with respect to the area center position Ps of the target parking area P. It is confirmed whether or not the offset processing for the position where the position has been performed has been executed (S110). If the offset process has not been executed yet (No in S110), the gazing point / viewpoint processing unit 38a determines whether the gazing point position 42 has been moved by a predetermined amount in the direction away from the vehicle 1 with respect to the area center position Ps. (S112). If the offset processing has been executed (Yes in S110), the gazing point / viewpoint processing unit 38a skips the processing in S112. Note that the offset processing may be performed until the virtual viewpoint position 44 reaches a position immediately above the vehicle 1. In the display stage of FIGS. 8 and 9, the gazing point position 42 is set to the region center position Ps. Processing may be performed. By offsetting the gazing point position 42 in the initial stage, it is possible to prevent a sudden change in the line of sight while the virtual viewpoint image 46 is being displayed.

  Then, the gazing point / viewpoint processing unit 38a sets a virtual line of sight L connecting the vehicle center position S set immediately above the vehicle 1 and the gazing point position 42 of the target parking area P (S114). 44 is determined (S116).

  When the gazing point position 42 and the virtual viewpoint position 44 are set, the determinant calculation unit 38c executes the calculation of the line-of-sight determinant (S118), the calculation of the perspective projection determinant (S120), and performs the line-of-sight determinant and the perspective projection. Based on the determinant, a transformation determinant that is a coordinate transformation formula for displaying the virtual viewpoint image 46 is calculated (S122).

  The drawing processing unit 38d draws the virtual viewpoint image 46 on the 3D model using the conversion determinant calculated by the determinant calculation unit 38c (S124). Subsequently, the index selecting unit 38f1 selects the own vehicle icon 48a corresponding to the vehicle 1 and displays the own vehicle icon 48a when the vehicle 1 is viewed at the current virtual viewpoint position 44 using the conversion determinant. The coordinate conversion is performed, and the own vehicle icon 48a is superimposed on the virtual viewpoint image 46 (S126). In addition, when the driver or the like requests the display of the index such as the virtual vehicle or the parking frame index 48b through the operation input unit 10 or the like (Yes in S128), the index selection unit 38f1 sets the requested index. Is selected, and the index is superimposed on the virtual viewpoint image 46 (S130). Further, the display mode changing unit 38f2 changes the display mode of the index to be superimposed and displayed according to, for example, the relative distance between the target parking area P and the vehicle 1. When the display image of the index is not requested (No in S128), the process of S130 is skipped.

  The output unit 40 includes a virtual virtual machine that includes at least a part of the own vehicle icon 48a (vehicle 1) and at least a part of the target parking area P (parking frame index 48b) generated by the image generating unit 38. Data indicating the virtual viewpoint image 46 (synthesized image) when viewed from the viewpoint position 44 is output to the display control unit 14d (S132), and the display device 8 displays the virtual viewpoint image 46 at the current position of the vehicle 1. When the surrounding monitoring unit 32 determines that the vehicle 1 has arrived at the target parking area P (Yes in S134), the vehicle 1 fits in the target parking area P in a predetermined posture, and parking assistance requiring surrounding monitoring is performed. It is determined that the process has been completed, and this flow is temporarily terminated. On the other hand, when it is determined that the vehicle 1 has not reached the target parking area P (No in S134), the process proceeds to S108, and based on the current position of the vehicle 1 moved from the previous display processing, the target parking area P is determined. , And the process of S108 is repeated to update the virtual viewpoint image 46 (synthesized image) displayed on the display device 8 and display it as a moving image.

  FIGS. 8 to 10 show an example in which the vehicle 1 reaches the target parking area P only by reversing. In another embodiment, depending on the guidance route, the vehicle 1 may temporarily move forward for turning back or the like. For example, when a change in the traveling direction is detected based on a detection signal from the shift sensor 21, the visual field setting unit 38b temporarily widens the visual field and changes the arrival position by switching back (the position at which the traveling direction is switched again). The virtual viewpoint image 46 may be included in the display range. In this case, the surrounding situation of the vehicle 1 can be more appropriately grasped, and the driver's sense of security can be improved. The same applies to the case where parking assistance is performed in forward running, and the same effect can be obtained.

  The program for the peripheral monitoring process executed by the CPU 14a according to the present embodiment is a file in an installable format or an executable format in a CD-ROM, a flexible disk (FD), a CD-R, a DVD (Digital Versatile Disk). It may be configured such that it is recorded on a computer-readable recording medium and provided.

  Further, the peripheral monitoring processing program may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Further, the peripheral monitoring processing program executed in the present embodiment may be provided or distributed via a network such as the Internet.

  Although the embodiment and the modification of the present invention have been described, the embodiment and the modification are presented as examples and are not intended to limit the scope of the invention. These new embodiments can be implemented in other various forms, and various omissions, replacements, and changes can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are also included in the invention described in the claims and their equivalents.

  DESCRIPTION OF SYMBOLS 1 ... vehicle, 8 ... display apparatus, 14 ... ECU, 14a ... CPU, 15, 15a, 15b, 15c, 15d ... imaging part, 30 ... parking assistance part, 30a ... target parking area setting part, 30b ... route acquisition part, 30c: guidance control unit, 32: peripheral monitoring unit, 34: image acquisition unit, 36: target acquisition unit, 38: image generation unit, 38a: gazing point / viewpoint processing unit, 38b: visual field setting unit, 38c: determinant calculation Unit, 38d: drawing processing unit, 38e: superimposition processing unit, 38f: index processing unit, 38f1: index selection unit, 38f2: display mode changing unit, 40: output unit, 42, 42a, 42b: fixation point position, 44, 44a, 44b: virtual viewpoint position, 46: virtual viewpoint image, 48a: own vehicle icon, 48b: parking frame index, E: virtual curved surface, L: virtual line of sight, L1: first virtual line of sight, L2: second virtual Line of sight 100 ... periphery monitoring system.

Claims (7)

An image acquisition unit that acquires captured image data of an area around the vehicle,
A target acquisition unit that acquires a target parking area for guiding the vehicle,
When guiding the vehicle to the target parking area, while continuously moving the position of the virtual viewpoint set above the vehicle directly upward in accordance with the moving situation of the vehicle, the captured image data An image generation unit that generates a virtual viewpoint image such that both at least a part of the target parking area based on the vehicle and at least a part of the vehicle continue to be included in a display area;
An output unit that outputs the virtual viewpoint image to a display device,
A peripheral monitoring device, including:   The image generation unit is configured to determine a length of a first virtual line of sight connecting a vehicle center position set immediately above the vehicle and the target parking area, and a second virtual line connecting the vehicle center position and the position of the virtual viewpoint. The surroundings monitoring device according to claim 1, wherein the virtual viewpoint image is generated while moving the position of the virtual viewpoint in accordance with a moving state of the vehicle while keeping a ratio with a line of sight line length constant.   The image generation unit is configured such that a virtual curved line that covers the vehicle, a virtual line of sight passing through the target parking area and a vehicle center position set immediately above the vehicle is on the opposite side of the target parking area across the vehicle. The surroundings monitoring device according to claim 1, wherein the virtual viewpoint image is generated while moving the position of the virtual viewpoint set at the intersection position according to a moving state of the vehicle.   The image generation unit sets a position of a gazing point when facing the target parking area from the position of the virtual viewpoint by a predetermined distance in a direction away from the vehicle with respect to an area center position of the target parking area. The peripheral monitoring device according to any one of claims 1 to 3.   The surroundings monitoring device according to any one of claims 1 to 4, wherein the image generation unit changes a viewing angle of the virtual viewpoint image according to a movement state of the position of the virtual viewpoint.   The image generation unit is a position corresponding to the target parking area in the virtual viewpoint image, at least a parking frame image indicating the target parking area and a virtual vehicle indicating a situation in which the vehicle has moved to the target parking area. The peripheral monitoring device according to any one of claims 1 to 5, wherein an index indicating one is superimposed and displayed.   The surroundings monitoring device according to claim 6, wherein the image generation unit displays the vehicle and the index in the virtual viewpoint image in different display modes.

Images