JP2017200000A - Periphery monitoring device and periphery monitoring method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To allow for correct recognition of a distance feeling to an object existing on the periphery of a vehicle by an operator.SOLUTION: A periphery monitoring system is applied to a vehicle equipped with an on-vehicle camera 10 for shooting the vehicle periphery, and a display device 11 for displaying an image, where an image captured by the on-vehicle camera 10 is converted into a bird's eye view, viewing the pavement from a virtual view point set above the vehicle, and displayed in the display device 11. An ECU 20 crops the image in the peripheral region of the ground part of the object on the pavement and the pavement from a captured image or the bird's eye view image, and connects the remaining images thus performing correction for matching the position of the object on an image displayed in the display device 11 to the actual position of the object.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a periphery monitoring device and a periphery monitoring method, and more specifically, a periphery monitoring device and a periphery monitoring method applied to a vehicle equipped with a camera having a photographing area around the vehicle and a display device for displaying an image. About.

  Conventionally, a situation around a vehicle is photographed with an in-vehicle camera, and a captured image photographed with the in-vehicle camera is converted into a bird's-eye view image and displayed on an in-vehicle display device (see, for example, Patent Document 1). ). In Patent Document 1, a photographed image is converted into a bird's-eye view image, and if the outline of the object is specified by the obstacle detection sensor, the outline of the object is included so as to include at least a region surrounded by the specified outline of the object. It is disclosed that a target region, which is a bird's-eye image region with a part of the boundary, is compressed in a direction toward the center of the camera when the photographed image is photographed and output as a final output image. Thereby, the distorted image is corrected by converting the captured image into the bird's-eye view image, and the situation around the own vehicle is more accurately grasped.

JP 2012-156632 A

  When a three-dimensional object such as a vehicle on the road surface is one of the subjects, a part floating from the road surface such as a bumper part is actually converted into a bird's-eye view of the road surface viewed from above the vehicle. The image is displayed as being located farther than the position of. In this case, there is a concern that the driver who sees the display image may misidentify the sense of distance from an object existing around the vehicle.

  The present invention has been made in view of the above problems, and an object thereof is to provide a periphery monitoring device and a periphery monitoring method that allow a driver to correctly recognize a sense of distance from an object existing around a vehicle. To do.

  The present invention employs the following means in order to solve the above problems.

  The present invention is applied to a vehicle (50) equipped with a camera (10) having a shooting area around the vehicle and a display device (11) for displaying an image. The present invention relates to a periphery monitoring device that converts a virtual viewpoint set above a vehicle into a bird's-eye image viewed on a road surface and displays the bird's-eye image on the display device. According to the first aspect of the present invention, an image of an area around the ground contact portion (54) between the object on the road surface and the road surface is cut out from the photographed image or the bird's-eye view image, and the remaining images after the cut-out are joined together. Accordingly, an image correction unit that performs correction for matching the position of the object on the image displayed on the display device with the actual position of the object is provided.

  In the above configuration, the image of the peripheral area of the contact portion between the object on the road surface and the road surface is cut out from the photographed image or the bird's-eye view image, and the remaining image after the cut-out is connected to thereby position the object on the display image. Correction to match the actual position is performed. When shooting a three-dimensional object such as a vehicle on the road surface, if it is converted to a bird's-eye view image on a plane assumption, the object is displayed at a position farther than the actual position on the bird's-eye view image, and the driver misrecognizes the distance to the object. There is a risk. In view of this point, with the above-described configuration, the display device can display an object at a correct position. As a result, the driver can correctly recognize a sense of distance from an object existing around the vehicle.

The figure which shows the mounting position of a vehicle-mounted camera. The block diagram which shows schematic structure of the vehicle periphery monitoring system of 1st Embodiment. The figure for demonstrating the display shift accompanying a viewpoint conversion. The figure explaining the specific aspect of an image correction process. 6 is a flowchart showing a processing procedure for image correction processing. The block diagram which shows schematic structure of the vehicle periphery monitoring system of 2nd Embodiment. The figure explaining the specific aspect of an image correction process. 6 is a flowchart showing a processing procedure for image correction processing. The figure explaining the method of calculating | requiring the reflective surface information of an object.

(First embodiment)
Hereinafter, a first embodiment will be described with reference to the drawings. In the following embodiments, parts that are the same or equivalent to each other are denoted by the same reference numerals in the drawings, and the description of the same reference numerals is used.

  First, with reference to FIG.1 and FIG.2, the vehicle periphery monitoring system of this embodiment is demonstrated. As shown in FIG. 1, the vehicle periphery monitoring system according to the present embodiment includes a vehicle (hereinafter referred to as “own vehicle 50”) in which an in-vehicle camera 10 that has an imaging area A1 around the vehicle and a display device 11 that displays an image are mounted. "). In the system, the ECU 20 performs control for converting the captured image captured by the in-vehicle camera 10 into a bird's-eye image viewed on the road surface from the virtual viewpoint set above the host vehicle 50 and displaying the image on the display device 11. .

  The in-vehicle camera 10 is composed of a monocular camera such as a CCD camera, a CMOS image sensor, or a near infrared camera. The in-vehicle camera 10 is attached to a predetermined height (for example, a position above the front bumper) in the center of the vehicle width direction at the front portion of the host vehicle 50, and has a region extending in a predetermined angle range toward the front of the vehicle. Shoot from a bird's eye view. The display device 11 is provided at a position (for example, an instrument panel) that is visible to the driver.

  FIG. 2 is a block diagram showing a schematic configuration of the vehicle periphery monitoring system of the present embodiment. As shown in FIG. 2, the system includes an ultrasonic sensor 12 that transmits a search wave and receives a reflected wave of the search wave as a distance measurement sensor that detects a distance to an object existing around the host vehicle 50. Is provided.

  The ultrasonic sensor 12 is, for example, a sonar, and transmits an exploration wave at every predetermined control period, and receives a reflected wave reflected by an object existing around the host vehicle 50. Based on the reflected wave time, which is the time from transmission to reception, the distance between the object existing around the host vehicle 50 and the host vehicle 50 is calculated. In the present embodiment, a plurality of ultrasonic sensors 12 are respectively attached to the front and rear bumper portions of the host vehicle 50 at predetermined intervals in the vehicle width direction.

  ECU20 is a computer provided with CPU, ROM, RAM, I / O, etc., CPU displays the condition of the periphery of the own vehicle 50 on the display apparatus 11 by running the program installed in ROM. Each function to make the driver grasp it. The ECU 20 of this embodiment includes an image acquisition unit 21, a viewpoint conversion unit 22, a distance calculation unit 23, an image correction unit 24, and an image output unit 25.

  The image acquisition unit 21 is connected to the vehicle-mounted camera 10 and acquires captured images captured by the vehicle-mounted camera 10 at a constant period. The acquired captured image is output to the viewpoint conversion unit 22. The viewpoint conversion unit 22 converts the captured image acquired by the image acquisition unit 21 into a bird's-eye image viewed on the road surface with the virtual viewpoint set above the host vehicle 50 as the viewpoint position by using a known coordinate conversion formula. To do. In the present embodiment, the image is converted into a bird's-eye view image in which the ground surface is looked down in the vertical direction. The viewpoint conversion unit 22 outputs the generated bird's-eye view image to the image correction unit 24.

  The distance calculation unit 23 is connected to the ultrasonic sensor 12 and inputs the reflected wave time as position information related to the object detected by the ultrasonic sensor 12. Further, based on the input reflected wave time, an actual distance d with respect to the object reflecting the exploration wave is calculated. The image correction unit 24 receives the bird's-eye image from the viewpoint conversion unit 22, performs correction as necessary, and outputs the image to the image output unit 25. The image output unit 25 outputs the bird's-eye image input from the image correction unit 24 to the display device 11 and causes the display device 11 to display the bird's-eye image.

  Here, when a three-dimensional object such as a vehicle existing on the road surface is one of the subjects, a portion that floats from the road surface, such as a bumper portion, is farther from the bumper portion when the captured image is converted into a bird's-eye view image. Assuming that the road surface under the vehicle that is supposed to be located at a position closer to the host vehicle 50, the bumper image is connected to the road image. For this reason, the bumper image is displayed on the display device 11 at a position farther than the actual position, and there is a concern that the driver who viewed the display image may misidentify the sense of distance from an object existing around the vehicle. The

  FIG. 3 is a diagram for explaining a display shift caused by viewpoint conversion. FIG. 3A is a photographed image obtained by photographing the other vehicle 51 stopped in front of the host vehicle 50 with the in-vehicle camera 10. b) is a view of the positional relationship between the host vehicle 50 and the other vehicle 51 as viewed from the side and above. The lower diagram in (b) corresponds to the bird's-eye view image. A white line 52 in the photographed image of FIG. 3A indicates the position of the rear end of the rear bumper of the other vehicle 51 on the road surface 53 with a tape or the like when the other vehicle 51 is looked down vertically toward the ground surface. It is a thing. An arrow A represents the vertical direction of the image, and an arrow B represents the horizontal direction of the image (hereinafter the same).

The conversion from the captured image to the bird's-eye view image is performed as an image at a position farther from the host vehicle 50 in order from the lower pixel in the captured image upward. Here, as shown in FIG. 3A, the point P on the rear end of the rear bumper of the other vehicle 51 and the white line 52 are actually the same distance from the host vehicle 50. Regardless, the point P is represented at a position above the white line 52 on the captured image. Further, the point P is represented at a position above the point Q (for example, the ground contact portion between the other vehicle 51 and the road surface 53) on the road surface below the vehicle on the photographed image. Therefore, when the captured image of FIG. 3A is converted into a bird's-eye view image, the rear end of the rear bumper of the other vehicle 51 corresponds to the number of pixels from the white line 52 to the lower end of the bumper. It is displayed at a far position. That is, as shown in FIG. 3B, the other vehicle 51 is displayed on the display device 11 as an obstacle located at a position farther from the host vehicle 50 by the display deviation amount x than the actual distance d. The display deviation amount x can be calculated by the following equation (1) using the mounting height Ch of the in-vehicle camera 10, the bumper lower end height Bh, and the actual distance d from the other vehicle 51.
x = (Bh / (Ch−Bh)) × d (1)

  In view of these points, the image correction unit 24 cuts out a part of the bird's-eye view image, more specifically, an image of a peripheral region of the ground contact unit 54 between the object (the other vehicle 51 in FIG. 3) and the road surface 53. After that, the remaining images are connected to each other so that the distance to the object on the display image is matched with the actual distance d.

  FIG. 4 is a diagram for explaining a specific mode of image correction processing for eliminating display shifts associated with viewpoint conversion in the present embodiment. In FIG. 4, (a) is a photographed image obtained by photographing another vehicle 51 stopped in front of the host vehicle 50, and (b) is a bird's-eye image generated by the viewpoint conversion unit 22. The image correction unit 24 first acquires a bird's-eye image from the viewpoint conversion unit 22. And the process which cuts out the image of the peripheral region of the grounding part 54 of the other vehicle 51 on the acquired bird's-eye view image is executed.

  Specifically, the image correction unit 24 uses the distance information of the object acquired by the ultrasonic sensor 12 and converts the distance information into coordinates on the bird's-eye view image, thereby specifying the sensor detection point. Estimate the actual position of the object at. In addition, an image of a predetermined area above the sensor detection point is set as a cutout area (see FIG. 4C). Here, based on the sensor detection point sequence, an image region having a predetermined width H1 in the upward direction of the image from the sensor detection point sequence and having a predetermined width H2 corresponding to a typical vehicle width in the left-right direction of the image is cut out. Set to area. In the present embodiment, the predetermined width H1 is set according to the display deviation amount x calculated using the above equation (1). The predetermined width H1 may be a predetermined constant value. The predetermined widths H1 and H2 may be set according to the vehicle type of the other vehicle 51.

  When the cutout area is set, the image of the cutout area is cut out from the bird's-eye view image. After the images are cut out, the images above the cut-out area are shifted downward to connect the images, and the images after the connection are displayed on the display device 11. The distance calculation unit 23 and the image correction unit 24 function as a position estimation unit.

  Next, the processing procedure of the image correction process will be described with reference to the flowchart of FIG. This process is executed by the ECU 20 at predetermined intervals.

  In FIG. 5, in step S11, a captured image is acquired from the in-vehicle camera 10, and in step S12, the captured image is subjected to viewpoint conversion to generate a bird's-eye view image. In subsequent step S13, the position information of the object acquired by the ultrasonic sensor 12 is read from the memory. In step S14, the actual position of the object on the bird's-eye view image is estimated from the position information of the read object, and a cutout area is set in the image area above the estimated position. In subsequent step S15, the image of the set cut-out area is cut out from the bird's-eye view image, and in step S16, the image above the cut-out area is shifted downward to connect the images. In step S17, the corrected bird's-eye view image is output to the display device 11, and the process ends.

  According to the embodiment described in detail above, the following excellent effects can be obtained.

  The image of the area around the ground contact portion 54 between the object on the road surface and the road surface 53 is cut out from the bird's-eye view image, and the remaining images after the cut-out are connected to match the position of the object on the display image with the actual position. It was set as the structure which performs correction | amendment. When a three-dimensional object with a partly floating structure such as a vehicle is converted into a bird's-eye view image on the assumption of a plane, the object is displayed at a position farther than the actual position on the bird's-eye view image. In view of this point, with the above-described configuration, the display device 11 can display an object at a correct position. As a result, the driver can correctly recognize the sense of distance from objects existing around the vehicle.

  The actual position on the image of the object recognized by the vehicle-mounted camera 10 is estimated, and based on the estimated position, an image clipping region is set in the peripheral region of the grounding unit 54, and the image of the set clipping region is set. Is cut out from a bird's-eye view image. According to such a configuration, it is possible to accurately grasp the display deviation amount x when it is a bird's-eye view image and cut out the image, and display position correction can be performed with high accuracy.

  In particular, in the present embodiment, the actual position of the object on the image is estimated based on the detection result of the distance measuring sensor, and the image clipping region is set based on the estimated position. According to the distance measuring sensor, the actual distance from the object can be detected with high accuracy, and the display deviation amount x can be accurately grasped.

  If the cutout area is too large, the image may be unnatural when displayed on the display device 11 or the process may take a long time. In view of this point, the image clipping region is set according to the object width. By adopting such a configuration, it is possible to provide the driver with a more natural image while eliminating the display deviation of the position of the object.

(Second Embodiment)
Next, the second embodiment will be described focusing on the differences from the first embodiment. In the first embodiment, the image correction unit 24 cuts out an image of the peripheral region of the grounding unit 54 from the bird's-eye view image, and performs correction for joining the remaining images after the cut-out. On the other hand, in the present embodiment, an image of the peripheral area of the grounding unit 54 is cut out from the captured image, and correction is performed to join the remaining images after the cut-out.

  FIG. 6 is a block diagram showing a schematic configuration of the vehicle periphery monitoring system of the present embodiment. In FIG. 6, the image correction unit 24 inputs a captured image of the in-vehicle camera 10 from the image acquisition unit 21, and performs a correction process for suppressing a display shift on the display image for the input captured image. The viewpoint conversion unit 22 converts the corrected captured image input from the image correction unit 24 into a bird's-eye image using a known coordinate conversion formula, and outputs the generated bird's-eye image to the image output unit 25. The image output unit 25 outputs the bird's-eye image input from the viewpoint conversion unit 22 to the display device 11 and causes the display device 11 to display the bird's-eye image.

  Next, a specific aspect of the image correction process of the present embodiment will be described with reference to FIG. FIG. 7A is a photographed image obtained by photographing the other vehicle 51 stopped in front of the host vehicle 50 with the in-vehicle camera 10. First, the image correction unit 24 performs a process of cutting out an image of a peripheral region of the ground contact unit 54 of the other vehicle 51 on the captured image acquired from the image acquisition unit 21.

  Specifically, the image correction unit 24 identifies the sensor detection point by converting the distance information to the object acquired by the ultrasonic sensor 12 into coordinates on the bird's-eye view image, and a predetermined region below the sensor detection point. Is set as a cutout area (see FIG. 7B). Here, based on the sensor detection point sequence, an image region having a predetermined width H4 in the lower direction of the image from the sensor detection point sequence and a predetermined width H3 corresponding to a typical vehicle width in the left-right direction of the image is cut out. Set to area. Then, the image of the cutout area is cut out from the captured image. The predetermined width H4 is a constant value determined in advance based on a typical height of the bumper lower end height Bh. However, an estimated value obtained by estimating the bumper lower end height Bh from the captured image may be used. After the image is cut out, the image above the cutout region is shifted downward to connect the images (see FIG. 7C), and the connected images are subjected to viewpoint conversion to generate a bird's-eye view image ( (Refer FIG.7 (d)). The bird's-eye view image generated in this way is displayed on the display device 11. The distance calculation unit 23 and the image correction unit 24 function as a position estimation unit.

  Next, the processing procedure of the image correction process will be described with reference to the flowchart of FIG. This process is executed by the ECU 20 at predetermined intervals.

  In FIG. 8, in step S21, a captured image is acquired from the in-vehicle camera 10, and in step S22, object position information (in the present embodiment, object distance information) acquired by the ultrasonic sensor 12 is read from the memory. In step S23, the actual position of the object on the bird's-eye view image is estimated from the position information of the read object, and a cutout area is set in the image area below the estimated position. In subsequent step S24, the image of the set cutout region is cut out from the bird's-eye view image, and in step S25, the image above the cutout region is shifted downward to connect the images. Thereafter, in step S26, the corrected captured image is subjected to viewpoint conversion to generate a bird's-eye view image, and in subsequent step S27, the corrected bird's-eye view image is output to the display device 11 and the process is terminated.

(Other embodiments)
The present invention is not limited to the above embodiment, and may be implemented as follows, for example.

  In the above embodiment, the position information of the object detected by the ultrasonic sensor 12 is used to estimate the actual position of the object on the image, and the cutout area of the image is set. The actual position of the object may be estimated from the recognized image. For example, the actual position of the object on the image is estimated based on the size of the entire object or a part of the object on the image. Specifically, the correspondence between an image of a certain size as in a vehicle, an image of a specified size as in a license plate, and the distance to an object is determined in advance, and The distance from the host vehicle 50 is estimated from the size of the object at.

  When the in-vehicle camera 10 is a monocular camera, the actual position of the object on the image is estimated by calculating the coordinates of the object with respect to the own vehicle 50 using the history of coordinates accompanying the movement of the own vehicle 50. Also good. In addition, when the in-vehicle camera 10 includes a stereo camera, the actual position of the object on the image may be estimated based on a plurality of images captured by the stereo camera.

  In a configuration using the ultrasonic sensor 12 as a distance measuring sensor, the distance information acquired by the direct wave and the distance information acquired by the indirect wave are used, and the relative of the object with respect to the host vehicle 50 using the principle of triangulation A specific position (coordinates) may be calculated, and an image clipping region may be set using the coordinates. Alternatively, using the history of reflection points accompanying the movement of the host vehicle 50, the coordinates of the object relative to the host vehicle 50 may be calculated using the principle of triangulation, and the image clipping region may be set using the coordinates. .

  -It is good also as a structure which acquires the reflective surface information of an object based on the detection result of a ranging sensor, and sets the cutout area of an image using the acquired reflective surface information. According to such a configuration, even when the object is not directly facing the host vehicle 50, it is possible to perform appropriate display position correction according to the direction of the object. Specifically, when there is another vehicle 51 that is tilted obliquely with respect to the traveling direction of the host vehicle 50 in front of the host vehicle 50, the host vehicle in the other vehicle 51 is used by using the reflection surface information. The closest position from 50 is estimated, and an image clipping region is set based on the estimated position. For example, when the left end portion of the rear portion of the other vehicle 51 is located closest to the host vehicle 50, the image clipping region is set so that the left end portion is parallel to the rear bumper.

  In addition, in order to acquire the reflection surface information of an object using the detection result of the distance measuring sensor, a direct wave reflection point (hereinafter referred to as a direct reflection point) and an indirect wave reflection point (hereinafter referred to as an indirect reflection point). )) On the same plane, a surface model that detects surface components from a plurality of reflection points on the object surface is used. More specifically, in FIG. 9, there are a direct reflection point S and an indirect reflection point T on the surface of the object 60, and the first sensor 61a that receives the reflected wave as a direct wave and the direct reflection point S are arranged. The intersection of the straight line passing through and the straight line passing through the second sensor 61b that receives the reflected wave as an indirect wave and the indirect reflection point T is X. The intersection point X can be obtained from the measurement distance L1 using a direct wave and the measurement distance L2 using an indirect wave. That is, the direct reflection point S is a midpoint between the first sensor 61a and the intersection point X. Further, the indirect reflection point T exists on an intersecting straight line that passes through the direct reflection point S and extends in a direction that intersects the straight line connecting the direct reflection point S and the first sensor 61a at a predetermined angle. In this case, the direction of the object 60 can be grasped by acquiring the intersecting straight line indicating the direction of the plane component of the object surface as the reflection surface information of the object 60. Further, since the direction of the reflecting surface and the width of the reflecting surface can be grasped by the line segment ST on the intersecting straight line, the width in the left-right direction of the cutout region of the image can be set according to the width of the reflecting surface.

  -It is good also as a structure which sets the cutout area of an image to the peripheral area | region of the grounding part 54 based on the edge extracted from the picked-up image or the bird's-eye view image. Specifically, for example, the bumper lower end portion 55 of the other vehicle 51 and the ground contact portion 54 with the tire are detected on the captured image using the edge extracted from the captured image. Then, an image area from the bumper lower end 55 to the grounding section 54 is set as a cutout area, and an image of the set cutout area is cut out from the captured image.

  As another aspect, the bumper lower end 55 of the other vehicle 51 on the captured image is detected using the edge extracted from the captured image, and is shifted from the bumper lower end 55 by a predetermined width H5 in the lower direction of the image. The position may be estimated as the actual position of the object on the image. In this case, for example, an image area having a predetermined width H5 in the lower direction of the image and having a predetermined width corresponding to a typical vehicle width in the left-right direction of the image with respect to the position of the bumper lower end 55 is set as a cut-out area. Set.

  When the obstacle imaged by the in-vehicle camera 10 is a vehicle, it is possible to specify to some extent the size of the image display displacement due to the viewpoint conversion by the typical height of the bumper lower end height Bh. In view of this point, a cutout area having a predetermined width in the vertical direction of the image is set in a peripheral area of the ground contact portion 54 of the other vehicle 51, and the image of the set cutout area is obtained from a captured image or a bird's-eye view image. It is good also as a structure cut out. According to this configuration, it is possible to perform display position correction using only an image captured by the in-vehicle camera 10 without using a position estimation unit such as a distance measuring sensor. At this time, it is preferable that the vertical width of the image in the cutout region is set variably according to the vehicle type (for example, a normal vehicle and a large vehicle). Specifically, a vehicle with a larger bumper lower end height Bh may be configured to increase the vertical width of the image in the cutout region.

  -When connecting the remaining images after cutting, depending on the difference in brightness, the material of the road surface, etc., when the corrected image is displayed on the screen of the display device 11, the part where the images are connected is uncomfortable. May occur. For example, when there is a road surface entity such as a manhole around the ground contact portion with the road surface 53 of the other vehicle 51 and a part of the road surface entity is included in the cutout region, the remaining image after the cutout is displayed on the remaining image. Some road surface objects remain, and when displayed on the display device 11, an unnatural image may be formed. In view of these points, it is determined whether or not a sense of incongruity occurs on the display screen at the joined portion of the remaining images after cutting, and when it is determined that a sense of incongruity occurs, the predetermined area including the joined portion is determined. It is good also as a structure which correct | amends a luminance value. Specifically, with respect to the remaining image after clipping, it is determined whether or not there is a portion where the luminance value of the edge is changing rapidly in the joined portion of the image, When the determination is made, a process of averaging the luminance values of a predetermined area including the joined portion in a stepwise manner is performed.

  In the above embodiment, the ground surface is converted into a bird's-eye view image looking down in the vertical direction, and the converted bird's-eye view image is displayed on the display device 11. However, the ground surface is converted into a bird's-eye image looking down from above. The converted bird's-eye view image may be displayed on the display device 11. In this case, it is desirable to set the vertical width (H1, H3) of the cutout region of the image according to the angle α (<90 °) at which the ground surface is looked down obliquely from above. Specifically, when displaying an image in which the ground surface is looked down obliquely from above, the vertical width of the cutout region is set smaller than in the case of displaying an image in which the ground surface is looked down in the vertical direction.

  Based on parallax information obtained from a plurality of images photographed from different positions, an image clipping region is set in the peripheral region of the object grounding unit 54, and the image of the set clipping region is determined from the captured image or the bird's-eye image It is good also as a structure cut out. The distance to the object has a correlation with the parallax, and the farther the distance to the object, the smaller the parallax. By using the parallax information, it is possible to grasp the actual depth position of the object in the image . Specifically, for example, in a vehicle including a stereo camera as the in-vehicle camera 10, the parallax is calculated for each pixel from the first captured image and the second captured image captured by each of the two cameras. If the captured image includes an area composed of pixels having a larger parallax than the area below the image, the area is set as a cutout area. According to such a configuration, the cutout region for correcting the display shift can be grasped in units of pixels, so that the display position can be corrected more accurately. In addition, the cutout area of the image can be minimized, and a more natural image can be displayed. In the case of a monocular camera, the above configuration can be applied based on the principle of moving stereo.

  -It is good also as a structure which determines whether the object recognized with the vehicle-mounted camera 10 is a vehicle, and implements the image correction process for eliminating the display shift accompanying viewpoint conversion on the condition that it determined with the vehicle. When the above image correction processing is performed on all obstacles present near the host vehicle 50, there are many portions where the images are cut out and joined together, and the display image may be difficult to see. Therefore, by limiting the correction target to the vehicle, the other vehicle 51 can be displayed at an accurate position while suppressing the display image from being difficult to see.

  In the above embodiment, the cutout area of the image is set according to the width of the object. However, the area from the left end to the right end of the image may be cut out.

  In the above embodiment, the case where the in-vehicle camera 10 is mounted in front of the host vehicle 50 has been described. However, the mounting position of the in-vehicle camera 10 is not particularly limited, and is mounted on the rear or side of the host vehicle 50. You may apply to the vehicle-mounted camera 10 which exists.

  In the above embodiment, the ultrasonic sensor 12 is provided as a distance measuring sensor. However, the present invention may be applied to a configuration including a millimeter wave radar, a laser radar, or the like.

  -Each said component is conceptual and is not limited to the said embodiment. For example, the functions of one component may be realized by being distributed to a plurality of components, or the functions of a plurality of components may be realized by one component.

  DESCRIPTION OF SYMBOLS 10 ... Car-mounted camera, 11 ... Display apparatus, 12 ... Ultrasonic sensor, 20 ... ECU (periphery monitoring apparatus), 21 ... Image acquisition part, 22 ... Viewpoint conversion part, 23 ... Distance calculation part, 24 ... Image correction part, 50 ... own vehicle, 51 ... other vehicle, 54 ... grounding section.

Claims (11)

The present invention is applied to a vehicle (50) equipped with a camera (10) having an imaging area around the vehicle and a display device (11) for displaying an image, and a captured image captured by the camera is placed above the vehicle. A perimeter monitoring device that converts a virtual viewpoint from a set virtual viewpoint into a bird's eye view seen on the road surface and displays the image on the display device,
An image of a peripheral area of the ground contact portion (54) between the object on the road surface and the road surface is cut out from the photographed image or the bird's-eye view image, and the remaining images after the cut-out are connected to be displayed on the display device. A periphery monitoring device comprising: an image correcting unit that performs correction for aligning the position of the object on the image with the actual position of the object. A position estimation unit for estimating an actual position of the object on an image;
The image correction unit sets an image clipping region in a peripheral region of the grounding unit based on the position estimated by the position estimating unit, and clips an image of the set clipping region from the captured image or the bird's-eye image The periphery monitoring device according to claim 1. A distance measuring sensor that transmits a search wave and receives a reflected wave of the search wave is provided in the vehicle,
The periphery monitoring device according to claim 2, wherein the position estimation unit estimates an actual position of the object based on a detection result of the distance measuring sensor. The position estimation unit estimates the actual position of the object by obtaining reflection surface information of the object based on a detection result of the distance measuring sensor,
The periphery monitoring device according to claim 3, wherein the image correction unit sets the cutout region based on reflection surface information of the object. The object is a vehicle;
The periphery monitoring device according to claim 2, wherein the position estimation unit estimates an actual position of the object on the image based on a size of the whole or a part of the object on the image.   The image correction unit sets a cutout region having a predetermined width in the vertical direction of the image in a peripheral region of the grounding unit, and cuts out the image of the set cutout region from the captured image or the bird's-eye view image The periphery monitoring device according to claim 1.   The image correction unit sets an image clipping region in a peripheral region of the grounding unit based on parallax information obtained from a plurality of images captured from different positions, and sets the image of the set clipping region as the captured image or The periphery monitoring device according to claim 1, wherein the periphery monitoring device is cut out from the bird's-eye view image.   The image correction unit determines whether or not a sense of incongruity occurs on the image displayed on the display device in the joined part of the remaining images, and when it is determined that a sense of incongruity occurs, the joined part is determined. The periphery monitoring device according to any one of claims 1 to 7, wherein a luminance value of a predetermined area including the correction is corrected.   The periphery monitoring device according to any one of claims 1 to 8, wherein the image correction unit sets an image clipping region according to a width of the object.   The periphery monitoring device according to claim 1, wherein the image correction unit sets an image clipping region according to an angle at which the road surface is looked down from above the virtual viewpoint. The present invention is applied to a vehicle (50) equipped with a camera (10) having an imaging area around the vehicle and a display device (11) for displaying an image, and a captured image captured by the camera is placed above the vehicle. A peripheral monitoring method for converting a virtual viewpoint set to a bird's-eye image viewed on a road surface and displaying the image on the display device,
An image of a peripheral area of the ground contact portion (54) between the object on the road surface and the road surface is cut out from the photographed image or the bird's-eye view image, and the remaining images after the cut-out are connected to be displayed on the display device. A periphery monitoring method, comprising: correcting the position of the object on an image to be matched with the actual position of the object.