JP2010193198A - Object bottom end position detection device and object bottom end position detection program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide technology capable of easily calculating the position of an object in an image of a top view generated from a photographed image by a projective transformation, especially the position of an object that does not touch the ground. <P>SOLUTION: This object bottom end detection device includes: a first straight line group calculating part 3a for calculating a first straight line group comprising straight lines for connecting respective characteristic points for characterizing an object shape of an object in a first conversion image of a top view based on a first position and an optical center point; a second straight line group calculating part 3b for calculating a second straight line group comprising straight lines for connecting respective characteristic points for characterizing an object shape in a second conversion image of a top view based on a second position and the optical center point; an intersection calculating part 4 for calculating intersections between the first straight line group and the second straight line group relatively displaced in accordance with distance between the first position and the second position; and an object bottom end calculating part 5 for calculating an object bottom end position on the basis of a distribution position of the intersections. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a technique for calculating the position of an object in an upper-view bird's-eye image generated by projective transformation from a photographed image photographed with a specific region as a subject, particularly an object that is not in contact with the ground.

  A viewpoint conversion process is known in which image processing is performed on a photographed image by a camera to convert the photographed image into an image viewed from a viewpoint different from the viewpoint of the camera. In this viewpoint conversion processing, an image captured by a camera mounted on a vehicle is converted into an overhead image that looks down on the periphery of the vehicle from above by moving the viewpoint upward, and the display inside the vehicle It is used for applications such as In the viewpoint conversion processing, image processing is generally performed on the assumption that all objects in an image exist on the same plane even if the objects are solid objects. For example, in the use in a vehicle, all objects are subjected to conversion processing as patterns existing on the road surface, like road markings and parking frame lines. Road markings and the like that actually exist on the road surface are converted into an image as seen from directly above without any sense of incongruity by linear conversion. However, the height of an object (that is, a three-dimensional object) such as another vehicle or an obstacle greatly distorts the image, and it is difficult to grasp the distance between the vehicle and the object, particularly the object that is not in contact with the ground.

  In view of such a problem, for example, Japanese Patent Application Laid-Open No. 2006-333909 (Patent Document 1) describes from a camera to an object in an actual image based on actual images captured at a plurality of different viewpoint positions by the camera. The distance is calculated, the vertical position of the object in the real image on the display screen, the distance from the camera to the object obtained by the calculation, the vertical position on the display screen of the real image and the real camera Based on the ground data indicating the relationship with the distance, it is determined whether or not the object has a height. If the object has a height, the object is perpendicular to the ground in the real image. A technique for creating a bird's-eye view image by compressing in a direction is disclosed. Furthermore, in this patent document 1, in order to display an object that is not in contact with the ground at a more accurate position on a bird's-eye view image, a vertical direction indicating a vertical direction in actual image data captured by a camera is disclosed. Data is created in advance and stored in a memory or the like, and when the image conversion processing unit compresses image data composed of pixels copied to the calculation memory at an arbitrary rate, it is grounded using vertical data. A technique is also disclosed in which an unoccupied object is moved in a direction perpendicular to the ground and compressed at a position touching the ground. The vertical data that needs to be obtained and stored in advance is the vertical on the ground, which is uniquely determined based on the optical characteristics of the camera lens of the camera that constitutes the camera, and the mounting position and mounting angle with respect to the vehicle. It is shown. Accordingly, it is necessary to create and store another vertical data even if one of the parameters of the camera lens optical characteristics and the mounting posture changes.

JP 2006-333209 A (paragraph numbers 0007 to 0008, 0011 to 0032, FIGS. 5 to 17 and the like)

  In the technique according to Patent Document 1 described above, in order to calculate the position of the object in the overhead view of the upper viewpoint generated by projective transformation from the photographed image, a calculation process for obtaining a distance from the camera to the object or a previously obtained ground A determination process for determining whether or not an object has a height using data is essential. Also, in order to calculate the ground contact position of an object that is not in contact with the ground, the object that is not in contact with the ground is moved in a direction perpendicular to the ground using vertical data that must be obtained in advance. Processing is required.

  The present invention was devised in view of the above circumstances, and can easily calculate the position of an object in an upper viewpoint image generated by projective transformation from a captured image, particularly an object that is not in contact with the ground. The purpose is to provide.

  In order to achieve the above object, a characteristic configuration of the object lower end position detection device according to the present invention includes: a first image receiving unit that receives a captured image captured at a first position as a first image with a specific region as a subject; A second image receiving unit that receives a captured image captured at a second position different from the first position as a second image; and a first projective conversion unit that performs projective conversion of the first image into a first converted image viewed from an upper viewpoint. A second projective transformation unit that projectively transforms the second image into a second transformed image viewed from above, and each feature point characterizing the object shape of the object in the first transformed image and the optical center point A first straight line group calculating unit for calculating a first straight line group composed of straight lines, and a second straight line group composed of straight lines connecting the feature points characterizing the object shape in the second converted image and the optical center point. Two straight line group calculation sections; An intersection point calculation unit for calculating an intersection point of the straight line groups generated when the first straight line group and the second straight line group, which are relatively displaced so as to correspond to the distance between the first position and the second position, are overlapped. And an object lower end calculating unit that calculates the lower end of the object based on the distribution position of the intersection.

  According to this feature configuration, when a feature point that characterizes the object shape of an object in each converted image is first detected using some object detection image processing algorithm such as density conversion or differentiation processing, the feature is detected in each converted image. A first straight line group and a second straight line group composed of straight lines connecting the points and the optical center point are calculated. Furthermore, the intersection of the straight line group of the first straight line group and the second straight line group that are relatively displaced according to the distance (in some cases, the displacement amount) between the first position and the second position is calculated, and these intersection points are calculated. The lower end of the object is calculated based on the distribution position. This intersection is a projection point on the horizontal plane (road surface) of the feature point of the object. If noise is taken into consideration, the point where the intersections are concentrated may be determined as the projection position of the object on the horizontal plane (road surface). When the object is in contact with the ground, the position of the gathering of the intersection points indicates the contact position of the object with the ground (the lower end position of the object). In addition, when the object is not touching the road surface, such as a bumper of an automobile, the contact position when the object is extended vertically on the road surface (in this case, this position is regarded as the lower end position of the object). Can be determined. The principle of detecting the lower end position of the object will be described in detail later, but the basic point of focus is the intersection of the straight lines obtained before and after the movement in a straight line passing through the optical center point and a certain point on the object. This means that the point represents a point projected on the road surface. For example, projective transformation of a photographed image obtained at a certain optical center point A first straight line passing through a predetermined point in the space and the optical center point in the bird's-eye image, and a new projective transformation obtained by moving the optical center point A second straight line passing through a predetermined point in the space in the bird's-eye view image and the optical center point after the movement intersects at a position immediately below the predetermined point. Therefore, the point where the intersections of the first line and the second line defined by many feature points belonging to the object are concentrated represents the lower end position of the object. This principle is utilized in the present invention. In other words, in the present invention, the bird's-eye view image of the upper viewpoint generated by the projective transformation from the photographed image only by the simple image processing technique of obtaining the intersection of the line connecting the feature point of the object and the optical center point in each of the two images. It is possible to easily calculate the position of an object in FIG.

  In the object lower end position detection device according to the present invention, the first straight line group calculation unit and the second straight line group calculation unit may perform horizontal differentiation that enhances the outline of the first converted image and the second converted image. It is preferable that the feature points are obtained from an image generated by thresholding after applying a filter. The straight line used for detecting the lower end of the object is a straight line connecting the optical center point and the image feature point indicating the lower ridge line of the object. Therefore, by emphasizing the edge component in the vertical direction by the horizontal differential filter and performing binarization processing using an appropriate threshold value, for example, many noise image areas are excluded, so that the image feature point indicating the lower ridge line Becomes easier to obtain. In addition, since the straight line connecting the optical center point and the image feature point is a straight line that radiates upward from the optical center point, the straight line group calculation processing load is determined by selecting the straight line using such a constraint condition. Is reduced.

  In the object lower end position detection device according to the present invention, the first image receiving unit determines that the vehicle has reached the first position based on a detection result of a movement detection unit that detects a movement amount of the vehicle. The second image receiving unit receives the captured image taken when the vehicle has reached the second position based on a detection result of a movement detection unit that detects a movement amount of the vehicle. The intersection image calculation unit obtains the movement distance of the optical center point between the first image and the second image from the detection result of the movement detection unit. It is preferable. According to this configuration, captured images before and after the movement of the optical center point when the optical center point moves by an appropriate distance can be obtained, so that subsequent object lower end position detection processing is facilitated.

  In order to achieve the above object, an object lower end position detection program according to the present invention receives, as a first image, a captured image taken at a first position with a specific region as a subject, and a second position different from the first position. A function of receiving a photographed image taken in step 2 as a second image, and projective transformation of the first image into a first converted image viewed from an upper viewpoint, and a second converted image viewed from the upper viewpoint A first line group consisting of straight lines connecting the feature points characterizing the object shape of the object in the first converted image and the optical center point, and calculating the object shape in the second converted image. A function for calculating a second straight line group composed of straight lines connecting the feature points and the optical center point, and a relative value corresponding to the distance between the first position and the second position. A function of calculating intersections of corresponding straight lines in the first straight line group and the second straight line group generated when the first converted image and the second converted image that are positioned are superimposed, and distribution of the cross points The computer realizes the function of calculating the lower end position of the object based on the position. This object lower end position detection program can also obtain the operations and effects described in the object lower end position detection apparatus described above.

Explanatory drawing explaining the principle of the object lower end position detection technique by this invention Explanatory drawing explaining projective transformation Explanatory drawing explaining the principle of object bottom position detection technology Explanatory drawing explaining the principle of object bottom position detection technology Explanatory drawing explaining the principle of object bottom position detection technology Explanatory drawing explaining the principle of object bottom position detection technology Flow chart showing object bottom position detection algorithm Illustration of the front of the driver's seat of the vehicle Block diagram showing the basic configuration of the vehicle Explanatory drawing showing an example of viewpoint conversion An example of an image taken by a camera having a viewpoint from a substantially horizontal direction An example of an image converted into a viewpoint image from a substantially vertical direction An example of a photographed image taken in a substantially horizontal direction An example of a GPT image obtained by converting the viewpoint of the photographed image of FIG. Explanatory drawing showing the three-dimensional object extraction principle Explanatory drawing which shows the example which extracts the area | region in which a solid object exists from the GPT image obtained from the picked-up image as shown in FIG. The block diagram which shows the structural example of a bird's-eye image generation apparatus typically

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The object lower end position detection technology of the present invention is incorporated in a driving support apparatus that performs driving support using a bird's-eye view image obtained by performing viewpoint conversion (projection conversion) on an image captured by an in-vehicle camera.

First, the basic principle of this object lower end position detection technique will be described with reference to FIGS.
Compared to objects that are in contact with the road surface, such as car stops placed on the road, objects that are overhanging, such as the bumper part of a car, have no entity between them. The position directly under the overhanging object (in the present invention, this position is also referred to as the object lower end position together with the road surface object contact position) cannot be clearly shown in the bird's-eye view image. The object lower end position detection technique is to detect the object lower end position in such a bird's-eye view image, and to be able to clearly indicate the position as necessary.

  FIG. 1 shows a point A as one feature point of an object in space, and a contact point C on the road surface, which is a point on which the point A is projected onto the road surface (which is a position directly below the overhang portion). The relationship between the optical center point O and the projection point A1 of the point A onto the road surface is shown in the form of a side view. FIG. 2 shows how the point A and the ground contact point C are displayed in the captured image and the bird's-eye view image (projection-transformed image on the horizontal plane). FIG. 3 shows a straight line connecting the optical center point O and the point A before and after the movement when the optical center point (viewpoint) O of the camera moves by ΔL with respect to the point A in the form of a bird's eye view. Here, O1 and O2 indicate the optical center point O before and after the movement, and A1 and A2 indicate the projection points on the road surface of the point A before and after the movement. From FIG. 3, the straight line passing through the optical center point O and the point A in the space exists at different positions on the bird's eye view coordinate system (XZ plane), and the straight line in the coordinates before and after the movement is the point A. It can be understood that they intersect at a position immediately below. The distance between the viewpoint and the bottom of the object (feature point A) is indicated by d. Naturally, since the shape of the object is defined by a large number of feature points A, FIG. 4 shows a diagram corresponding to FIG. 3 when there are a large number (two in this case) of feature points. As is clear from FIG. 4, when the optical center point moves with respect to the feature point group defining the object, the straight line passing through the optical center point and the feature point group existing in the space has a different position on the bird's eye view coordinate system. The straight lines in the coordinates before and after the movement intersect at a position immediately below the object in the space. In addition, in the case of feature point groups on the same object, the point intersections immediately below the object are concentrated at the intersection corresponding to the lower end position according to the shape of the object. Such an example is shown. Of course, there are many intersections of straight lines calculated from the actual image, but intersections that do not represent the lower end position are also included as noise, so it is necessary to select highly reliable intersections from the many intersections. In order to solve this problem, it is simple and effective to use the fact that the intersections of the straight line groups are densely located at the position corresponding to the lower end position of the object feature points even if the intersections that are noises are included. It is. For example, when a distribution of intersection groups as shown in FIG. 6 is obtained, a likely intersection concentration position is estimated from this distribution, and the intersection concentration position may be regarded as the lower end position of the object.

FIG. 7 shows a flowchart of the algorithm of the basic principle of the object bottom position detection technique according to the present invention described above. The processing flow is as follows.
Step # 01
A first image (captured image) is captured at the first position, and a second image (captured image is acquired at the second position. At this time, two cameras having different optical center points are simultaneously captured and first captured. One image and the second image may be acquired, or the first image and the second image may be acquired by photographing each before and after the movement of the viewpoint with one camera.
Step # 02
A first converted image (bird's eye image) obtained by projective transformation from the first image and a second converted image (bird's eye image) obtained by projective transformation from the second image are generated.
Step # 03
In order to clearly indicate the vertical shape feature points of the object in the image with respect to the generated first and second converted images, a binary image obtained by extracting vertical edges is generated. This is because the normal optical center point is located on the lower side of the image, and the vertical shape of the object is important in order to obtain an effective straight line connecting the optical center point and the object feature point. A horizontal differential filter, such as a vertical Sobel filter, is preferably applied to extract the vertical edges. The threshold value used in the binarization process may be a fixed value when the density characteristics of the acquired image are adjusted in advance, but in other cases, the threshold value is calculated based on the histogram of the image. Is preferred. By this process, pixel groups other than the pixel group toward the optical center point direction are removed, and only a straight line passing through pixels (feature points) included in the pixel group toward the optical center point direction is calculated as much as possible. The processing burden can be reduced.
Step # 04
In each obtained binarized image, a straight line group (first straight line group and second straight line group) connecting the feature point included in the pixel group indicating the object shape feature in the vertical direction and the optical center point. Is calculated. In this case, it is preferable to use a method in which one point is set as the origin and the other point is randomly selected and a predetermined number of straight lines are obtained by applying a feature point to the virtual line and obtaining a satisfactory result. For example, for this purpose, it is possible to apply a subspace to the trajectory of feature points using RANSAC (RANdom SAmple Consensus), and to take a measure of removing pixels with a large residual by threshold processing.
Step # 05
Between the first line group obtained from the first converted image and the second line group obtained from the second converted image, the intersection point is calculated as shown in FIGS. The selection to be processed from the straight line group between the straight line groups can be performed on the condition that the azimuth angles are similar (from the smallest angle difference to the predetermined order), but the selection is difficult. In case, it is brute force.
Step # 06
The lower end position of the object is calculated from the calculated intersection group. For example, when the distribution of intersections as shown in FIG. 6 is obtained, when about 10 to several hundred feature points of an object are detected, the distribution of intersections is concentrated near the true position and Gaussian distributed. However, when there are few patterns on the object, many feature points cannot be obtained and the number of intersections is expected to be small. In any case, the least median method (LMedS) is used to estimate the probable intersection concentration position, several samples are extracted at random, and applied to the least squares method, and the square error at all intersections is repeated. The estimation when the median of is the smallest should be regarded as the correct estimation.

  8 and 9 show the basic configuration of the vehicle 30 on which the bird's-eye image generation device that employs the object bottom edge detection technology that operates based on the above-described principle is mounted. The steering 24 provided in the driver's seat is interlocked with the power steering unit 33 and transmits the rotational operation force to the front wheels 28f to steer the vehicle 30. An engine 32 and a speed change mechanism 34 having a torque converter, CVT, and the like that shift the power from the engine 32 and transmit the power to the front wheels 28f and the rear wheels 28r are arranged at the front of the vehicle body. Power is transmitted to the front wheel 28f and / or the rear wheel 28r according to the driving method of the vehicle 30 (front wheel drive, rear wheel drive, four wheel drive). In the vicinity of the driver's seat, an accelerator pedal 26 as an accelerator operating means for controlling the traveling speed, and a brake pedal 27 for applying a braking force to the front wheels 28f and the rear wheels 28r via the brake devices 31 for the front wheels 28f and the rear wheels 28r, Are arranged in parallel.

  A monitor 20 is provided at the upper position of the console near the driver's seat. The monitor 20 is a liquid crystal type equipped with a backlight. The monitor 20 is also formed with a pressure-sensitive or electrostatic touch panel, and is configured to receive an instruction input by a user by outputting a contact position of a finger or the like as location data. The touch panel is used, for example, for inputting instructions for starting parking assistance. The monitor 20 is also provided with a speaker, which can emit various guidance messages and sound effects. When the navigation system is mounted on the vehicle 30, it is preferable that the monitor 20 is also used as a display device for the navigation system. The monitor 20 may be a plasma display type or an organic EL display type, and the speaker may be provided in another place such as the inside of the door.

  A steering sensor 14 is provided in the operation system of the steering 24, and the steering operation direction and the operation amount are measured. The operation system of the shift lever 25 is provided with a shift position sensor 15 to determine the shift position. The operation system of the accelerator pedal 26 is provided with an accelerator sensor 16 to measure an operation amount. The operation system of the brake pedal 27 is provided with a brake sensor 17, which detects the presence or absence of operation.

  In addition, a rotation sensor 18 that measures the amount of rotation of at least one of the front wheel 28f and the rear wheel 28r is provided as a movement distance sensor. In this embodiment, the case where the rotation sensor 18 is provided in the rear wheel 28r is illustrated. As for the movement distance, the transmission mechanism 34 may measure the movement amount of the vehicle 30 from the rotation amount of the drive system. The vehicle 30 is provided with an electronic control unit (ECU) 10 serving as the core of the bird's-eye image generation apparatus, and an object lower end detection program according to the present invention is also incorporated in a computer constituting the ECU 10.

  A camera 12 that captures a scene behind the vehicle 30 is provided at the rear of the vehicle 30. The camera 12 is a digital camera that incorporates an image sensor such as a charge coupled device (CCD) or a CMOS image sensor (CIS) and outputs information captured by the image sensor in real time as moving image information. The camera 12 includes a wide-angle lens and has a field angle of about 140 degrees on the left and right, for example. The camera 12 has a substantially horizontal viewpoint and is installed so that a scene behind the vehicle 30 can be photographed. The camera 12 is installed with a depression angle of, for example, about 30 degrees toward the rear of the vehicle 30 and photographs a region up to about 8 m behind. The captured image is input to the ECU 10.

  As shown in FIG. 10, the bird's-eye image generation device converts an image shot by the camera 12 having a substantially horizontal viewpoint into an image shot by a virtual camera 12A having a viewpoint looking down from a substantially vertical direction. To do. FIG. 11 is an example of an image photographed by the camera 12. FIG. 5 is an example of an image converted into an image photographed by a virtual camera 12A having a viewpoint looking down from a substantially vertical direction. This converted image is a so-called bird's-eye view image. In FIG. 12, blank areas are generated at the lower left and lower right due to the absence of image data.

  First, various methods are known for specifying a three-dimensional object when generating a bird's-eye view image. Here, a method using a difference between two images with parallax is employed. In this method, first, a GPT (ground plane transformation) image as a kind of bird's-eye view image that is a projected transformation image to the upper viewpoint is generated from one of the two images with parallax, and the viewpoint of the other image is generated. The GPT image from is predicted, and the position of the generated GPT image is corrected. Next, when a GPT image is generated from the other image and a difference from the GPT image generated from the one image and subjected to position correction is taken, a three-dimensional object is extracted. The two images that generate parallax may be images taken by a plurality of different cameras or images taken by moving the same camera. In this embodiment, the movement of the same camera The previous and next images are used.

  FIG. 13 is an example of a photographed image photographed from the camera 12 in a substantially horizontal direction. FIG. 14 is an example of a GPT image obtained by projective transformation (viewpoint transformation) of the captured image of FIG. As shown in FIG. 13, a substantially rectangular solid object 40 exists around the vehicle 30. It is distinguished whether the three-dimensional object 40 is an object that is in contact with the road surface, or an object that is overhanging from a support body (vehicle body) that is in contact with the road surface, such as an automobile bumper, and that is not in contact with the road surface. Can not. Therefore, in the case of an overhanging object, the lower end position of the object (position where it intersects the road surface) cannot be grasped from the GPT image. For this reason, in this bird's-eye view image generation device, the lower end position is obtained based on the above-described object lower end position detection technique and is shown in the GPT image.

  FIG. 15 is an explanatory diagram showing the principle of three-dimensional object extraction. In FIG. 15, the three-dimensional object on the GPT image is shown as a trapezoid for convenience. In FIG. 15, a trapezoid a is a three-dimensional object 40 in the GPT image of one of the two captured images in which parallax is generated. The trapezoid a indicates the three-dimensional object 40 viewed from the upper viewpoint before the camera 12 moves.

  The trapezoid b is the three-dimensional object 40 in the first converted image obtained by projective conversion of the captured image (first image) at the first position of the two captured images where parallax occurs. That is, the trapezoid b shows the three-dimensional object 40 in the second converted image obtained by projective conversion of the captured image (second image) at the second position where the camera 20 has moved by ΔL. The trapezoid a 'in the hatched portion indicates the three-dimensional object 40 obtained by predicting the case where the camera 20 has moved by ΔL from the first position based on the first converted image. When the difference between the trapezoid b and the trapezoid a 'is taken, the pixel remaining as the difference corresponds to the position where the three-dimensional object 40 that causes parallax exists. Thereby, the area where the three-dimensional object 40 exists is extracted.

  FIG. 16 is an explanatory diagram illustrating an example in which a region where the three-dimensional object 40 exists is extracted from the GPT image obtained from the photographed image as illustrated in FIG. 13. As described based on FIG. 15, the difference between the GPT image obtained by performing the position correction of ΔL on the GPT image before the camera 12 moves and the GPT image after the camera 12 moved by ΔL is obtained. A difference image S is obtained. When an edge is extracted from the difference image S, a trapezoidal region having four sides e1 to e4 is extracted as a three-dimensional object region R as shown in FIG.

  Here, the side e3 and the side e4 correspond to the upper base and the lower base of the trapezoid. The side e1 and the side e2 are two sides other than the upper and lower bases of the trapezoid, that is, sides corresponding to the legs. The side e1 and the side e2 intersect at the optical center point C. That is, the three-dimensional object region R is extracted in a trapezoidal shape that intersects at the optical center C when the two non-parallel sides e1 and e2 are looked down from a substantially vertical viewpoint. In the above example, since the three-dimensional object 40 is a rectangular parallelepiped, the three-dimensional object region R and the three-dimensional object 40 substantially coincide with each other. However, even if the three-dimensional object 40 is not a rectangular parallelepiped but has an arbitrary shape, the trapezoidal three-dimensional object region R is extracted. Of the four sides e1, e2, e3, and e4 that define the three-dimensional object region R, the side e2 indicating the lower end position of the object is used in the above-described object lower end detection technique according to the present invention if the object is an overhanging object. Will be corrected on the basis. That is, when the object lower end position obtained based on the object lower end detection technique is different from the position of the side e2, the object lower end position based on the object lower end detection technique is prioritized.

  FIG. 17 is a block diagram schematically showing a configuration example of a bird's-eye image generation device (ECU 10) incorporating the object lower end position detection device according to the present invention. As shown in FIG. 17, the bird's-eye image generation device includes a first image receiving unit 1a, a second image receiving unit 1b, a first projective conversion unit 2a, a second projective conversion unit 2b, and a first straight line group detection. Unit 3a, second line group detection unit 3b, intersection calculation unit 4, object lower end calculation unit 5, three-dimensional object region extraction unit 6, projective distortion correction unit 7, own vehicle position calculation 8, and superposition unit 9 and the image control unit 11. The ECU 10 is configured by, for example, a microcomputer, and each of the functional units can share the function with a program or the like. Accordingly, the functional units do not need to be physically provided independently, and it is sufficient if the functions are realized by cooperating with software such as a program using the same hardware.

  The first image receiving unit 1a is a functional unit that receives, as a first image, a photographed image photographed at a first position by a camera (vehicle-mounted camera) 12 that photographs the periphery of the vehicle 30. The second image receiving unit 1b is a functional unit that receives, as a second image, a captured image captured by the camera 12 at a second position where the vehicle 30 has passed a predetermined movement amount (ΔL) from the position at which the first image was captured. . The image control unit 11 is a functional unit that controls the timing at which the first image receiving unit 1a and the second image receiving unit 1b receive a captured image.

  The first projective conversion unit 2a is a functional unit that performs projective conversion of the first image into a first converted image from an upper viewpoint from a substantially vertical direction, and the first projective conversion unit 2b converts the second image from a substantially vertical direction. It is a functional part which carries out projective conversion to the 2nd conversion image from an upper viewpoint.

  The first straight line group calculation unit 3a is a functional unit that calculates a first straight line group composed of straight lines connecting the feature points that characterize the object shape of the object in the first converted image and the optical center point. The calculating unit 3b is a functional unit that calculates a second straight line group including straight lines connecting the feature points characterizing the object shape in the second converted image and the optical center point.

  The intersection calculation unit 4 calculates a straight line group generated when the first straight line group and the second straight line group, which are relatively displaced so as to correspond to the displacement amount of the first position and the second position, are overlapped. It is a functional unit that calculates intersection points. The intersection calculation unit 4 uses straight lines as a plurality of vertical edges included in the three-dimensional object, and these straight lines are obtained without taking correspondence between the azimuth straight line groups on the two planes obtained before and after the movement. It is configured to calculate the intersection between groups. However, the intersection calculation unit 4 takes the correspondence in a predetermined condition between the straight lines based on the image information on the image through which each azimuth line passes in the azimuth line group on the two planes obtained before and after the movement. The intersection between corresponding straight lines may be calculated. Alternatively, both of these algorithms may be provided and selectively used. Further, the object lower end calculation unit 5 identifies the position where the intersections are concentrated as the object position, that is, the lower end position of the object, by analyzing the distribution state of the intersection group obtained by the intersection calculation unit 4.

  As described above, the three-dimensional object region extraction unit 6 takes the difference between the first converted image and the second converted image B that have been subjected to the position correction of ΔL, and based on this difference, the three-dimensional object 40 is present. This is a functional unit that extracts the region R into a trapezoid shape in which two non-parallel sides (e1 and e2) intersect at the optical center C.

  The projective distortion correction unit 7 is a functional unit that reduces the image of the three-dimensional object region R in the second converted image B to the direction of the optical center C when looking down from a substantially vertical viewpoint and corrects it to a projection distortion corrected image. . The superimposing unit 9 performs mask processing on an area other than the area where the image of the reduced three-dimensional object 40 exists in the three-dimensional object area R on the second converted image, or notifies the presence of the three-dimensional object 40. It is a functional part that superimposes.

  The own vehicle position calculation unit 8 receives the detection results of various sensors such as the steering sensor 14, the shift position sensor 15, the accelerator sensor 16, the brake sensor 17, and the rotation sensor 18 and calculates the movement amount and position of the vehicle 30. Part. The various sensors 14 to 18 and the vehicle position calculation unit 8 correspond to the movement detection unit 13 of the present invention. The vehicle position calculation unit 8 may be included in an ECU, a control device, or the like different from the ECU 10 and only the calculation result may be input to the ECU 10.

  The superimposing unit 9 performs a mask process on a region other than the region where the image of the reduced three-dimensional object 40 exists in the three-dimensional object region R on the monitor display image, or displays a notification display for notifying the presence of the three-dimensional object 40. It is a functional part that is superimposed.

  As can be understood from the above description, each functional unit included in the object lower end position detecting device that detects a contact position of an object that is not in direct contact with a road surface such as an overhanging object is, in this embodiment, the first functional unit. Intersection calculation of image receiving unit 1a, second image receiving unit 1b, first projection conversion unit 2a, second projection conversion unit 2b, first straight line group detection unit 3a, and second straight line group detection unit 3b A unit 4 and an object lower end calculating unit 5.

The relationship between the processing of each step in the object lower end position detection algorithm shown in FIG. 7 and each functional unit of the object lower end position detection apparatus is as follows.
The process of step # 01 for acquiring the first image and the second image is executed by the first image receiving unit 1a and the second image receiving unit 1b. The process of step # 02 for generating the first converted image and the second image, which are bird's-eye images, is executed by the first projection conversion unit 2a and the second projection conversion unit 2b. The processing of step # 03 for detecting object feature points (feature areas) by edge enhancement or binarized image generation is executed as preprocessing in the first straight line group detection unit 3a and the second straight line group detection unit 3b. The process of step # 04 for calculating the first straight line group and the second straight line group connecting the object feature point and the optical center point is executed by the first straight line group detection unit 3a and the second straight line group detection unit 3b. . The process of step # 05 for calculating the intersection of the corresponding lines between the first line group and the second line group is executed by the intersection calculation unit 4. The object bottom end calculation unit 5 executes the process of step # 06 in which the intersection indicating the lower end position of the object is estimated from the calculated intersection point group and the lower end position of the object is calculated. Here, the functions required for the object lower end position detection algorithm are only divided into functional units for easy understanding, and the function execution units can be arbitrarily assigned.

  As described above, a specific area such as the periphery of the vehicle is used as a subject, and for example, an image captured by an in-vehicle camera is not displayed as it is, so that the user can easily grasp the sense of distance between the three-dimensional object and the vehicle. In addition, when a bird's-eye view image is generated by projective transformation and displayed, even if the object is not in direct contact with the road surface, such as an overhanging object, the road surface can be detected by using the object bottom position detection technology according to the present invention. The lower end position of the object, which is the contact position with the object, can be detected, and a display that gives the user a sense of the correct distance from the object can be performed.

[Other Embodiments]
(1) The straight line group (first straight line group and second straight line group) calculation process of step # 04 and the intersection calculation process of step # 05 in the algorithm of the basic principle of the object lower end position detection technique described with reference to FIG. Can be performed by the processing described below.
1. A first converted image and a second converted image, which are GPT images, are generated from the first image and the second image.
2. Apply vertical Sobel filter processing to both the above images to suppress the horizontal component, and then obtain a pixel group subjected to threshold processing (peaks may be detected for intensity above the noise level). The purpose of this processing is to reduce the number of iterations (to ensure responsiveness), that is, to remove outliers other than the pixel group toward the optical center.
3. Although there are usually a plurality of pixel groups obtained above, straight line fitting is performed by applying a constraint condition passing through the optical center position to each pixel group to obtain an azimuth line. For linear fitting, RANSAC that does not require grouping or noise removal as preprocessing is used. Basically, straight line fitting with two samples is used. Here, one point is selected at random from the origin, and the other is randomly selected from the pixel group. Assuming that there are a plurality of pixel groups, the consensus obtains n straight lines from the top.
4. A second straight line group (n lines) from the second converted image (GPT image) and a first straight line group (m lines) from the first converted image (GPT image) are obtained.
5. The straight line equation of the first straight line group from the first converted image is converted into a coordinate system in which the second straight line group from the second converted image is defined, and the first straight line converted with the second straight line group Calculate the intersection with the group.
It is conceivable to calculate the intersection point between straight lines that have similar azimuth angles (the smallest angle difference, insurance: and then the smallest), but the line detected at another location on the object Since the slopes may be more similar, the intersection calculation can be a round-robin battle.
Since this brute force method is inefficient, when a plurality of azimuth line groups are obtained, efficiency is achieved by the following method.
(a) A candidate region for a three-dimensional object is extracted from a difference image between the first converted image and the second converted image.
(b) A azimuth line group existing in the candidate area is detected. The second straight line group (n lines) and the first straight line group (n lines) are obtained.
(c) Since both straight line groups are arranged in the order of arrangement of the vertical edges unique to the three-dimensional object, it can be considered that the number of straight lines and the arrangement order are almost the same between the images before and after the movement.
(d) Label each straight line in azimuth order (numbering). One line is selected from the straight line from the second converted image in numerical order. By selecting a straight line from the second converted image corresponding to the number and a predetermined number of straight lines before and after that, an intersection point between a straight line from the second straight line group and a part of the first straight line group is calculated.
(e) This method can greatly reduce the amount of calculation compared to a complete round-robin battle. The intersection of the straight line group is detected at a position on the road surface directly below the point in the space of the overhanging three-dimensional object.

(2) Various preferred methods of the minimum median method (LMedS), which is one of the robust estimation methods effective in the lower end position calculation process in Step # 06 in the algorithm of the basic principle of the object lower end position detection technique described with reference to FIG. A suitable application method is described below.
(a) Basic method
1. Randomly select one from all data (if the number is small, you can do exhaustive search) and use this as a temporary average.
2. Calculate the squares of the residuals with the other samples with respect to the tentative mean value and arrange them in ascending order.
3. Check the value with the square of the residual being the mth largest (= 1 = 2 of the number of samples: Median) and record this as the Median evaluation value.
4. Repeat steps 1 and 2 above, and assume that the tentative average value when the Median evaluation value is minimized is a rough model representing the group.
5. The final precision model is obtained by taking the average value of the sample point group whose residual square from the provisional average value is within the minimum Median evaluation value.
(b) Simple method
1. Find the average of all samples
2. Calculate the residual squares with the mean and arrange them in ascending order
3. Check the value with the square of the residual being the mth largest (= 1 = 2: Median number of samples)
4. Remove samples whose residual square is greater than the median value
5. Use the average value of the remaining samples as the final precision model.
(c) Improved method from (a)
When processing with the basic method shown in (a), unless the sample point group has a steep distribution, multiple positions exhibiting the minimum Median evaluation value occur, and the median value of the distribution cannot be indicated. A phenomenon can occur. In order to detect the median value of the distribution even if it is a gentle distribution or a distribution that does not have a trapezoidal peak, the square of the residual is not used as it is, but the accumulated value is used.
1. Randomly select one from all data (if the number is small, you can do exhaustive search) and use this as a temporary average.
2. Calculate the squares of the residuals with the other samples with respect to the tentative mean value and arrange them in ascending order.
3. Calculate the cumulative value (cumulative value of the residual square) up to the mth of the residual squared value (= 1/2 of the number of samples: Median), and record this as the Median evaluation value.
4. Repeat steps 1 and 2 above, and assume that the tentative average value when the Median evaluation value is minimized is a rough model representing the group.
5. The final precision model is obtained by taking the average value of the sample point group whose residual square from the provisional average value is within the minimum Median evaluation value. If the sample has a plateau-like distribution, the median value cannot be detected only by the above accumulation method. In such a case, when a plurality of minimum Median evaluation values based on the cumulative value are generated, an average value for the generated position is output as a median value.

1a: first image receiving unit 1b: second image receiving unit 2a: first projection conversion unit 2b: second projection conversion unit 3a: first straight line group calculation unit 3b: second straight line group calculation unit 4: intersection point group calculation unit 5: Object lower end calculation unit 6: Solid object region extraction unit 7: Projection distortion correction unit 9: Superimposition unit 10: ECU (bird's-eye view image generation device incorporating an object lower end position detection device)
12: Camera (vehicle camera)
13: Movement detector 30: Vehicle 40: Solid object

Claims (4)

A first image receiving unit for receiving, as a first image, a photographed image photographed at a first position with a specific region as a subject;
A second image receiving unit that receives a captured image captured at a second position different from the first position as a second image;
A first projective transformation unit for projectively transforming the first image into a first transformed image viewed from an upper viewpoint;
A second projective transformation unit that projectively transforms the second image into a second transformed image viewed from above.
A first straight line group calculating unit for calculating a first straight line group composed of straight lines connecting the feature points characterizing the object shape of the object in the first converted image and the optical center point;
A second straight line group calculating unit for calculating a second straight line group composed of straight lines connecting the feature points characterizing the object shape in the second converted image and the optical center point;
An intersection point calculation unit for calculating an intersection point of the straight line groups generated when the first straight line group and the second straight line group, which are relatively displaced so as to correspond to the distance between the first position and the second position, are overlapped. When,
An object bottom end calculating unit for calculating a bottom end position of the object based on a distribution position of the intersection;
An object lower end position detecting device having   The first straight line group calculating unit and the second straight line group calculating unit are generated by applying a threshold value process after applying a horizontal differential filter that enhances an outline to the first converted image and the second converted image. The object lower end position detection apparatus according to claim 1, wherein the feature point is obtained from the obtained image.   The first image receiving unit captures a captured image captured when it is determined that the vehicle has reached the first position based on a detection result of a movement detection unit that detects a movement amount of the vehicle. And the second image receiving unit captures a captured image captured when it is determined that the vehicle has reached the second position based on a detection result of a movement detection unit that detects a movement amount of the vehicle. 3. The object lower end position detection device according to claim 1, wherein the intersection calculation unit obtains the movement distance of the optical center point between the first image and the second image from the detection result of the movement detection unit. . A function of receiving a photographed image photographed at a first position with a specific region as a subject as a first image, and receiving a photographed image photographed at a second position different from the first position as a second image;
A function of projectively converting the first image to a first converted image viewed from an upper viewpoint, and a projective conversion of the second image to a second converted image viewed from an upper viewpoint;
Calculating a first straight line group consisting of straight lines connecting the feature points characterizing the object shape of the object in the first converted image and the optical center point, and characterizing the object shapes in the second converted image; A function for calculating a second straight line group composed of straight lines connecting the optical center points;
The first straight line group and the second straight line generated when the first converted image and the second converted image that are relatively displaced so as to correspond to the distance between the first position and the second position are superimposed. A function for calculating the intersection of corresponding straight lines with the group;
A function of calculating a lower end position of the object based on a distribution position of the intersection;
Is a computer program for detecting the lower end position of an object.

Images