KR20080054094A - Method for object recognizing and distance measuring - Google Patents

Abstract

A method for object recognition and distance measurement is provided to discriminate a solid body and a plane and to prevent erroneous responses to characters or patterns on a road surface by using a plurality of cameras. A method for object recognition and distance measurement includes the steps of: photographing an object by using a plurality of photographing devices(3) installed in a vehicle(1); performing homography transformation by performing projective transformation on the images through a homography matrix and position matching between the images; performing stereo processing by extracting a solid candidate and calculating the corresponding positions of the images to the extracted solid candidate; and determining a position of an intersection point after discriminating whether the photographed object is a planar or solid body.

Description

Method for object recognizing and distance measuring}

1 is a view showing the configuration of the distance measuring device according to the present invention and the installation state of each component;

2 is a view for explaining a first calibration in the present invention,

3 is a view for explaining a second calibration in the present invention,

4 is a view showing an original image scratched with a plurality of scratch apparatus in the present invention,

5A and 5B are diagrams showing examples of images corrected in the present invention;

6 is a view for explaining the extraction of height information in the present invention,

7 is a view for explaining three-dimensional object detection in the present invention,

8 is a view for explaining intersection detection in the present invention,

9 is a diagram for explaining a time series image.

<Explanation of symbols for main parts of the drawings>

1: vehicle 2: measuring device

3: Image pickup device 4: Object recognition / distance measurement unit

5: Notification control unit 6: Lighting device

The present invention relates to a method for object recognition and distance measurement, and more particularly, to an object recognition and distance measurement method that can be used for vehicle driving assistance or automatic control in an automobile or the like.

Today, environmental awareness (e.g., environmental maps, course prediction, movement control) in industrial machinery, recreational toys, etc. or vehicle driving assistance or automatic control (e.g. obstacles, pedestrian detection, pre-crash safety) For pre-crash safety, etc.), a lot of research is being conducted on devices or methods that can accurately perform object recognition and distance measurement.

In particular, in recent automobile fields, a system for recognizing an object in front of the vehicle, controlling the speed of the vehicle, or performing a hazard avoidance has been put into practical use, and the most important thing in the construction and operation of such a system is precise and rapid. It can be called object recognition and distance measurement.

For object recognition and distance measurement, sensors that monitor specific directions are used.

However, even when the sensor is used, in case of an object popping out suddenly from a shade or the like and a falling object falls suddenly from the air, the time is limited until the control is performed to respond to the target object, and the risk is avoided within a given time. It is true that there are many difficulties in carrying out and completing the control of.

In addition, even in the case of a system using image recognition technology, a road marking drawn on a road surface is often mistaken as an obstacle, and in reality, a system capable of identifying an obstacle for all images has not been developed.

For accurate object recognition, a technique for identifying an object from the shaded information of the image is required.

Accordingly, the present invention has been invented to solve the above problems, by using the information of the sensor to monitor the front while driving in the vehicle to detect the place where the object may suddenly pop out (intersection, etc.) to the driver It is an object of the present invention to provide a method of object recognition and distance measurement to call attention or to perform automatic control such as speed control and danger avoidance in the case of mechanical control so that accidents can be prevented in advance.

In addition, the present invention by using a plurality of cameras to identify three-dimensional objects and planes, and to prevent incorrect reaction to letters or patterns, such as road signs drawn on the road surface, through which a system having accurate object identification and more flexibility can be built The object is to provide a method for object recognition and distance measurement.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1 is a view showing a configuration of a distance measuring device according to the present invention and an installation state of each component, and showing an embodiment of a distance measuring device mounted on a vehicle.

As shown in the drawing, the measuring device 2 of the present invention mounted on the vehicle 1 includes a plurality of imaging devices 3, an object recognition / distance measuring unit 4, a notification control unit 5, and an illumination device 6 It is configured to include).

The imaging device 3 includes a lens having a predetermined image angle, and a plurality of imaging devices are installed toward a specific direction of the vehicle. In the illustrated example, the plurality of imaging devices 3 display an image in front of the vehicle. It is installed so that it can image.

The image pickup device 3 picks up a front image at a predetermined period, for example, at a rate of 30 frames per second, picks up a time-series image, that is, a moving image, and outputs the picked-up moving image.

As the imaging device 3, cameras using devices such as CMOS, CCD, and tube systems can be used without limiting the method of the imaging device, and any device capable of generating an image can be used regardless of the type. .

However, it is preferable that semiconductor devices such as CMOS and CCD can be used while being usable in an outdoor environment and mounted in a vehicle.

In addition, the imaging device 3 may be capable of capturing an image of the front side at a constant period (a constant period is preferable but may be a non-stationary period), and outputs a moving image having no change in time series.

Next, the object recognition / distance measuring unit 4 sequentially receives images of the time series captured by the imaging device 3 and outputted therefrom, measures the distance of an object existing in front of the vehicle, and then generates a signal according to the measurement result. This is the component that outputs.

In addition, the measurement signal output by the object recognition / distance measurement unit 4 is transmitted to the notification control unit 5, which notifies the driver or the crew in the vehicle of the received measurement result or automatically. It is a component for controlling the running state of the vehicle.

In addition, the illumination device 6 is a device that illuminates the field of view of the imaging device 3 by irradiating light including near infrared rays to the front, and the imaging device 3 photographs an object, for example, toward the front of the vehicle. If the position can be irradiated with light, it may be mounted on any part, and a separate lighting device using a head lamp or a fog lamp or the like is provided.

In the present invention, distance measurement refers to the action of a human being recognizing the distance (including height) to an object existing in the outer world through a visual system, that is, when the unknown input pattern is recognized by both eyes. Refers to a computer performing a series of acts of evaluating the similarity with the standard pattern and recognizing the distance between the input pattern by considering the standard pattern of the highest similarity and the parallax of both eyes.

In general, the imaging device generates and outputs a sequence (video) of still images of several tens of frames per second (30 frames per second in NTSC), and the processing apparatus processes the images of the still images one by one. It is not necessary to record (record) this image, and it is sufficient to have a storage area (ring buffer or the like) in which only the time required for processing can refer to the past image.

However, the imaging device may be a real-time imaging of a subject, but is not limited thereto.

For example, a video camera (recorder) may reproduce and output a subject previously photographed or recorded, but the accuracy is increased when the photographing times of the plurality of imaging devices are completely coincident with each other.

This is because when there is a temporal shift in the image to be imaged, it is impossible to determine whether the object has moved or if there is a parallax.

Here, "time series" is interpreted to mean that a still image is correctly continuous on the time axis at a fixed interval, for example, NTSC (30 frames / second).

Hereinafter, the object recognition and the distance measuring process of the present invention using the above-described distance measuring device will be described in detail.

1. Homography Transform

A plurality of imaging devices 3 mounted forward on the top of the vehicle 1 pick up an image including the subject in front of the vehicle at a constant or irregular period and output a time series image, that is, a moving image.

Then, the object recognition / distance measuring unit 4 receives the time series images output from the imaging device in turn, and the viewpoint conversion means in the object recognition / distance measurement unit converts the viewpoint to the image based on the positional relationship between the imaging apparatuses. Do this.

In the viewpoint transformation process, a homography matrix is formed by using an image obtained by capturing a correction pattern, and a spatial position relationship between a plurality of imaging apparatuses, that is, a plurality of cameras, is used to perform projective transformation on an image using the homography matrix. The positional relationship between the picked-up images is matched.

Here, the correction pattern will be described as a square lattice pattern used for correction of optical lens distortion of the camera, difference in image pickup position, etc., and may be corrected semi-automatically by visual correction or image processing. Some people use directionality such as board patterns or QR codes.

In addition, when the projection transformation is explained, this is a kind of geometric transformation in image processing, which is the theory of changing the coordinates of pixels, and performing the transformation by blocking the image projected from a point at a desired output plane by rearranging the pixels.

Then, the three-dimensional object candidate extracting means extracts the three-dimensional object candidate included in the image, and the corresponding position recognition means calculates the corresponding positions of the plurality of images with respect to the extracted three-dimensional object candidate.

The corresponding position recognizing means performs stereo processing by block matching to calculate the distance from the image corresponding to the positional relationship by the viewpoint conversion means to the photographed object based on the parallax according to the distance.

Then, the plane recognition means determines the three-dimensional object from the corresponding position recognition candidate, and whether the object photographed using the distance image output by the corresponding position recognition means is an object belonging to the plane or an object three-dimensionally present from the plane. Will be identified.

Then, the output means determines and outputs the intersection position from the three-dimensional object determination result, aggregates and recognizes the object recognized as the plane and the object recognized as the three-dimensional object, and outputs the area determined as the intersection point.

In the case where a plurality of cameras are provided in a vehicle as the imaging device, a slight deviation may occur between the left and right cameras, and the homogeneous deviation is corrected on the image and the parallax between the left and right cameras becomes zero. The transformation is performed using a Homography matrix.

The homography matrix is a matrix for calculating positions or postures between cameras when an image is taken using a plurality of cameras, and a transformation for equalizing a positional relationship between cameras using the homography matrix is called a homography transformation. Call.

By performing such a homography conversion, strict installation conditions of the camera can be alleviated, and since the coordinates between the cameras can be common, only parallax can be extracted.

The homography matrix defines the positional relationship of the left and right cameras, and the value is fixed when the camera is fixed.

Therefore, calibration should be performed when the arrangement, spacing, and fixing method of the camera are changed.

1-1. Calibration 1

As the first calibration, correction is performed so that left and right disparities occur in accordance with the distance, similarly to the human visual characteristics.

As shown in FIG. 2, the homography conversion is corrected so that the pair of cameras is perpendicular to the plane provided at a distance in front of the camera.

This uses a homography matrix, where the parallax of the left and right correspond to the distance from the camera to the object.

1-2. Calibration 2

As in the first calibration, the same calibration is performed using the plane where the vehicle exists (here, ground).

In this case, it is performed using a road surface (horizontal plane) in which the plane performed by the first calibration is changed by 90 ° forward. As shown in FIG. 3, the parallax between the left and right cameras includes a depth distance direction and a height direction. An image in which parallax becomes zero with respect to the area can be obtained.

In the case of measuring the height of an object, it is a common method to install the camera in a ceiling or the like, and to measure the height as a distance after installing the camera in the shape of a distance from the front, but using the homography matrix corrected on the road surface, the left and right cameras By calculating the parallax between images, the composite distance of "distance + height" including both the height of the object and the distance to the object can be calculated pseudo-wise.

1-3. Stereo Processing

In order to measure the distance to the picked-up object, the distance is calculated using a BlockMatching process which detects a corresponding point between the images from the parallax image obtained through the calibration process.

Here, two images with parallax are converted into distance images in which the distance information is expressed in shades of light.

The distance images obtained in the above procedure become the two images of FIGS. 5A and 5B attached according to the respective calibration methods.

As shown in FIG. 5A in which the front of the camera pair is calibrated in this manner, an image according to the distance in front of the camera is output, and in FIG. 5B in which the road surface is corrected, only a portion (three-dimensional object having a height) is output.

In this way, it is possible to detect three-dimensional objects existing at a different height from the plane where the vehicle exists (here, the road surface), and thus it is possible to apply the vehicle to automatic control or obstacle avoidance.

1-4. Stereoscopic object detection processing

By combining stereo distance images using the two types of correction, the height and distance of an object existing in the image can be extracted separately.

As a result, as shown in the following equation (1) and FIG. 6, only the height information is extracted by obtaining the difference between the image of the first correction having a depth distance and the image of the second correction including the height and the depth.

This is equal to the difference of the distance image calculated by the stereo processing.

(1)

(One)

By using the above technique, it is possible to make a detailed determination by adding the height of the object to the conventional depth distance, and to prevent the object not having a height, such as a road mark or a white line, displayed on a road surface, to be detected, and having a three-dimensional structure having a height. It is possible to build a system that only detects.

1-5. Intersection detection processing

For the detection of the intersection points, the positional relationship between the plurality of imaging devices is used, and two images (Fig. 5A) in which the parallax is zero at the front of the imaging device and two images (Fig. 5B) in which the parallax is zero for the ground area are shown. I use it.

An intersection point is a plane that exists on the same plane as its own car path, but where a pedestrian, vehicle, etc. exist in a direction intersecting the car path direction in an area that is blinded by a building such as a building. Point to the intersection of.

First, a three-dimensional object detection process is used to detect an area different in height from the road surface existing in the image.

This is equivalent to taking out only the height information in FIG. 5B, and detecting a three-dimensional object having a different height from the road surface (see FIG. 7 attached).

In addition, it searches for a radial straight line from the front to the infinite origin (the point where the line converges. In general, the point of convergence of the white line on the road) from the front, and the road area exists farther than the three-dimensional structure such as a building. In this case, it is determined that the road area exists behind the three-dimensional structure (side road, T-shape, intersection point), and the position is called an intersection point possible area (see FIG. 8 attached).

By determining the distance existing in the intersection possible area by the depth distance image, the notification control unit 5 warns, notifies, or controls the driver, so that an accident or the like can be avoided in advance.

As described above, in the present invention, a pedestrian position candidate that satisfies a predetermined condition is recognized from a plurality of images including time-series consecutive subjects, and a still object and a person are discriminated. When applied in combination with an automatic control device between vehicles, it is possible to avoid danger, reduce damage, support driving, and call attention. In addition, unlike the existing system, since it is a system for discriminating an object from both sides of a street image and a shaded image, problems such as the case of late control may be solved even if the detection has been a problem in the practical application of environmental recognition technology. Application to the system is possible.

Claims (4)

Imaging an image including a subject using a plurality of imaging devices mounted on the vehicle; Projection transformation is performed on the images picked up from the plurality of imaging devices using a homography matrix created to correspond to the spatial positional relationship between the plurality of imaging devices using the image picked up the calibration pattern, and the positional relationship between the picked up images is obtained. Performing a corresponding homography transform; Extracting the three-dimensional object candidate included in the image and calculating the corresponding positions of the plurality of images with respect to the extracted three-dimensional object candidate, and calculating the distance from the image corresponding to the position relation to the photographed object based on the parallax according to the distance. Performing stereo processing; Determining a three-dimensional object from a corresponding position recognition candidate, and identifying an intersection point after identifying whether the photographed object is an object (a road surface area) or a three-dimensional solid object by using the stereo processed distance image; Object recognition and distance measurement method comprising a. The method according to claim 1, The performing of the homography transformation may be performed by using the homography matrix to correct the image to be perpendicular to the photographed front plane while generating parallax between left and right according to distance with respect to the images photographed from the plurality of imaging devices. A first correction step for performing the correction and a road surface in which the plane made in the first correction step is changed by 90 ° by using the homography matrix to obtain an image having zero parallax for the ground area; 2 calibration steps, In the stereo processing, the object recognition and the distance measuring method are characterized by obtaining a distance image in which the distance information is expressed using a block matching process for detecting a corresponding point between the images from the parallax image obtained through the correction process. . The method according to claim 2, The process of identifying whether the photographed object is an object belonging to a plane or a three-dimensional object existing in three dimensions by using the stereo processed distance image, Characterized in that the image difference between the image having the distance information processed from the image of the first correction and the image having the height and distance information processed from the image of the second correction is obtained to identify the three-dimensional object having the height from the road surface. Object recognition and distance measurement method. The method according to claim 1, The determining of the intersection point location, A radial straight line directed toward the infinite origin in front is searched from the front in front, and when the road area exists farther than the identified solid object different from the road area, it is determined as the intersection position where the road area exists behind the three-dimensional object. Object recognition and distance measuring method characterized in that.

Images