JP2002366932A - Device and method for detecting object - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device that surely detects a high object by applying a method for photographing a monitoring range with two cameras, performing projection conversion so as to make the position of pixels on a reference plane of one image equal to the position of pixels on a reference plane of the other and detecting an area having height with respect to the reference plane. SOLUTION: An object detecting device 10 consists of two image inputting parts 12 and 14, a reference image generating part 16, a projection conversion image generating part 18, an image comparing part 20, a reliability image generating part 22, a mask image storing part 24 and an object structure detecting 26, decides the existence (existence/non-existence) of an object by using information of an area having high responsibility in such a manner that a projection difference image is converted into a multi-level reliability image to be utilized, and prevents an error of a detection position with respect to an object position in such a manner that information of an area having low reliability and the information of an area having high reliability are combined to be utilized.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus for processing an image photographed by a stereo camera.
3D distance measurement, 3D object reconstruction, 3D object recognition,
The present invention relates to various image processing apparatuses such as a traffic monitoring system, an intruder monitoring system, and a front / rear / side obstacle detection apparatus using an in-vehicle camera, and a method thereof.

[0002]

2. Description of the Related Art Various image processing methods have been proposed which take a stereo image as input, and perform three-dimensional measurement, three-dimensional object recognition, traffic monitoring, intruder monitoring, analysis of a moving trajectory of a person, detection of an obstacle ahead, and the like. .

Problems such as image processing for traffic monitoring and detection of a forward obstacle using a camera mounted on an automobile are as follows.
It can be formulated as “detection of a tall object existing in a planar background such as a road surface”.

In response to this problem, reference 1 (Kazunori Onoguchi:
As described in Japanese Patent Application Laid-Open No. H11-328365, "image surveillance apparatus and method"), first, a road surface is set as a reference plane, and a monitoring range is photographed by two cameras. A method has been proposed in which a region having a height relative to the reference plane is detected by projecting and transforming the image so as to be equal to the position of a pixel on the reference plane of the image and comparing the two images.

[0005] Further, the method of Reference 1 will be described in detail.

First, the image photographed by the first camera is projected and transformed so that the road plane of the image photographed by the second camera coincides with the road plane of the image photographed by the second camera.

Next, the difference between the two images whose road surface positions match by projection transformation is calculated. Then, an area on the road surface or an area without height relative to the road surface (for example,
Since the positions of shadows, white lines, and the like match, the luminance of the image of the object is canceled out and the residual becomes zero. On the other hand, an image of an object having a height with respect to the road surface is displaced, so that a residual appears.

When this difference image is binarized, only a region having a height appears as a residual.

[0009] However, when the method of Reference 1 is applied to an image photographed in an actual environment, the residual region includes not only a region having a height but also some error regions. When an error region is included, the error region is erroneously recognized as a region having a height. as a result,
False recognition occurs in both the traffic monitoring system and the detection of obstacles ahead.

[0010] The reason for the occurrence of this error region is caused by an error in the projection conversion coefficient, distortion of the lens, or the fact that the road surface is not strictly flat but has a bale shape. Ideally, points on the road surface that should be perfectly matched by the projection transformation are slightly shifted due to these causes, and an error region appears. If there is a slight shift, an error region appears in, for example, the outline of a white line, the outline of a shadow of a building, or the outline of a shadow of a vehicle.

In order to remove the error region, it is possible to use, for example, a method for removing a small residual region or an elongated residual region. Alternatively, it is possible to use a method of dividing the image into small blocks, finely shifting the small blocks, finding the position where the residual is minimized, and compensating for the small positional deviation.

However, when such error regions are arranged, there arises a problem that not only the error region but also a part of a necessary residual region is removed at the same time. For example, when the method of Reference 1 is applied to a traffic monitoring system, the rear surface and the ceiling surface of a dark truck with poor texture have little change between the projected and converted image and the original image. When error region removal processing is performed there,
An area where such a residual hardly appears is deleted at the same time as the removal of the error area, which may cause detection omission.

As described above, in the residual image including the error region, there is a problem that the error region is erroneously recognized as an object having a height and a problem that the detection position of the object is erroneously affected by the error region.

On the other hand, in the residual image from which the error region has been deleted,
There is a problem that an obstacle where a residual does not easily appear or a target to be monitored is missed, or a part of a necessary error area is missing, so that a detection position is erroneously detected.

[0015]

As described above,
A reference plane is defined within the monitoring range, the monitoring range is photographed by two cameras, and the projection is transformed so that the position of the pixel on the reference plane of one image is equal to the position of the pixel on the reference plane of the other image. In the method of detecting a region having a height relative to the reference plane, there is a problem that an error region appears due to an error in a projection conversion coefficient, lens distortion, or the strictness of the reference plane.

If an error region exists, there is a problem that the error region is erroneously recognized as an object having a height, and a problem that the detection position is wrong.

Further, when the error region is deleted, there is a problem that some obstacles or monitoring targets are missed, or the detection position is incorrect.

Therefore, according to the present invention, a reference plane is defined within a monitoring range, the monitoring range is photographed by two cameras, and the position of the pixel on the reference plane of one image is set to the position of the pixel on the reference plane of the other image. Provided is an object detection device that reliably detects a tall object by applying a method of projecting and transforming to be equal and detecting a region having a height with respect to a reference plane, and a method thereof.

[0019]

According to a first aspect of the present invention, an object having a height relative to a reference plane constituting a common background of both images is obtained from a reference image having a common field of view and a comparative image. In the object detection device to detect, the reference image and the comparison image, a pair of image input means to input respectively,
Projection conversion means for projecting and converting a position of a pixel in the comparison image to a position of a corresponding pixel in the reference image with reference to the reference plane to generate a projection conversion image; Image comparison means for generating, from the reference image and the reference image, a projection difference image in which the region of the object appears as a residual; and three or more multiplicity indicating reliability when the object is detected based on the projection difference image. A reliability image generating means for generating a reliability image composed of pixels of a value, and an object structure detection for detecting presence / absence, type, or position of the object based on information of multi-valued pixels of the reliability image And an object detection device.

According to a second aspect of the present invention, in the reliability image generating means, the multi-valued pixel is a class representing a region having no height, a class representing a region having low reliability of having a height, 2. The object detection device according to claim 1, wherein the object detection device is classified into at least three types of classes representing regions having high reliability of existence.

According to a third aspect of the present invention, the multi-valued pixel is further classified into a class representing a non-applicability in the object detection processing. It is.

According to a fourth aspect of the present invention, in the object structure detecting means, the presence / absence of the object is determined by using information of a class representing a region with high reliability, and the position of the object is determined with high reliability. 2. The object detecting apparatus according to claim 1, wherein the determination is performed by simultaneously using the information of the class representing the region and the information of the class representing the region with low reliability.

According to a fifth aspect of the present invention, in the object structure detecting means, a projection image model of the object expected to appear in the projection difference image is matched with the reliability image, and 2. The method according to claim 1, wherein the presence, the type, or the position of the object is detected.
An object detection device according to any one of the preceding claims.

According to a sixth aspect of the present invention, when the object structure detecting means matches a projection image model of the object expected to appear in the projection difference image with the reliability image, the projection image model And counting the number of the classified pixels in a portion overlapping the area of the classified pixels of the reliability image, based on the count number of each class, the presence or absence of the object, the type, 3. The object detecting device according to claim 2, wherein the position is determined.

According to a seventh aspect of the present invention, in the object structure detecting means, the projection image model includes an upper area, a lower area,
It is configured of at least a rear adjacent area, and determines the presence or absence of the object based on the lower area and the count number of each class, and determines the position of the object based on the rear adjacent area and the count number of each class. The object detection device according to claim 5, wherein

According to an eighth aspect of the present invention, there is provided an object detection method for detecting, from a reference image and a comparison image having a common field of view, an object having a height with respect to a reference plane constituting a common background of the two images. A pair of image input steps for inputting the reference image and the comparison image, respectively, and projecting and converting the positions of the pixels in the comparison image to the positions of the corresponding pixels in the reference image with reference to the reference plane. A projection conversion step of generating a projection conversion image by using the projection conversion image and the reference image, an image comparison step of generating a projection difference image in which the region of the object appears as a residual from the projection conversion image and the reference image, and the projection difference image A reliability image generation step of generating a reliability image composed of three or more multi-valued pixels indicating the reliability at the time of detecting the object based on the reliability image; Presence of the object based on the information of the pixel, types, or a object detection method and a object structure detection step for detecting a position.

According to a ninth aspect of the present invention, there is provided an object detection method for detecting, from a reference image having a common field of view and a comparison image, an object having a height with respect to a reference plane constituting a common background of the two images. In a program realized by a computer, the reference image and the comparison image are respectively input to a pair of image input functions, and a position of a pixel in the comparison image with respect to the reference plane is set to a corresponding pixel in the reference image. A projection conversion function of generating a projection conversion image by performing projection conversion to a position of an image, and an image comparison function of generating a projection difference image in which the region of the object appears as a residual from the projection-converted projection conversion image and the reference image. And 3 indicating the reliability at the time of detecting the object based on the projection difference image.
A reliability image generation function of generating a reliability image composed of the above-described multi-valued pixels, and detecting presence / absence, type, or position of the object based on information of the multi-valued pixels of the reliability image. A program for an object detection method, wherein the object structure detection function is realized by a computer.

According to the present invention, when comparing the projection-converted comparative image with the reference image, the differential image of the two images is converted into a multi-valued reliability image instead of being binarized and processed. Process.

As a result, for example, the information on the area obtained by binarizing the difference image and the information on the area remaining after the error area has been deleted can be processed simultaneously as one image. Was. Further, by performing multi-value processing, different pixel values can be given to pixels that have been subjected to other processing such as masking processing. Thus, it has become possible to integrate a lot of information into one image at the same time.

By using the multi-valued reliability image, the existence (presence / absence) of the object can be determined by using the information of the “region where the reliability of presence of the object is high”. It is possible to prevent an area having a low degree of certainty from being erroneously recognized as an object.

[0031] Even in the case of information on an area having a high reliability with a high height, a region having a high reliability with a height is included in an area, and an area with a high reliability having a height is included. By judging the continuity of the target, it is possible to prevent detection omission even for a target in which a residual hardly appears.

The detection of the position of the object is prevented by using the information of the area of low reliability of the presence of the height and the information of the reliability of the presence of the height in combination. be able to.

As described above, it is possible to correctly determine the existence (presence or absence) of an object existing in the monitoring range, and it is possible to enhance the detection accuracy of the position of the object.

By utilizing this result, it is possible to improve the performance of detecting an obstacle ahead of a vehicle-mounted camera, and to improve the performance of detecting an object such as a traffic monitoring system or an intruder monitoring system. Will be possible.

[0035]

DESCRIPTION OF THE PREFERRED EMBODIMENTS One embodiment of the present invention will be described below with reference to the drawings.

(Embodiment 1) FIG. 1 is a basic configuration diagram of an object detection apparatus 10 showing an embodiment of the present invention.

FIG. 3 is a schematic diagram of a traffic monitoring system, which is one of the embodiments of the object detecting device 10. FIG.
Shows an input image of the first embodiment and an image in the process of processing.

As shown in FIG. 1, the object detection device 10
Two image input units 12, 14, a reference image generation unit 16, a projection conversion image generation unit 18, an image comparison unit 20, a reliability image generation unit 22, a mask image storage unit 24, and an object structure detection unit 26
It is composed of In addition. Each of these devices realizes functions described below by a program stored in a computer.

The outline of the function of each of these devices will be described step by step.

Image input units 12 and 14, reference image generation unit 1
6. The projection conversion image generation unit 18 and the image comparison unit 20 generate a projection difference image or a residual image by the planar projection stereo method (first-stage processing).

The reliability image generation unit 22 and the mask image storage unit 24 generate a reliability image (second stage processing).

The object structure detecting section 26 detects a vehicle by a matching process (third stage process).

(1) First-Step Processing (1-1) Role of Each Device As described above, the first-step processing generates a projection difference image or a residual image by the planar projection stereo method.

As shown in FIG. 3, the image input units 12 and 14 are fixed to a frame installed over a road.
Images are input from each of the cameras.

The reference image generator 16 receives image data from the image input unit 12 connected to the left camera, and uses this image as a reference image. This reference image is used for the planar projection stereo method.

The projection conversion image generation unit 18 receives image data from the image input unit 14 connected to the right camera, and generates a projection conversion image from this image for use in the planar projection stereo method.

The image comparing section 20 generates a projection difference image and a residual image by using the reference image and the projection conversion image by the planar projection stereo method described below.

(1-2) Description of Planar Projection Stereo Method First, the "planar projection stereo method" which is an important principle used in the present invention will be described.

As shown in FIG. 3, a fixed camera system is constructed by arranging two cameras side by side so as to obtain parallax. One is a reference image (FIG. 5: upper left), and the other is a comparative image (FIG. 5: upper right). In this embodiment, the image of the left camera is used as a reference image, and the image of the right camera is used as a comparison image.

FIG. 4 shows the projection relationship of the camera. At this time, the projection relationship of each camera is expressed by the following equation using a homogeneous coordinate system.

[0051]

(Equation 1)

(Equation 2) The transformation matrix T of the equation (2) can be solved by sampling the coordinates of the corresponding point on the road surface and using the least squares method. Transformation matrix T
Is determined by the relative position between the two cameras. In a system for monitoring such as a traffic monitoring system, since the camera position is fixed, the transformation matrix T is also fixed. Further, even in the case of an in-vehicle image processing system, since the distance between the two cameras is fixed, the transformation matrix T is also fixed. Therefore, the transformation matrix T may be obtained when the camera is installed.

The relational expression (2) holds only for points existing on the same plane as the road surface, and does not hold for points not existing on the same plane as the road surface.

For all pixels of the input comparison image, (2)
Performs projection transformation of the expression. This image is called a projection conversion image (FIG. 5: lower left). Since the relationship of the expression (2) holds only at points on the road surface, an object having a height relative to the road surface is converted to a position different from a corresponding point in the reference image.
Here, the difference between the reference image and the projection conversion image is obtained. This processing is called “projection difference processing”, and the generated image is called “projection difference image” (FIG. 5: lower right).

In the projection difference image, a region having a height appears as a residual. On the other hand, objects on the road surface such as shadows have no residual because the luminance is offset. The projection difference image is binarized using a threshold value. The binarized image is called a residual image. An area having a height relative to the road surface appears in the residual image.

The above processing method is referred to as "planar projection stereo method".
Call.

(2) Second-Step Processing After the above-described first-step processing, the second-step processing of generating a reliability image is performed by the reliability-image generating unit 20. Hereinafter, the processing will be described in order.

(2-1) Necessity of Matching First, before describing the process of generating a reliability image, the necessity of the matching process, which is the third stage process, will be described. Explain that images are useful.

Although an area having a height appears in the residual image,
The shape of the object in the residual image is a deformed image different from a normal projected image. In general, the higher the area, the less satisfied the relationship of the expression (2), and thus the higher the area, the laterally shifted in the projected image. Therefore, in the projection difference image,
The higher the height of the object, the wider the shape becomes.

For example, in the image of the traffic monitoring system, the vehicle is projected in a shape spread horizontally. To spread sideways,
If there are a plurality of vehicles in the image, they may overlap each other. Therefore, when the vehicle is individually cut or the position is recognized, a desired result may not be obtained simply by binarizing the projection difference image and performing the labeling process.

In the example of the traffic monitoring system, it is common that a large number of vehicles are present in an image, and it is necessary to correctly detect the position of each vehicle for the purpose of the system. In the case of an obstacle detection system, it is necessary to detect the position of an obstacle in an image. That is, in general, when a system is configured, it is not enough to simply detect a region having a height by the planar projection stereo method, and the vehicle region is identified from the shape of the region shown in the residual image or the projection difference image. Processing is required.

In order to identify the vehicle region, the shape of the vehicle in the residual image or the projection difference image is determined by:
It is effective to retain the model and identify the model by matching it with a residual image or a projection difference image.

When considering application to a traffic monitoring system, it is considered that attention should be paid to the vehicle detection stage.

The reason is that as the vehicle travels farther, the vehicles overlap earlier than an image using a monocular camera. Regarding vehicle detection, the planar projection stereo method is advantageous because there is no false detection of a shadow or the like, but the tracking ability of the vehicle is disadvantageous compared to the image processing of a monocular camera.

Therefore, the present invention takes advantage of the high detection capability of the vehicle to take charge of a portion for quickly and surely correctly detecting the vehicle that has entered the image. Then, after the apparatus of the present invention, a system configuration in which the vehicle area detected by the image processing apparatus of the present invention is tracked using an image processing system of a monocular camera (for example, an area tracking apparatus using normalized correlation). Take. By doing so, it is possible to detect the movement locus of the vehicle in the image.

With this system configuration, in the matching process, it suffices if the process of “detecting the vehicle area that has entered the image most recently” can be executed. In other words, even if many vehicles are photographed in the depth direction, it is only necessary to detect only one vehicle at the forefront, which is the largest image. Matching processing with such conditions is simplified and can be realized by the procedure described below.

Assuming a matching process, it is effective to generate an intermediate image suitable for the matching process instead of simply obtaining a binarized residual image from the projection difference image.

Therefore, in this embodiment, a “reliability image” indicating the reliability of having a height relative to the road surface is generated, and a matching process is performed on the image.

(2-2) Generation of Reliability Image Hereinafter, a process in which the reliability image processing unit 22 generates a reliability image will be described.

When the plane projection stereo method is applied to the reference image and the comparison image, a projection difference image is obtained. An image obtained by binarizing the projection difference image is referred to as a residual image A. Under ideal conditions, the residual image A shows only a region having a height relative to the road surface (that is, the region of the vehicle), but the residual image A for the actual input image includes not only the region of the vehicle but also , Some error regions are included (FIG. 7: upper left).

The error region appears, for example, in a white line outline, a vehicle shadow outline, a building shadow outline, and the like. The error area appears on the contour line because of the error of the projection conversion coefficient, the distortion of the lens, and the position of the comparative image and the position of the reference image that are projected and converted because the road surface is not strictly flat but a bale shape. Is slightly shifted.

On the other hand, the areas unnecessary for the matching process are as follows.
Examples include roadside implants, guardrails and vehicles on opposite lanes.

Therefore, these error regions and regions unnecessary for the matching process are removed. The removal of the error region and the unnecessary region is performed separately for the region where the position is fixed and the region where the position is changed.

(2-2-1) Removal of Error Region with Fixed Position First, the error region with the fixed position is removed by masking processing.

The objects to be removed in the first masking process are
It is the area of implantation and the area of the oncoming lane on both sides of the road surface. These areas are unnecessary for detecting a vehicle. Now, assuming that the camera is a fixed camera system, the positions in these images are also fixed. Therefore, the area outside the road surface is masked and excluded from the processing range.

The object to be removed in the second masking process is
This is an error region that appears near the outline of the white line. Although the error region appears only near the outline of the white line, the removal is performed simultaneously for the entire white line and the vicinity of the outline. If the camera is a fixed camera system, the position of the white line is fixed in the image,
It can be excluded from the processing range by the masking processing.

The mask image for the masking process is created by storing a sample image taken in advance in the mask image storage unit 24, manually calling the sample image from there, and designating the position.

(2-2-2) Exclusion of error region where position changes Next, an error region where position changes will be excluded. An error region whose position changes is removed by using a block matching technique.

Some error regions appearing due to a slight shift in the projection conversion are fixed in the image, such as the position of the white line, but some of the positions change.

For example, the shadow of the vehicle itself moves with the movement of the vehicle.

The shadows of the planting, the poles, the guardrails, and the like also move with the movement of the sun.

Further, when the lighting conditions change due to a change in weather, the contrast of the shadow may change, or the shadow itself may disappear.

Therefore, such an error region is removed by using a block matching technique instead of a masking process.

The block matching is processed in the following procedure.

In step 1, the projection conversion image is divided into small blocks.

In step 2, second, the position (x, y) of each of the divided small areas is slid by (Δx, Δy) in the reference image, and the two images in the small area are shifted. Find the absolute value sum (SAD) of the luminance difference,
Search for the position with the smallest residual.

Thirdly, in step 3, the obtained minimum residual is used as the residual of the small area (FIG. 6).

In step 4, fourth, when the residual of the small area is binarized by a threshold value, a small position error equal to or less than the slide amount is absorbed, and the error area is deleted.

As described above, the error region whose position changes is excluded.

(2-2-3) Multi-value conversion of residual image An image in which an error region and an unnecessary region are deleted by masking and block matching is shown in FIG. 7 (upper right). This will be referred to as a residual image B.

In the residual image B that has been subjected to the block matching processing, an error region and a region unnecessary for matching have been removed, but a portion having a small difference in image has also been removed at the same time. The part with a small difference in image is specifically a dark part below the vehicle, a body of a Kuroho truck, a body of a streamlined passenger car, or a part with poor texture such as a back of a duralumin truck. is there. In such a portion having a poor texture, there is no change in the texture even after the projection conversion (because the texture is poor), and there is almost no difference between the projection conversion image and the original image. for that reason,
When the block matching process is performed, these regions are also removed. As described above, the residual image B has a bad influence when performing the matching of the projection images because a part of the vehicle body is missing.

On the other hand, since the residual image A has many error regions, the matching process of the projected image cannot be applied as it is.

Therefore, a new residual image having both characteristics of the residual image A and the residual image B is created by expanding the residual image into multiple values as follows. The pixel values of the new residual image are defined in the following four classes. These four classes are indices indicating the reliability when detecting a tall object.

A0 Pixel in the case where there is no residual a1 Pixel a2 in the case of masking a2 Pixel in the case of the residual having low reliability of height a3 Pixel of high reliability in the presence of height Pixels in the case The multi-valued residual image is shown in FIG. 7 (lower left). This is called a residual image C. FIG. 8 is an enlarged view of FIG. 7 with an explanation of each part added.

The “a0” class is a pixel value assigned to an area having no residual, and indicates that this area has no height.

The “a1” class is a pixel value given to a masked area.

The “a2” class is a pixel value to be assigned to a region having a residual and having low reliability of having a height.

The “a3” class is a pixel value assigned to an area having a residual and having a high degree of reliability of having a height.

In the residual image C, the residual area of the residual image A is classified into a multi-valued class. That is, the pixels represented by the four classes are pixels having multi-valued (quaternary) pixel values.

1) Since the area not to be monitored is unnecessary for matching, it is classified into the class “a0” having no residual. As described above, an area unnecessary for matching can be classified as “a0”.

2) A masked area, such as a white line area, within the monitoring target is classified into class "a1".

3) In the residual image A, the region which was a residual region but has been deleted by a process of deleting an error region such as a block matching process is classified into a class “a2”. Although this region is temporarily deleted as an error region, there is a possibility that a region having a height is actually deleted by mistake. Therefore, it can be said that the class “a2” is “a pixel whose height is low in reliability but cannot be said to have no height”. Pixels of class “a2” will be referred to as “residues with low reliability” in the following description.

4) “a3” is a residual area of the residual image A, which is a residual area that remains even after the error area is deleted. It can be said that this region is a “pixel having high reliability in having a height”. Pixels of class “a3” are referred to as “residuals with high reliability” in the following description.

This residual image C is hereinafter referred to as a “reliability image”.

(2-2-4) Modification In the above embodiment, four multi-levels of a0, a1, a2, and a3 are performed. However, when three multi-levels are performed, masking is performed. The multi-value of a1 may be omitted.
At this time, the object detecting device 10 does not have the mask image storage unit 24 as shown in FIG.

(3) Third-Step Process Since the reliability image is generated by the above-described second-stage process, the position and the number of vehicles are determined by matching the projected image model of the vehicle with the reliability image. Is detected.

Here, a vehicle projection image model required for matching is prepared.

(3-1) Preparation of Vehicle Projection Image Model The vehicle projection image model includes a reliability image, that is, a vehicle image expected to appear in a projection difference image or a residual image.
It is a projection image modeled by a two-dimensional silhouette shape. The model projection image is stored in the object structure detection unit 26.

(3-1-1) Creation of Vehicle Projection Image Model The vehicle projection image model mathematically creates a projection image of the vehicle appearing in the projection difference image according to the three-dimensional model of the vehicle such as a car or truck and the camera parameters. I do. The specific procedure is as follows.

In step 1, camera parameters of two cameras are measured in advance. In step 2, a projection image of the vehicle is created based on each of the reference image and the comparison image based on the camera parameters, and the projection image of the comparison image is projected and transformed.

In step 3, a projected image of the projection difference image is created by superimposing the projected transformed image of the comparative image and the projected image of the reference image. Here, in order to simplify the matching process, the contour of the projected image is approximated by a hexagon. This is defined as a projection image model.

FIG. 9 is a schematic diagram simulating the above procedure.

FIG. 9 simulates a truck with a carrier. Images of passenger cars, light trucks and the like can be generated in a similar manner. The projected image changes for each type of vehicle, but in practice it can often be handled by preparing two types of images of a large car and a small car and devising a matching process. If there is enough processing capacity, preparing a large number of vehicle projection images according to various types of vehicles and performing the matching process allows the vehicle to be recognized more reliably.

Here, a camera parameter and a three-dimensional model of the vehicle are required to generate the model. However, this parameter does not need to be data as precise as required in general stereo image processing. It is sufficient if an image of the vehicle as shown in FIG. 9 can be generated.

(3-1-2) Creation of Vehicle Projection Image Model in Consideration of Vehicle Position By the way, the projection image of the vehicle changes according to the position in the image as well as the type of the vehicle.

For example, a vehicle in front (below the image) has a large projected image, and a vehicle in the back (above the image) has a small projected image. In addition, a change in the depth causes a change in the appearance of the ceiling and the back of the vehicle. Also, the shape of the projected image changes with the movement in the left-right direction.

However, in consideration of the matching process, the projected image for the matching position (search position) is uniquely determined for each vehicle type. That is, even if the projection image models are prepared for all matching positions, the number of projection images is large but the processing amount does not increase. Therefore,
If there is no problem in the storage capacity, the vehicle projection image model may be stored for all the search positions for each vehicle type and the matching process may be executed.

In practice, there is a case where the storage capacity is limited. Therefore, a method of preparing a vehicle projection image model for each sampled position instead of preparing a vehicle projection image model at all search positions. Take.

As shown in FIG. 10, in the horizontal direction of the image, a vehicle projection image model is prepared for each lane. For example, if a three-lane road is to be monitored, three types of vehicle projection image models are prepared for each lane.

As for the vertical direction of the image, there is a method of vertically dividing the image into several zones and preparing a projection image for each zone. For example, if the image is vertically divided into 12 zones, there is no practical problem.

As described above, a vehicle projection image model is generated by sampling each of the vehicle type, the horizontal direction, and the vertical direction, and stored. For example, two types of vehicles (large and small), three types in the horizontal direction (three lanes), and twelve types in the vertical direction (12 zones)
, The image is divided into 3 × 12 = 36 zones, two types of models are stored for each zone, and only 72 vehicle projection images need to be stored.

FIG. 11 shows a partial example of a sampled vehicle projection image model. In FIG. 11, the vehicle types are sampled into two types, a large vehicle and a normal vehicle, and images for three lanes are displayed. In practice, variations due to differences in vertical position are also required.

(3-1-3) Addition of Partial Region to Vehicle Projection Image Model As described above, the basic form of the vehicle projection image model is generated. In order to improve the matching accuracy, the vehicle projection image model Is added to the partial area.

First, since the purpose of the traffic monitoring system is to extract the moving locus and speed of the vehicle, it is important to correctly detect the position of the rear end point of the vehicle as the position in the image of the vehicle. This is because, in order to correctly calculate the three-dimensional position of the vehicle, the position of the ground point in the vehicle area, that is, the position of the rear end point of the vehicle is required.

Therefore, as shown in FIG. 12, the inside of the contour of the projection image model is divided into an “upper region” and a “lower region”.
The position of the dividing line between the “upper region” and the “lower region” is set based on the ratio set by the parameter to the height of the hexagon of the projection image model. Here, in FIG. 12, the upper part of the figure is the traveling direction.

Next, as shown in FIG. 12, a “rear adjacent area” is set behind the vehicle. In the rear adjacent area, both ends of the lower side of the outline of the projected image model approximated by a hexagon are set to be extended by a ratio set by a parameter with respect to the height of the hexagon.

In order to accurately detect the left and right positions of the vehicle, a “right adjacent area” and a “left adjacent area” are set as shown in FIG. The left and right adjacent regions are set such that the left and right sides of the contour of the projection image model are extended by a ratio set by a parameter with respect to the width of the hexagon.

How to handle these partial areas and adjacent areas will be described in the paragraph of matching processing.

(3-2) Matching of Vehicle Projection Image Model The vehicle projection image model generated as described above is matched with a “reliability image” (residual image C, for example, FIG. 7: lower left) as an image to be matched. , It is possible to determine the existence (presence or absence) of the vehicle and determine its position.

The matching hierarchically forms a five-layer loop as shown below. In these loops,
Processing is performed to evaluate whether the projection image of the vehicle projection image model matches the reliability image. The process of selecting the matching result with the highest evaluation value is repeated while performing the repetitive processing in the loop.

The hierarchical structure of the loop has the following configuration. This hierarchical structure is not limited to this. In short, matching processing is performed on all vehicle types and positions, and the best result may be selected.

Loop 1 Search for each vehicle type Loop 2 Search for each lane (1st lane to 3rd lane) Loop 3 Search for each zone Loop 4 Search for up / down position in zone Loop 5 Search for left / right position Briefly, in loop 1 it is determined whether the vehicle is large or small, in loop 2 it is determined that the vehicle is in the first, second, or third lane while it is large, and in loop 3 it is determined that the vehicle is large and the first lane. In the loop 4, it is determined that the zone is large and the first lane is 8 zones, and the vertical position in the zone is determined.
The right and left positions are determined among the large positions in the first lane and the positions above the zones in the eight zones.

(3-3) Calculation of Matching Evaluation Value A procedure of a process for evaluating matching in the above loop will be described.

The matching of the present embodiment is different from the process of matching a shape to a general binary image, and is a process of matching a model of a projected image to a multivalued residual image.
Since a multi-valued residual image is used, matching evaluation is performed using the following four counters.

E0 The number of counted pixels without residuals e1 The number of counted pixels of residuals with high reliability e2 The number of counted pixels of residuals with low reliability e3 The number of pixels outside the lane area Thus, the feature of the present embodiment is that the evaluation counter can be divided according to the difference in the reliability.

In the step of evaluating the matching, the position of the projection image of the vehicle projection image model is set, and the suitability of the reliability image with respect to the position of the model is evaluated. Therefore, for each partial region of the projection image model, a process of counting the residual pixels of the reliability image (there are two types of high reliability residual and low reliability residual) is performed. At this time, each counter is incremented for each line (scanning line) based on the following conditions.

<Condition 1> If there is a “residual pixel with high reliability” in the own lane, the number of “residual pixels with high reliability” is added to e1, and the “reliability pixel” is added to e2. Pixel number of the “low residual pixel”. At this time, if the projected image model protrudes into an adjacent lane, the protruding pixels are similarly assigned e1 and e2 for each class.
Is added to the number of pixels.

<Condition 2> If there is no "residual pixel with high reliability" in the own lane, neither e1 nor e2 is counted.

Conditions 1 and 2 will be referred to as “own lane existence conditions”.

FIG. 13 shows a conceptual diagram of the own lane existence condition. FIG. 13 shows, as an example, whether or not to count the residual in each case on the “line of interest”.

The example of (a) shows that the “line of interest” is
Since “residuals with high reliability” exist in the own lane, the residuals are counted.

On the other hand, in the examples of (b), (c), and (d), the “line of interest” is not counted because the “highly reliable residual” does not exist in the own lane. In particular, in the example of (b), the area under the projected image satisfies the own lane condition, but the “line of interest” does not satisfy the own lane condition.
Is not counted.

The own lane condition is “residual pixel with high reliability”.
Is a condition (condition 1) for counting both the “residual pixel with high reliability” and the “residual pixel with low reliability” under the condition (condition 2) that exists. That is, if there is no “residual pixel with high reliability”, the evaluation is 0, so that the determination of the existence (presence or absence) of the object depends on the “residual pixel with high reliability”. As will be described later, when determining the position of the object, the position of the object is determined by the evaluation value obtained by adding both the “high-reliability residual pixel” and the “low-reliability residual pixel”. The determination of the position will be determined using both residuals.

In summary, the existence (presence / absence) of the object is determined by using the information of the pixels in the region where the reliability of presence of the object is high. In determining the position of the object, the determination was made by simultaneously using both the pixels in the region with high reliability of the presence of the height and the pixels in the region with low reliability.

Here, the effect that the residual area is multi-valued with reliability is shown.

The "own lane existence condition" is particularly effective for a large vehicle. Due to the height of large vehicles,
The vehicle projection image model largely protrudes into the adjacent lane.

[0146] For example, it is assumed that the vehicle exists in the second lane and does not exist in the third lane. Here, assuming that a vehicle exists in the third lane, the vehicle projection image is matched. The center of the assumed vehicle projection image is the third lane, but the vehicle projection image protrudes into the adjacent second lane. The actual vehicle is second
In an area that exists in the lane but protrudes into the second lane,
This contributes to the matching evaluation amount of the vehicle projection image of the third lane. The “own lane existence condition” effectively functions to prevent such erroneous matching contribution.

When calculating the matching evaluation value, not only the counting of the number of compatible pixels in the hexagonal model but also the counting of the number of compatible pixels in the rear and right and left adjacent areas are performed simultaneously. The count of the number of compatible pixels in the rear and left and right adjacent areas is also
Counting takes into account the “own lane existence condition”.

Here, when counting the residual pixels in the left and right adjacent areas, a part of the adjacent area may be masked due to a white line. In such a case, as shown in FIG. 14, an alternative area is set in an area where the white line is skipped, and residual pixels are counted.

(3-4) Judgment of Matching Evaluation Value Verification and comparison of the matching evaluation value in the projection image model are performed as follows.

(3-4-1) Verification of Conditions of Neighboring Regions of Vehicle Projection Image Model The evaluation of matching is performed by sequentially checking the conditions of each partial region shown in FIG. Is performed to select the candidate having the highest evaluation value from.

(3-4-1-1) Evaluation of Lower Region As described above, since it is important to correctly determine the rear end point of the vehicle in matching, the lower region inside the outline of the vehicle projected image model is correctly determined. Matching is required.

Therefore, it is first verified whether the number of compatible pixels in the lower region is sufficient.

[0153] B of the lower region, the area of the _{N B} lower region,
When the B 'was equipped with a self-lane existence condition in the lower region area, obtains the evaluation value V ₁ of the equation (3).

[0154]

(Equation 3) Note that the numerator on the right side of the equation (3) is e in the region of B ′.
The sum of 1 and e2 is shown.

If V ₁ is larger than the predetermined threshold value Thr ₁ , the condition is satisfied and the next evaluation is performed.

(3-4-1-2) Evaluation of Rear Adjacent Region Next, it is verified whether the number of residual pixels in the rear adjacent region has a sufficiently low density.

If the vehicle projection image model is correctly matched, there should be almost no residual pixels in the rear adjacent area. Therefore, the adjacent region of the backward R, the area of the region of the N _R, when the region provided with the vehicle moving lane presence proviso R 'at the rear of the adjacent regions, obtaining the evaluation value V ₂ of the equation (4).

[0158]

(Equation 4) Note that the numerator on the right side of the equation (4) is e in the region of R ′.
The sum of 1 and e2 is shown.

[0159] and, in this case _{V 2} the threshold Thr ₂ smaller than a preset, do the following evaluation.

(3-4-1-3) Evaluation of Left and Right Neighboring Regions Next, left and right neighboring regions are evaluated.

The left and right adjacent areas are different from the rear adjacent area, and even if the matching is correct, residual pixels may be present in the left and right adjacent areas. For example, a case where a large vehicle runs parallel to an adjacent lane, or a case where the projected image of the vehicle is larger than the width of the model (not only when the width of the model is large but also when the height is high, The width may increase due to the effect of projection).

Therefore, the judgment on the left and right adjacent areas is as follows.
Processing changes depending on the model of the model. First, it is determined whether a residual pixel exists in the left and right adjacent areas. Therefore, the left or right flanking region of the S, the area of the region of the N _S, M _S
When was the number of pixels removed by the masking within the region, checks greater than the threshold value Thr ₃ set in advance (5) evaluation value V ₃ of the formula.

[0163]

(Equation 5) Here, the reason why the number of pixels removed by masking is subtracted from the denominator is that the left and right adjacent areas often include a white line area, which is actually a no-count area. This is because it is reasonable to subtract from the denominator when calculating.

[0164] the evaluation value V _3, if it is determined that the residual pixel exists on the left and right flanking regions, if the model is a standard car is judged as "a state of being matched to a portion of large vehicles" This matching candidate is rejected.

On the other hand, when the model is a large car, the evaluation value V
₃ , even if it is determined that the residual pixels exist in the left and right adjacent regions, if there is a residual region with a sufficient density in the lower region of the projection image model, “the large vehicle is running in parallel. It is in a state ", and this matching result is recognized.

That is, the predetermined threshold value Thr ₄
Is determined using the following.

[0167]

[Table 1] There is a case where a large vehicle does not reject candidates even if the density of the left and right adjacent areas is high because vehicles may run in the own lane and the adjacent lane in some cases. Are there conform pixels inside the contour line is not protruding from the adjacent lane, to determine that the vehicle is present certainly own lane, it introduces a new threshold Thr _4.

After verifying the conditions of the adjacent area of the vehicle projection image model as described above, a candidate for a position that best fits is finally selected.

(3-4-2) Determination of Vehicle Position This step is for determining the position of the vehicle.

In this step, the evaluation value is basically determined by the number of matched pixels, not by the ratio to the area of the model. This is because a relatively small projection image is advantageous when compared by area ratio. When compared by area ratio, for example, a vehicle having a large projected image such as a truck with a black hood and having a missing residual area therein has a low density, which makes detection difficult.

[0171] That is, (6) using the evaluation value _{V 5} of formula, selecting a candidate of the best position.

(Equation 6) When the matching process is performed in the above-described loop hierarchical structure, the best position candidate is detected for each vehicle type and each lane.

Here, as means for indicating the position, for example, the coordinate value of the center of the line at the lower end of the lower area is used. These coordinate values are represented by (x, y) with the horizontal direction of the image as the x axis and the vertical direction as the y axis.

When overlapping candidates are obtained, they are merged into candidates having high evaluation values. The method of integration processing are considered some, but there is simply (6) how to integrate towards V ₅ is large type. As the result obtained for each lane, the result obtained individually may be output in parallel as it is.

Finally, it is checked that the evaluation value V ₆ (density) of the equation (7) is larger than the threshold value Thr ₆ , and the passed candidate (vehicle projected image model) is set as the final output.

[0175]

(Equation 7) Thus, the processing for the input of one frame (a pair of stereo images) is completed.

As a result, in which lane, in which lane,
It can be reliably determined at which position it exists.

(Embodiment 2) In the first embodiment, with respect to the evaluation method using the reliability image, only the evaluation of the density inside the model and the evaluation of the density of the adjacent area are performed. There are items that can be evaluated.

For example, there is a case where the right / left balance of the matching is checked in order to improve the matching accuracy of the model. In this case, since the position of the model is determined, it is effective to process the information of the residual pixels having low reliability.

Checking the left / right balance of matching
For example, as shown in FIG. 15, a center line can be set from a hexagon of the vehicle projection image model, and the inside of the contour can be divided into left and right regions. It can be determined by counting the number of residual pixels included in each of the left and right regions, calculating the ratio, and verifying whether the ratio is within a predetermined range.

(Embodiment 3) As another example using a reliability image, a method of extending the model shape from the lower region to the upper portion will be described.

In the first embodiment, an example is described in which only two types, a large car model and a normal car model, are used. However, there are actually various vehicles such as medium-sized cars.

When a normal car model is applied to a medium-sized car, the model cannot be applied despite the presence of a residual area in front of the contour. In addition, when a large-sized vehicle model is applied to a medium-sized vehicle, an upper region is filled, and in a bad case, a part of a vehicle ahead is included in the upper region.

Therefore, the projected image of the vehicle is not counted in the whole area of the hexagonal model at once, but is expanded in the upper area while checking the continuity of the residual area after matching in the lower area. The method may be effective.

As described above, even when the projection image model is extended from the lower region to the upper region, the use of the reliability image is very effective.

FIG. 16 is an explanatory diagram of a method of extending a model from a lower region to an upper portion.

Now, it is assumed that the contour line of the projected image model is a model of a large vehicle, and that the situation matches a medium-sized vehicle appearing in the residual area.

Even when the area is extended, the contour of the projection image model is necessary, and it is necessary to extend the contour so as not to exceed the contour.

[0188] The reason is that simply connecting continuous areas without a boundary line results in an overlapped vehicle becoming one lump and becoming larger without stopping.
If the projected image model of the vehicle is generated from the three-dimensional model of the large vehicle, there is no connection beyond this contour. In this sense, the outline of the projection image model is important information.

Now, it is assumed that there is a sufficient density of residual pixels in the lower region inside the contour and that the density of 0 pixels in the adjacent region behind is also lower than the predetermined threshold value. Here, the area is extended from the lower area to information. The condition for expansion is that “a highly reliable residual pixel exists in the line of interest”. If the condition is satisfied, both residual pixels are counted in the evaluation amount regardless of the degree of reliability.

In the example shown in FIG. 16, the image is expanded up to the position of the horizontal line (L), but is not expanded above L since there is no highly reliable residual pixel.

Even in such a case, the reliability image is effectively used.

(Embodiment 4) In the above embodiments, the reliability level of the reliability image has only two values of "high" and "low". However, there is a method of extending the reliability image to more values.

(1) First Method If the difference value of the projection difference image is directly used as the reliability, a multi-level reliability image as many as the input image can be constructed.

(2) Second Method Using the binarized residual image (for example, residual image A or residual image B in FIG. 7), the number of residual pixels around a certain pixel is reduced to 8 There is also a method of counting in the vicinity and using the number as reliability.

According to this method, the reliability of a thin linear region is about 1 to 2, and the reliability of the central portion of a block-like region is 8 or the like.

(3) Third Method There is also a method of evaluating temporal continuity and making the continuous time reliability.

For example, based on the residual image A, the time during which the pixel of interest continues as a residual pixel is measured.

For example, if the pixel is a residual pixel continuously from 10 frames before, the reliability is 10; if the pixel is the first residual pixel in the current frame, the reliability is 1. If the duration exceeds 200, the reliability can be set to 200 or the like.

(4) Fourth Method A negative value can be given to the reliability.

For a position that is lower than the road surface and is definitely in an error area even if a residual appears, it is determined that the value is positive or negative even if the residual appears for a position that is known in advance. Conversely, negative reliability can be given.

The meaning of the portion is the same as the reliability of 0, but the effect of transmitting information using the value of the reliability using the value of the reliability is shown in the processing of the neighboring area. There is.

[0202]

As described above, it is possible to correctly determine the existence (presence or absence) of an object existing in the monitoring range, and it is possible to improve the detection accuracy of the position of the object.

By utilizing this result, it is possible to improve the performance of detecting an obstacle ahead of a vehicle-mounted camera, and to improve the performance of detecting an object such as a traffic monitoring system or an intruder monitoring system. Will be possible.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of an object detection device according to a first embodiment of the present invention.

FIG. 2 is a configuration diagram of a modified example of the first embodiment.

FIG. 3 is a conceptual diagram of a system.

FIG. 4 is a projection relation diagram.

5 is an explanatory diagram of an input image and a projection difference, wherein the upper left is an image of a left camera, the upper right is an image of a right camera, and the lower left is an image obtained by projecting and converting an image of a right camera; The lower right part is a projection difference image.

FIG. 6 is an explanatory diagram of a method of removing an error region by block matching.

FIG. 7 is an explanatory diagram of a residual image.
(A binarized projection difference image), the upper right is a residual image B (an image in which an error region and an unnecessary region are deleted by masking and block matching), and the lower left is a residual image C (a multi-valued residual). Image).

FIG. 8 is an explanatory diagram of a residual image.

FIG. 9 is a diagram illustrating a method of generating a vehicle projection image model.

FIG. 10 is an example of dividing an image into blocks.

FIG. 11 is an example of a vehicle projection image model.

FIG. 12 is a detailed view of a vehicle projection image model.

FIG. 13 is a conceptual diagram of an own lane condition.

FIG. 14 is a schematic diagram of a process of checking left and right adjacent areas.

FIG. 15 is a schematic diagram of a balance check.

FIG. 16 is an explanatory diagram of a method of extending a model from a lower region to an upper portion.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 10 Object detection apparatus 12 Image input part 14 Image input part 16 Reference image generation part 18 Projection conversion image generation part 20 Image comparison part 22 Reliability image generation part 24 Mask image storage part 26 Object structure detection part

 ──────────────────────────────────────────────────続 き Continued on the front page F-term (reference)

Claims (9)

[Claims] 1. An object detection apparatus for detecting an object having a height relative to a reference plane forming a common background of both images from a reference image having a common field of view and a comparison image, wherein: The comparison image is a pair of image input means for inputting, respectively, a projection-transformed image by projecting and converting a position of a pixel in the comparison image to a position of a corresponding pixel in the reference image with reference to the reference plane. Projection conversion means for generating, from the projection-converted projection-converted image and the reference image,
Image comparison means for generating a projection difference image in which the region of the object appears as a residual; and three or more multi-valued pixels indicating the reliability when detecting the object based on the projection difference image. Reliability image generation means for generating a reliability image, and object structure detection means for detecting the presence, type, or position of the object based on multi-valued pixel information of the reliability image. An object detection device characterized by the above-mentioned. 2. The reliability image generating means, wherein the multi-valued pixel is a class representing a region having no height, a class representing a region having low reliability of having a height, and a class representing a region having a height. At least three of the classes representing regions with high confidence
2. The method according to claim 1, wherein the class is classified into types.
The object detection device according to claim. 3. The object detection apparatus according to claim 2, wherein the multi-valued pixels are further classified into classes indicating that the multi-valued pixels are not applicable to the object detection processing. 4. The object structure detecting means, wherein the presence or absence of the object is determined by using information of a class representing a high reliability area, and the position of the object is determined by a class of a high reliability area. 2. The object detection apparatus according to claim 1, wherein the determination is performed by simultaneously using the information and information of a class representing an area with low reliability. 5. The object structure detection means, wherein a projection image model of the object expected to appear in the projection difference image is matched with the reliability image, and the presence / absence and type of the object in the reliability image Or
2. The object detection device according to claim 1, wherein the position is detected. 6. The object structure detecting means, wherein when the projection image model of the object expected to appear in the projection difference image is matched with the reliability image, the projection image model and the reliability image Counting the number of the classified pixels in a portion overlapping with the area of the classified pixels, based on the count number of each class, the presence or absence of the object, the type,
3. The object detection device according to claim 2, wherein the position is determined. 7. The object structure detecting means, wherein the projection image model is at least composed of an upper area, a lower area, and a rear adjacent area, and the presence or absence of the object is determined based on the lower area and the count number of each class. The object detection apparatus according to claim 5, wherein the position of the object is determined based on the rear adjacent area and the count number of each class. 8. An object detection method for detecting, from a reference image having a common field of view and a comparison image, an object having a height with respect to a reference plane constituting a common background of the two images, The comparison image is a pair of image input steps to be input, respectively, Projection conversion image by projecting the position of the pixel in the comparison image to the position of the corresponding pixel in the reference image with reference to the reference plane A projection conversion step to generate, and from the projection-converted projection-converted image and the reference image,
An image comparison step of generating a projection difference image in which the region of the object appears as a residual; and, based on the projection difference image, three or more multi-valued pixels indicating reliability at the time of detecting the object. A reliability image generating step of generating a reliability image, and an object structure detecting step of detecting presence / absence, type, or position of the object based on information of multi-valued pixels of the reliability image. An object detection method characterized by the following. 9. A program for realizing, by a computer, an object detection method for detecting, from a reference image and a comparison image having a common field of view, an object having a height with respect to a reference plane constituting a common background of the two images. A pair of image input functions for inputting the reference image and the comparison image, respectively; and projecting a pixel position in the comparison image to a corresponding pixel position in the reference image with respect to the reference plane. And a projection conversion function of generating a projection conversion image, from the projection-converted projection-converted image and the reference image,
An image comparison function for generating a projection difference image in which the region of the object appears as a residual; and three or more multi-valued pixels indicating reliability when the object is detected based on the projection difference image. A computer-implemented reliability image generation function of generating a reliability image, and an object structure detection function of detecting the presence / absence, type, or position of the object based on multi-valued pixel information of the reliability image A program for an object detection method characterized by performing the following.