JP2005024463A - Stereo wide visual field image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a wide effective visual field by also utilizing a non-stereoscopic imaging area, while obtaining precise distance information from a stereoscopic imaging area. <P>SOLUTION: The optical axes of respective cameras 1a and 1b of a stereo camera 1 are arranged so as to be non-parallel on the same plane, the effective visual field is enlarged by the stereoscopic area RM and the non-stereoscopic area RS, a pivot 3 is provided to the central position of a stay 2, and the overall camera system is rotated by a driving motor 4 so that the wide area can be scanned. Accordingly, in the application to an intruding object monitoring apparatus, when the object intrudes into the visual field of either camera, the object can be detected. Thus, the detectable range can be enlarged, when compared with the apparatus using the stereo camera in which the optical axes are arranged parallel. Further, when compared with the monitoring apparatus in which a plurality of monocular cameras are merely provided together, the secure detection simultaneously using the stereo system can be performed so that the reliability can be improved by preventing incorrect detection. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a stereo wide-field image processing apparatus capable of obtaining accurate distance information and a wide effective visual field using a stereo camera.
[0002]
[Prior art]
Generally, as a three-dimensional measurement technique using images, the corresponding position between a pair of images captured by a stereo camera is obtained, and the principle of triangulation using the camera parameters such as the stereo camera mounting position and focal length from the parallax of the corresponding position There is known a technique of image processing by a so-called stereo method for obtaining a distance by the above method. This stereo image processing technology can be applied to monitoring devices that monitor intrusions into buildings and sites, vehicle surroundings monitoring devices that detect other vehicles and pedestrians in front of the vehicle, and the like. The presence / absence of an object and the three-dimensional information of the object can be obtained from the stereo image.
[0003]
As long as there is a stereo viewing area where the fields of view of the two cameras overlap each other, in principle, each camera may be placed at any position. For convenience, a so-called parallel stereo camera is often used in which the other camera is arranged at a position where one camera is translated and the optical axes of the other cameras are parallel to each other on the same plane. A parallel stereo camera may be referred to as a standard stereo camera.
[0004]
In addition to the parallel stereo cameras described above, there are also known image processing techniques using a stereo camera arranged in a direction in which the optical axes of the cameras cross each other and a stereo camera arranged in a direction in which the optical axes of the respective cameras open. The former is disclosed in Patent Document 1 (JP-A-6-331349), and the latter is disclosed in Patent Document 2 (JP-A 2001-312018).
[0005]
[Patent Document 1]
JP-A-6-331349 [0006]
[Patent Document 2]
[Patent Document 1] Japanese Patent Laid-Open No. 2001-312018
[Problems to be solved by the invention]
However, conventionally, an apparatus that detects an object by processing an image captured by a stereo camera can obtain accurate distance information, but only a stereo vision region is processed from the entire field of view of the camera. There is a problem that the effective visual field is narrow. In the stereo cameras disclosed in Patent Document 1 and Patent Document 2, the field of view is enlarged as compared with the parallel stereo camera, but is limited to the area of stereo vision.
[0008]
That is, the technique disclosed in Patent Document 1 is limited to the function of stereo distance measurement, and is not useful at all for the non-stereoscopic region in the entire visual field of the stereo camera. Further, the technique disclosed in Patent Document 2 obtains a stereo image that reproduces the human visual field in a form that is close to natural, and considers only the function that humans see, and does not consider the function of stereo ranging.
[0009]
The present invention has been made in view of the above circumstances, and a stereo wide-field image capable of obtaining a wide effective visual field by using a non-stereoscopic imaging region while obtaining accurate distance information from the stereoscopic imaging region. An object is to provide a processing apparatus.
[0010]
[Means for Solving the Problems]
In order to achieve the above object, a stereo wide-field image processing device according to the present invention is a set of cameras having imaging fields of view that overlap each other, and the optical axis of one camera and the optical axis of the other camera are non-parallel. A stereo camera arranged, non-stereo processing means for detecting the presence of an object from images captured by each of the stereo cameras, and the presence of an object in a region where the imaging fields of a pair of images captured by the stereo camera overlap each other. Stereo processing means for detecting the distance to the object is provided.
[0011]
DETAILED DESCRIPTION OF THE INVENTION
Embodiments of the present invention will be described below with reference to the drawings. 1 to 5 relate to a first embodiment of the present invention, FIG. 1 is an explanatory diagram showing the arrangement of a stereo camera, FIG. 2 is a block diagram of a wide-field image processing apparatus, and FIG. 3 is an intruder detection process. FIG. 4 is an explanatory diagram showing intruder candidates in non-stereo view, and FIG. 5 is an explanatory diagram showing confirmation of an intruder in stereo view.
[0012]
In FIG. 1, reference numeral 1 denotes a stereo camera composed of two cameras 1a and 1b synchronized with each other. In FIG. 1, an arrangement viewed from a direction perpendicular to the installation surface of both cameras 1a and 1b is shown. . Each camera 1a, 1b constituting the stereo camera 1 is a camera using, for example, a CCD (Charge Coupled Device) as an imaging device, and has an optical axis (imaging surface vertical axis) while having a region where the imaging fields of view overlap each other. ) Are installed on the stay 2 so as to have a non-parallel deviation on the same plane. Here, the deviation is a shake with respect to the case where the optical axes of both cameras 1a and 1b are arranged in parallel.
[0013]
The deviation of the optical axes of the cameras 1a and 1b is set in the direction in which the optical axes open as shown in FIG. 1 or in the direction in which the optical axes cross each other. Therefore, with respect to a stereo camera (standard stereo camera) in which the optical axes of the cameras are arranged in parallel, a stereo vision region RM where the imaging fields of view of both cameras 1a and 1b overlap, and a monocular non-stereo for each of the cameras 1a and 1b. The effective visual field can be enlarged by the viewing area RS, and the stereo camera 1 can be a stereo camera for wide-field images.
[0014]
Furthermore, in this embodiment, a turning shaft 3 for turning the entire camera system is provided at the center position of the mounting interval between both cameras 1a and 1b of the stay 2 on the center line of the camera system. A drive motor 4 is connected to the turning shaft 3 directly or via a gear or the like, and a camera rotation mechanism is formed that rotates the entire stereo camera 1 while maintaining the relative positions of the cameras 1a and 1b.
[0015]
Then, the entire motor system is swiveled along a plane formed by both optical axes around the swivel axis 3 by the drive motor 4 to enable wide-area scanning. However, in this embodiment, the turning of the camera system is not continuous, but is discrete at a predetermined finite number of rotation angles, for example, every 10 degrees in steps of a camera system center line. Is something.
[0016]
Note that each of the cameras 1a and 1b may be rotated without rotating the entire stereo camera 1, but in this case, it is necessary to precisely manage the relationship between the camera position and the rotation angle. .
[0017]
In the following description, it is assumed that the cameras 1a and 1b are horizontally arranged in the stereo camera 1. One camera 1a is a right camera that captures a reference image in stereo matching processing, and the other camera 1b is stereo matching processing. The left camera that captures the reference image in FIG. 1, the base image is represented as the right image, and the reference image is represented as the left image.
[0018]
A pair of images captured by the stereo camera 1 as the wide-field image stereo camera is processed by the wide-field image processing device 20 shown in FIG. The wide-field image processing device 20 corresponds to the system of each camera 1a, 1b, and the analog image signal is proportionally amplified by the amplifiers 5a, 5b and the amplifiers 5a, 5b for proportionally amplifying the image signals from the cameras 1a, 1b. A / D converters 6a and 6b that convert the signal into a digital image signal of a predetermined luminance gradation (for example, a gray scale of 256 gradations), correction circuits 7a and 7b that perform correction of intrinsic gains of the amplifiers 5a and 5b, etc. Stereo matching circuit 8 for obtaining the parallax (pixel shift amount) of the corresponding position by performing stereo matching such as searching for the corresponding position in the scanning line direction in the stereo viewing region RM between the left and right images that have passed through the circuits 7a and 7b, stereo A distance image memory 9a for storing the parallax obtained by the matching circuit 8 as distance image data and a right image for storing the original image data on the right after passing through the correction circuit 7a An image memory 9 comprising a memory 9b and a left image memory 9c for storing left original image data that has passed through the correction circuit 7b, distance image data stored in the image memory 9, and left and right original image data are mainly used for various image processing. A microcomputer 10 that performs image recognition, and a turning control circuit that drives the drive motor 4 by the angle instruction value of the turning shaft 3 that is appropriately output from the microcomputer 10 and turns the entire stereo camera 1 by an angle according to the angle instruction value. 11 is configured.
[0019]
The microcomputer 10 includes an image recognition processing unit 10a as firmware. This image recognition processing unit 10a includes functions as a non-stereo processing means and a stereo processing means according to the present invention, and uses a wide field of view information by non-stereo view and detailed distance information by stereo view. Recognition processing such as recognition of the presence and position of a person or object in intrusion monitoring into a building, recognition of the presence and position of a preceding vehicle or pedestrian in front / side monitoring of the vehicle, and the like.
[0020]
Further, the image recognition processing unit 10a also includes a function as a camera rotation control unit according to the present invention, and an angle instruction value for instructing the turning control circuit 11 about the turning angle of the turning shaft 3 when necessary in the recognition processing. Then, the stereo camera 1 is turned by the drive motor 4 to change the imaging direction.
[0021]
Hereinafter, the case where the wide-field image processing device 20 using the stereo camera 1 is applied as an intruder monitoring device for monitoring an intruder (intruder) into a building / site is taken as an example. The intruder detection process will be described with reference to the flowchart of FIG.
[0022]
The intruder detection process shown in FIG. 3 shows a process for each frame of the captured image. First, in step S1, an intruder detection process by a monocular method is performed on both the right image data and the left image data. Since the intruder detection in this step S1 is a process by a non-stereo method, the detection accuracy of the intruder is low, and there is a possibility of detecting a virtual image such as a shadow reflected on the ground or reflection of external light. Accordingly, in the intruder detection process in step S1, an area that is highly likely to be an intruding object in the image is extracted as a “candidate” for the intruder for the time being, and the presence / absence and position of the intruder candidate are detected. .
[0023]
As a technique for extracting such an intruder candidate, a known technique, for example, an area extraction technique using a background subtraction method can be employed. In the region extraction by the background subtraction method, an image that does not contain an intruder is used as a background image. In this embodiment, in order to cope with the turning of the entire camera system, a different background image is stored in advance in the memory for each angle increment described above. Keep it.
[0024]
Then, an intruder candidate is detected by selecting a background image corresponding to the current turning angle of the camera system and comparing it with the image of the current frame. For example, the luminance of each pixel of the background image and the image of the current frame is compared, and a set of pixels whose absolute value of the luminance difference is equal to or greater than a predetermined threshold is extracted as an intruder candidate region. The position of the intruder candidate is represented by the center of gravity of the region.
[0025]
Next, the process proceeds from step S1 to step S2, and it is checked whether or not there is an intruder candidate in at least one of the left and right images. As a result, if no intruder candidate exists in either of the left and right images, the process waits until the next image frame and performs the process of step S1 again on the next frame.
[0026]
If there is an intruder candidate in at least one of the left and right images, the process proceeds from step S2 to step S3 to check whether there are intruder candidates in both the left and right images. If there is an intruder candidate in only one of the left and right images, that is, if the intruder candidate appears in a non-stereoscopic region where the fields of view of both cameras 1a and 1b do not overlap, the process proceeds from step S3 to step S4. The process of turning the entire stereo camera 1 is performed, and if there are intruder candidates in both the left and right images, the process proceeds from step S3 to step S5, and the intruder is determined by the stereo method.
[0027]
However, in the same frame, if different intruder candidates appear in the non-stereoscopic area of the left image and the nonstereoscopic area of the right image, the installation position and monitoring position of the stereo camera 1 are displayed. The process advances from step S3 to step S4 assuming that there is an intruder candidate only on the image side in the direction set in advance based on the priority.
[0028]
When there is an intruder candidate in only one of the left and right images, in step S4, an angle instruction value for turning the entire stereo camera 1 so that the intruder candidate enters the stereo viewing area is calculated based on the position information of the intruder candidate. This angle instruction value is output to the turning control circuit 11 to rotate the drive motor 4 to turn the entire stereo camera 1. That is, when the intruder candidate appears in the right image, the entire stereo camera 1 is rotated to the right, and when the intruder candidate appears in the left image, the entire stereo camera 1 is rotated to the left and waits for the next frame. To do.
[0029]
For example, as shown in FIG. 4, when the intruder candidate S ′ is detected in the non-stereoscopic region of the right image in the process of step S1, this intruder candidate S ′ It may be a virtual image such as light reflection. Accordingly, the entire stereo camera 1 is rotated to the right to move the line L passing through the center of gravity of the intruder candidate S ′ into the stereo viewing region of the left and right images, and the next frame is photographed as shown in FIG. The intruder candidate S ′ is obtained by performing stereo matching on the intruder image SR reflected in the stereo view area of the right image and the intruder image SL reflected in the stereo view area of the left image. To see if it really captures the intruder.
[0030]
When the intruder candidate appears in both the left and right images and the intruder candidate enters the stereo vision region, in step S5, the intruder detection process is performed again using the stereo method, and the intruder is accurately determined. As the intruder detection process using the stereo method, for example, a matching technique using background parallax described in Japanese Patent Application No. 2002-255417 by the present applicant can be used.
[0031]
In this matching by background parallax, stereo matching is performed in advance on left and right images that do not contain an intruder, and parallax (background parallax) at each position on the reference image (right image) is obtained and stored as a background distance image. . Then, the left and right original images are shifted by the background parallax of the background distance image, and the degree of coincidence of the left and right images is calculated. However, in this embodiment, in order to cope with the turning of the entire camera system, a different background distance image is stored in advance for each angle increment described above.
[0032]
When there is no intruder in the image, a point on the right image and a point on the left image shifted by the amount of background parallax show the same position in real space. As a result of calculating the degree of coincidence, if the degree of coincidence is high, it can be determined that an object that originally exists in the background has been detected (no intruder exists), and if the degree of coincidence is low, it is determined that an intruder exists. be able to.
[0033]
Then, it progresses to step S6 and the presence or absence of an intruder is determined from the processing result in step S5. If the intruder candidate is a false detection due to the influence of noise or the like and it is determined that there is no intruder, the process waits for the next frame and restarts the processing from step S1 for the new frame. If it is determined that there is an intruder, a warning by sound or light is output to warn of the presence of the intruder by sounding a buzzer or turning on a red rotating rod.
[0034]
In the present embodiment, the stereo camera 1 in which the optical axes of the cameras 1a and 1b are arranged with a deviation is used, and the non-stereo visual field area and the stereo visual field area of the stereo camera 1 are used as observation areas. Therefore, when applied to an intruder monitoring device, if an object enters the field of view of one of the cameras, the intruder can be detected, and a standard stereo camera with optical axes arranged in parallel was used. Compared with the device, the detectable range can be expanded.
[0035]
In addition, compared to a monitoring device with multiple monocular cameras, it can perform reliable detection using the stereo method, preventing false detection of shadows, reflected light, and other virtual images. And reliability can be improved.
[0036]
Next, a second embodiment of the present invention will be described. 6 to 8 relate to the second embodiment of the present invention, FIG. 6 is an explanatory diagram showing the arrangement of the stereo camera, FIG. 7 is a block diagram of the monitoring device, and FIG. 8 is a flowchart of the front / side monitoring process. It is.
[0037]
The second form shows an application example to a front / side monitoring device mounted on a vehicle, and takes into account the processing speed required for a moving body such as a vehicle and prevention of deviation from the monitoring area. The camera 1 is fixed so that the center axis of the camera system faces the front in the traveling direction of the vehicle without turning the camera 1.
[0038]
That is, as shown in FIG. 6, the turning shaft 3 and the drive motor 4 for turning the stereo camera 1 are abolished and the stereo camera 1 is fixed to the vehicle, and the stereo camera 1 is fixed. As shown in FIG. 7, in the front / side monitoring apparatus 20A of the present embodiment, the turning control circuit 11 is omitted from the wide-field image processing apparatus 20 described in the first embodiment.
[0039]
The front / side monitoring process of the host vehicle by the front / side monitoring device 20A is shown in the flowchart of FIG.
[0040]
The flowchart in FIG. 8 shows processing for each frame of the captured image. First, in step S11, a pedestrian detection process by a monocular method is performed for each of the left and right images. In pedestrian detection by this monocular method, it is difficult to apply the background subtraction method explained in the first embodiment because the scenery of the vehicle's outside world changes variously. For example, an optical flow that is a motion vector between frames is used. Use to detect pedestrians.
[0041]
In the pedestrian detection process using the optical flow, an object in an image is captured as a lump moving area, and an object in a specific direction is detected based on the magnitude and direction of a motion vector in this area. In other words, by examining the optical flow between the images, the region from the center of the traveling road of the vehicle toward the outside in the image can be recognized as an object such as a power pole or a tree standing beside the road, and The area toward the center of the vehicle traveling path can be recognized as a pedestrian.
[0042]
When the pedestrian detection process in step S11 is completed, the process proceeds to step S12 to check whether there is a pedestrian that may collide with the host vehicle. When a pedestrian that may collide with the host vehicle is detected, an alarm is output by, for example, sounding a buzzer installed near the driver's seat to prompt the driver to perform a collision avoidance operation.
[0043]
As in the first embodiment, the stereo camera 1 is provided with the turning shaft 3, and the drive motor 4 is driven via the turning control circuit 11 only at an extremely low speed equal to or lower than the set speed so that the stereo camera 1 is turned. In addition, it is possible to detect the pedestrian more accurately by adding the distance information between the own vehicle and the pedestrian by the stereo distance measurement to the detection of the pedestrian by the monocular method. In this case, when the set speed is exceeded, the center axis of the camera system of the stereo camera 1 is returned to the front in the traveling direction of the vehicle, and turning by the drive motor 4 is prohibited.
[0044]
On the other hand, if a pedestrian that may collide with the host vehicle is not detected, the process proceeds from step S12 to step S13, and the process proceeds to forward monitoring in the stereo viewing area where the fields of view of the right camera 1a and the left camera 1b overlap. In this forward monitoring, the left and right images are stereo-matched, and the distance between the preceding vehicle and the host vehicle in front of the host vehicle is measured in consideration of the deviation of the optical axis.
[0045]
In step S14 following step S13, the distance between the vehicles measured in stereo is compared with a preset threshold value to determine whether there is a preceding vehicle that abnormally approaches the host vehicle. If the inter-vehicle distance is larger than the threshold and there is no possibility of collision or contact with the preceding vehicle, there is a preceding vehicle that waits until the next frame and the inter-vehicle distance is too close below the threshold, and there is a possibility of collision or contact. In such a case, a buzzer is similarly sounded to output an alarm to prompt the driver to perform an avoidance operation.
[0046]
In this case, in addition to the inter-vehicle distance between the host vehicle and the preceding vehicle, the relative speed between the host vehicle and the preceding vehicle is calculated, and the possibility of a collision or contact is determined based on the inter-vehicle distance and the relative speed. You may do it.
[0047]
In the second embodiment, when applied to a monitoring device mounted on a vehicle, the vehicle periphery is monitored both in non-stereo view and in stereo view, compared to a conventional device using only the function of stereo ranging. The effective visual field can be expanded, and the preventive safety in avoiding contact / collision with pedestrians and selection vehicles can be enhanced.
[0048]
【The invention's effect】
As described above, according to the present invention, it is possible to obtain accurate distance information from the imaging area for stereo vision, and to obtain a wide effective field of view by utilizing the imaging area for non-stereo vision. Can be detected.
[Brief description of the drawings]
FIG. 1 is an explanatory diagram showing the arrangement of a stereo camera according to a first embodiment of the present invention. FIG. 2 is a block diagram of a wide-field image processing apparatus. FIG. 3 is a flowchart of intruder detection processing. FIG. 4 is an explanatory diagram showing an intruder candidate by non-stereo view. FIG. 5 is an explanatory diagram showing confirmation of an intruder by stereo view. FIG. 6 is related to the second embodiment of the present invention and is stereo. FIG. 7 is a block diagram of the front / side monitoring apparatus. FIG. 8 is a flowchart of the front / side monitoring process.
DESCRIPTION OF SYMBOLS 1 Stereo camera 2 Stay 3 Rotating shaft 4 Drive motor 8 Stereo matching circuit 10 Microcomputer 10a Image recognition process part 11 Rotation control circuit 20 Wide-field image processing apparatus

Claims (6)

A set of cameras having imaging fields of view that overlap each other, and a stereo camera in which the optical axis of one camera and the optical axis of the other camera are arranged non-parallel,
Non-stereo processing means for detecting the presence of an object in a captured image of each camera of the stereo camera;
Stereo processing means for detecting the presence of the object and the distance to the object in a region where the imaging fields of the pair of images captured by the stereo camera overlap;
A stereo wide-field image processing apparatus comprising: A camera rotation mechanism for rotating the stereo camera;
Camera rotation control means for controlling the camera rotation mechanism so that the object detected by the non-stereo processing means is reflected in the processing target range of the stereo processing means;
The stereo wide-field image processing apparatus according to claim 1, further comprising: The camera rotation mechanism is
3. The stereo wide-field image processing apparatus according to claim 2, wherein the stereo wide-field image processing apparatus is a mechanism that rotates the entire stereo camera while maintaining a relative position of each camera. The non-stereo processing means is:
4. A stereo wide-field image processing apparatus according to claim 1, wherein a new image is detected by comparing a background image in a state in which no moving object exists with a current image. . The camera rotation control means includes:
Select one of a preset number of rotation angles to control the camera rotation mechanism,
The non-stereo processing means is:
The background image corresponding to the current camera position is selected from the background images stored separately corresponding to each of the finite number of rotation angles, and compared with the current image. Item 5. A stereo wide-field image processing device according to Item 4. The non-stereo processing means is:
The stereo wide-field image processing device according to claim 1, wherein an object moving in a specific direction is detected based on an optical flow between captured images.

Images