JP6241083B2 - Imaging apparatus and parallax detection method - Google Patents

Description

  The present invention relates to a stereo imaging device that images the same measurement object from different viewpoints, and a parallax detection method in the imaging device.

In recent years, a three-dimensional object recognition device has been developed that recognizes a pile of components individually and recognizes the position and orientation of each component in order to enable accurate operation of the components by a robot arm on a production line. Yes.
As one technique for measuring an object three-dimensionally, a stereo method is known in which two cameras capture the same measurement object from different viewpoints.
A pixel block that correlates with a pixel block in the image of one camera is identified in the image of the other camera, and distance data is calculated using the principle of triangulation from the parallax that is the relative shift amount in both images It is.
An imaging device including a pair of cameras used for the stereo method is also called a stereo camera.

In the stereo camera, the distance information to the measurement target can be obtained from the parallax between the two cameras.
One of the images captured by each camera is used as a reference image and the other as a comparison image. In the comparison image, a scan operation (corresponding point search) is performed using an infinite correspondence point as a reference point, and the corresponding point of the comparison image is searched. Parallax is detected.
The larger the parallax, the longer the scanning range and the larger the processing content.

In Patent Document 1, the center of the light receiving region of the imaging element is set with respect to the center of the imaging range of the optical lens for the purpose of securing a necessary and sufficient distance information generation range without impairing the left-right symmetry of the stereo camera. An offset configuration is described.
Specifically, with a configuration that is offset by a predetermined amount in the direction from the main camera to the sub camera on the stereo baseline, substantially the same effect as widening the right field of view by tilting the camera optical axis of the sub camera to the right It is obtained by offset.
4 and 5 of Patent Document 1 describe a configuration in which the center of the light receiving region of the imaging element is offset with respect to the center of the imaging range of the optical lens in the sub camera.
FIG. 6 of the same document describes a configuration in which the offset is performed in both the main camera and the sub camera for the purpose of sharing parts.
According to the configuration of Patent Document 1, it is possible to expand the field of view (measurable area) particularly in the measurement of a short distance.

In the corresponding point search in Patent Document 1, a corresponding matching search range is set on the comparison image for a small region that is a correlation source on the reference image.
In the coincidence search range, the correlation with the correlation source is evaluated for each correlation destination candidate while sequentially shifting the small area as the correlation destination candidate one pixel at a time.
As shown in FIG. 3 of Patent Document 1, the matching search range is basically set to be horizontally long with reference to the point corresponding to infinity.
In other words, scanning is performed while shifting the small area in the horizontal direction using the point corresponding to infinity as a reference point.

As described above, in the method described in Patent Document 1, the field of view can be expanded, but since the corresponding point search is performed using the corresponding point at infinity as a reference point, it is inevitable that the content of the arithmetic processing increases. It was.
For this reason, there was a problem that the operation speed was slow.

  The present invention has been created in view of such a current situation, and its main object is to provide an imaging device capable of reducing the content of arithmetic processing and performing high-speed processing operation while ensuring a sufficient field of view. .

To achieve the above object, the present invention provides a camera having an optical lens and an imaging device is provided with a pair at predetermined intervals, the region where the imaging area of each camera overlap is viewing, of the respective cameras The parallax between the cameras is detected by searching the corresponding point of the comparison image while shifting the corresponding point projected on the imaging region and the periphery of the corresponding point in the predetermined interval direction, and triangulation is performed from the parallax. of an imaging apparatus of a stereo system for calculating the distance data by the image processing means by using the principle with respect to the optical axis of the optical lens, is offset to the center of the image sensor in the predetermined interval direction, the based on the corresponding points identified in one of the camera image among the image taken by each camera and have your to other camera of the comparison image, the offset the corresponding points projected on the comparison image Characterized in that a point which has a reference point of the corresponding point search start.

  According to the present invention, a sufficient field of view can be secured, and a high-speed processing operation can be performed without increasing the cost associated with arithmetic processing.

1 is a schematic plan view showing a configuration of an imaging apparatus according to an embodiment of the present invention. It is a schematic plan view which shows the structure of each camera. It is a figure which shows the difference in the width of a visual field when not offsetting the case where an imaging center is offset. It is a figure which shows the conventional stereo camera structure and the processing content of corresponding point search. It is a figure which shows the structure of the imaging device in this embodiment, and the processing content of corresponding point search. It is a figure which shows the processing content of a corresponding point search at the time of offsetting only the left camera. It is a figure which shows the processing content of a corresponding point search at the time of offsetting only a right camera. It is a figure explaining the ranging principle of a stereo camera. It is a control block diagram of an imaging device.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a schematic plan view showing the configuration of the imaging apparatus according to the present embodiment. The stereo imaging device 2 includes a left camera 4 and a right camera 6 as a pair of cameras.
The left camera 4 and the right camera 6 are arranged with a baseline length B (160 mm) as a predetermined interval.
As shown in FIG. 2, each camera includes optical lenses 8 a and 8 b and imaging elements (hereinafter also simply referred to as “sensors”) 10 a and 10 b, and the base member length is maintained in the stay member 12. It is fixed integrally.
The optical lens 8 is a lens having a focal length f: 3.6 mm.
The image sensor 10 is composed of sensors having a sensor size H (horizontal): 4.8 mm and a pixel pitch: 0.00375 mm.

As shown in FIG. 1, the left and right cameras 4, 6 are opposite to each other in the baseline direction (predetermined spacing direction) with respect to the optical center as the optical axis of the optical lens. Are offset by the same amount.
That is, the left camera 4 is offset in the −x direction (left) and the right camera 6 is offset in the + x direction (right) by an offset amount S: 50 pix, respectively.
The offset amount S is an amount by which the image sensor 10 after the offset is within the imaging range of the optical lens 8.
The search for corresponding points at the time of parallax detection is based on the offset point, and the search width is set to 64 pix.
With the above stereo camera configuration, it is possible to measure a distance L of about 1 m to 1.5 m with a 6-bit width, and the field of view X is about 1200 mm at 1 m and about 1900 mm at 1.5 m. Can do.

The configuration of the imaging apparatus 2 according to the present embodiment and the processing content of the corresponding point search will be described in comparison with the conventional configuration.
As shown in FIG. 3A, a general stereo camera is configured such that the optical center coincides with the imaging center.
In that case, a region where the image regions of the left and right cameras overlap is a field of view as a stereo camera, that is, a measurable range.
For example, when using a stereo camera with a horizontal angle of view ω: 10 deg and a base length B: 100 mm of the optical lens as a measurement target, the measurable range is ± 252.6 mm from the camera center.

On the other hand, in the present invention, the left camera is offset by -x in the left direction and the right camera is shifted by + x in the baseline direction (left and right direction) with respect to the optical center.
By offsetting the imaging centers of the left and right cameras, the image areas of the left and right cameras are shifted to the center of the camera as shown in FIG.
As a result, the field of view of the imaging device 2 as a stereo camera can be obtained at a shorter distance, and a wider measurable range can be obtained even when the measurement distance is the same distance as compared to the conventional case.

For example, when the imaging center of the left and right cameras is offset by 0.3 mm (sensor horizontal length: 2.4 mm) with respect to the above-described stereo camera (lens horizontal angle of view ω: 10 deg, baseline length B: 100 m), It becomes street.
That is, the measurable range at a measurement distance of 1 m is ± 296.7 mm from the camera center, and the measurable range of 80 mm or more can be expanded over the entire image.
In the stereo camera, the distance information to the target can be obtained from the parallax between the two cameras.

In order to detect parallax, it is necessary to correctly check the correspondence between the pixels in the left and right images.
This is called corresponding point search. 4 and 5 show schematic diagrams of the corresponding point search.
When the object A is at a certain distance, the corresponding points L and R are projected on the left and right camera sensors (imaging devices).
If the right camera is considered as a reference now, a small calculation area (window) is assumed around the corresponding point R projected on the image of the right camera.
Corresponding points are detected by taking a window of the same size in the left image (comparison image) and comparing it with the reference image while shifting the window, that is, scanning.

There are various comparison methods at this time. For example, the most basic method is SSD (Sum of Squared Difference).
In that case, the sum of squares of the difference of each pixel is calculated, and the point at the minimum within the search range is detected as the corresponding point.
As described above, in the corresponding point search, the corresponding points are searched by sequentially scanning the windows in the comparison image.

That is, if the parallax is large, the scanning range is correspondingly long, and the content of the arithmetic processing is large.
In a general stereo camera, as shown in FIG. 4, the optical center coincides with the imaging center, and the point corresponding to infinity is the reference point.
A window is taken with reference to the corresponding point R confirmed in the reference image (FIG. 4E), the search start position of the comparison image is set to the infinity corresponding point, and the necessary search width is scanned therefrom (FIG. 4D). ). The parallax between the cameras is the length of D.

On the other hand, in the present invention, as described above, the imaging center is offset. As shown in FIG. 5, an offset point is set as a reference point using the fact that the imaging center is offset.
That is, the necessary search width is scanned from the offset point as the reference point without using the point corresponding to infinity as the reference.
In other words, the corresponding point confirmed in the reference image is determined as it is as the corresponding point start position in the comparison image (FIG. 5D). The parallax is D1 <D.
For example, in the above-described stereo camera (lens horizontal angle of view ω: 10 deg, baseline length B: 100 m), when measuring a distance of 1.5 m with a general stereo camera in which the optical center and the imaging center coincide with each other, The parallax is 242 pix (sensor pitch: 3.75 μm).

On the other hand, when the sensor is offset by 0.3 mm, the reference point is offset by 80 pix and the corresponding point of the comparative image is also offset by 80 pix, resulting in a parallax of 82 pix.
When outputting a parallax value with a bit width, an 8-bit bit width is required for 242 pix, but a parallax value can be output with a 7-bit bit width for 82 pix.

When this is mounted on a circuit, it is directly connected to the bit width of the output parallax data.
That is, the circuit scale can be reduced by reducing the parallax.
By setting the offset point as the corresponding point search start position as it is, a special parallax calculation circuit is not necessary and a general-purpose product can be used.
In addition, since the required search width is reduced to half or less because the parallax value itself is reduced, the calculation processing time is significantly shortened.

As described above, in the present invention, a sufficient field of view (measurable region) can be secured by offsetting the imaging center with respect to the optical center, and the calculation processing time can be significantly shortened.
In the present embodiment, as described above, the imaging centers of both the cameras 4 and 6 are offset by the same amount in the opposite direction.
This condition is a pattern that allows the widest field of view and the shortest parallax.
The present invention is not limited to this pattern, and even if the imaging center is offset in either one of the pair of cameras 4 and 6, the calculation processing time can be shortened as compared with the conventional configuration.

FIG. 6 shows an example in which the imaging center is offset only by the camera 4. The window is omitted. Although the parallax D2 is longer than the above example offset by both of the pair of cameras, it is shorter than the parallax D in the conventional configuration (D2 <D).
FIG. 7 shows an example in which the imaging center is offset only by the camera 6. In this case, the same parallax length can be obtained.
6 and 7, illustrations corresponding to (e) of FIGS. 4 and 5 are omitted.

The distance measurement principle (triangulation principle) of a stereo camera will be briefly described.
As shown in FIG. 8, the subject image a1 obtained through the optical lens 8b of the right camera 6 and the subject image a2 obtained through the optical lens 8a of the left camera 4 are shifted by the parallax D from the same point on the object A. Then, each image is formed on the surface of the image pickup area of the image pickup device 10 and received by a plurality of light receiving elements (pixels) to be converted into electric signals.
The parallax is D, the distance (baseline length) between the optical axes of the optical lenses 8a and 8b as distance measuring lenses is B, the distance between the optical lens 8 and the object A is L, and the focal length of the optical lens 8 is fa. When L >> fa, the following formula (1) is established.
L = B · fa / D Formula (1)
Since B and fa are known, the distance (distance data) L can be calculated by calculating the parallax D.

FIG. 9 is a hardware block diagram of the imaging apparatus 2.
The left camera 4 and the right camera 6 are installed with different baseline directions and horizontal directions.
The imaging device 2 has image processing means 14. The image processing unit 14 includes an image correction unit 16 corresponding to the left camera 4, an image correction unit 18 corresponding to the right camera 6, a correction parameter storage unit 20, and a parallax calculation unit 22.
The image correction unit 16 and the image correction unit 16 are portions that perform image correction according to the correction parameters.

The correction parameter storage unit 20 records correction parameters used by the image correction units 16 and 18, and is composed of a nonvolatile memory.
The parallax calculation unit 22 calculates the parallax from the two images and outputs a parallax image. The parallax calculation unit 22 is also a main body that executes corresponding point search.

With the above configuration, the images captured by the left camera 4 and the right camera 6 are geometrically corrected by the image correction units 16 and 18, respectively, according to the parameters stored in the correction parameter storage unit 20.
The RAM is not used for distortion correction. That is, the distortion correction processing in this embodiment uses a method using a distortion correction formula instead of the distortion correction processing using the reference table.

The occurrence of distortion is irregular and there are many patterns, but the distortion that appears most commonly is a radially symmetric distortion due to the symmetry of the optical lens.
Radiation distortion can be generally classified into two types (pincushion distortion and barrel distortion).
These distortion patterns can be corrected by geometric transformation based on the distortion model.
For example, a distortion correction model function (distortion correction formula) for a two-dimensional optical system is expressed by a polynomial function as shown in the following formula (2), for example.

However, in the equation (2), (x, y) is the coordinates of the image signal to be processed, and D _x (x, y) is the polynomial geometry (coordinate) conversion equation of the coordinate system of each of x and y. It is.
c _pq and d _pq are coefficients of a polynomial function, N _x and N _y are polynomial orders, and p and q are polynomial function terms.

The coefficients of the polynomial are obtained in advance by a known calibration operation that is performed, and the distortion correction formulas of the individual imaging devices differ in the coefficients in the polynomial function.
The polynomial function can be calculated using the following forward difference operator, for example.
However, in this forward difference operator, f ⁿ is the value of the function f at the n ^th point, Δf ⁿ is the primary forward difference of the function at the n ^th point, and Δ ^k f _n is the n ^th point. Is the k-th order forward difference of the function at.

  By correcting to images captured by two cameras in which the baseline direction and the horizontal direction coincide with each other in a pseudo manner, the parallax in the horizontal direction may be calculated, and an accurate parallax image can be output.

  In the above-described embodiment, the imaging center is physically offset with respect to the optical center. However, even if the imaging center is virtually offset on the software of the image processing unit 14, the same effect as described above can be obtained. .

4 Left Camera as Camera 6 Right Camera as Camera 8a, 8b Optical Lens 10a, 10b Image Sensor 14 Image Processing Means

JP 2008-40115 A

Claims (6)

A pair of cameras having an optical lens and an image sensor is provided at a predetermined interval, and an area where the imaging areas of each camera overlap is a visual field,
Among the cameras, by detecting the corresponding points of the comparison image while shifting the corresponding points projected on the imaging region and the periphery of the corresponding points in the predetermined interval direction, the parallax between the cameras is detected,
A stereo imaging device that calculates distance data by image processing means using the principle of triangulation from the parallax ,
Wherein the optical axis of the optical lens, is offset to the center of the image sensor in the predetermined interval direction,
Wherein based on the corresponding points identified in one of the camera image among the image taken by each camera and have your to other camera of the comparison image and the corresponding points which are projected on the comparison image is said offset An imaging apparatus , wherein a point is used as a reference point for starting the corresponding point search. The imaging device according to claim 1,
An image pickup apparatus characterized in that the center offset of the image pickup device is virtually made on software of the image processing means. The imaging device according to claim 1 or 2 ,
The image pickup apparatus, wherein the offset is offset by the same amount in opposite directions in the predetermined interval direction so that the field of view can be obtained at a closer distance . In the imaging device according to any one of claims 1 to 3,
The amount of the offset is an amount by which the image sensor after the offset is within the imaging range of the optical lens. A pair of cameras having an optical lens and an image sensor is provided at a predetermined interval, and an area where the imaging areas of each camera overlap is a visual field,
Detecting the parallax while shifting the corresponding point projected on the image of each camera and the periphery of the corresponding point in the predetermined interval direction ;
In a parallax detection method in a stereo imaging device that calculates distance data from the parallax using the principle of triangulation,
The center of the image sensor is offset in the predetermined interval direction with respect to the optical axis of the optical lens ,
Wherein based on the corresponding points identified in one of the camera image among the image taken by each camera and have your to other camera of the comparison image and the corresponding points which are projected on the comparison image is said offset A parallax detection method characterized by using a point as a reference point for starting the corresponding point search. The parallax detection method according to claim 5,
The parallax detection method characterized in that the center offset of the image sensor is virtually made on the software of the image sensor.

Images