JP2012002683A - Stereo image processing method and stereo image processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To acquire three-dimensional information of an object with high accuracy at a high speed in accordance with the distance between the object and an imaging device.SOLUTION: A stereo image processing device includes an imaging device for imaging an object, an image acquisition part for acquiring a plurality of high-resolution images formed by imaging the same object from a plurality of directions, a low-resolution image generation part for generating a low-resolution image, a correspondence point search part for searching for a correspondence point on a compared image corresponding to a point on reference image, a three-dimensional image calculation part for calculating three-dimensional information based on the search result of the correspondence point, a distance range determination part for determining a distance range of search point on the high-resolution images from the three-dimensional information based on the low-resolution image, a window size determination part for determining a search window size in the high-resolution image based on the distance range, and a search range limitation part for limiting a search range of the correspondence points on the high-resolution images based on a parallax range determined by the distance range.

Description

  The present invention relates to a stereo image processing method and a stereo image processing apparatus that enable high-speed and high-precision calculation of three-dimensional data when an object is three-dimensionally measured with a stereo image.

  Conventionally, a stereo method using the principle of triangulation is known as a method for acquiring three-dimensional data of a predetermined object. In the stereo method, the same object is photographed from two or more viewpoints (different line-of-sight directions) using a plurality of cameras, and the position of the measurement object in the three-dimensional space is calculated.

  FIG. 13 shows an outline of the stereo method. As shown in this figure, in the stereo method, the coordinates (u) on the reference image of the same point in the three-dimensional space are obtained by two images of a reference image and a comparison image obtained by imaging the same object from different viewpoints C and C ′. , v) and the coordinates (u ′, v ′) on the comparison image are associated, and the three-dimensional coordinates (X, Y, Z) of the target point to be measured are obtained based on the principle of triangulation.

For example, a distance image generation apparatus described in Patent Document 1 is known as an apparatus that generates three-dimensional data (distance image) of a predetermined object using a stereo method.
FIG. 14 shows a distance image generating apparatus described in Patent Document 1. The distance image generation device described in Patent Literature 1 includes imaging devices 2a and 2b that acquire a base image and a comparison image (reference image), and a plurality of hierarchical images having different resolutions for the base image and the comparison image (reference image). The multi-resolution image generation unit 13 to be generated, the SAD (Sum of Absolute Difference) calculation unit 6 that performs corresponding point search for at least the lowest resolution hierarchical image, and the corresponding point search for at least the highest resolution hierarchical image. A POC (Phase-Only Correlation) calculation unit 7 is provided, the corresponding point search is started from the low-resolution hierarchical image, the corresponding point search is sequentially performed on the hierarchical image having the higher resolution, and the corresponding points are sequentially reached until the highest-resolution hierarchical image. Corresponding point search unit 15 that repeats the search, search region setting unit 14 that sets a search region for the next layer image based on the search result for the immediately previous layer, search result And a distance image generation unit 16 for generating a distance image based on.

JP 2008-216126 A

  Conventionally, because of problems such as processing speed of the processing apparatus, there are many purposes for obtaining three-dimensional data of an object within a limited distance range. In the process of searching for corresponding points, a reference image A predetermined area (hereinafter referred to as a window size) is selected as a reference area, and the reference area is compared with a similar area (comparison area) having the same window size in the comparison image. I was searching.

  However, if the distance from the object to the imaging device varies greatly depending on the object, such as when acquiring 3D data of piled parts or acquiring 3D data of products flowing on the conveyor, the window size is set to the distance. If the window size is adjusted to an object that is far away, the difference in characteristics between the reference area and the comparison area is reduced in an object that is close to the object, and the search accuracy for corresponding points is reduced. On the other hand, if the window size is a window size that matches an object with a short distance, the processing time required for the search increases because the window size is larger than necessary for an object with a long distance.

  Therefore, the present invention has been made in view of the above problems, and acquires three-dimensional information of an object with high accuracy and high speed according to the distance between the object and an imaging device.

In order to solve the above problems, a stereo image processing method according to the present invention includes:
An image acquisition step of acquiring a plurality of captured images obtained by imaging the same object from a plurality of directions;
A low-resolution image generation step of generating a low-resolution image from the captured image;
One of the low-resolution images is a first reference image, the other is a first comparison image, and a first reference area including a first reference coordinate point at a first search window size determined in advance from the first reference image. And selecting a plurality of first comparison areas from the first comparison image at the first search window size, obtaining a similarity relationship between the first reference area and the first comparison area, and based on the similarity relation A first corresponding point searching step of searching for a first corresponding coordinate point on the first comparison image corresponding to the first reference coordinate point;
A first three-dimensional information calculation step of calculating three-dimensional information of the low-resolution image based on a search result of the first corresponding point search step;
A distance range determining step for determining a distance range from the imaging device that calculates the three-dimensional information of the captured image from the three-dimensional information of the low-resolution image;
A window size determining step for determining a second search window size based on the distance range;
A search range limiting step of limiting the search range of the second corresponding coordinate point based on the parallax range determined by the distance range;
One of the captured images is a second reference image, the other is a second comparison image, and a second reference coordinate point is included in the second search window size from an area within the distance range of the second reference image. A reference area is selected, a plurality of second comparison areas are selected from the area within the search range of the second comparison image with the second search window size, and the second reference area and the second comparison area A second corresponding point searching step for obtaining a similarity relationship and searching for the second corresponding coordinate point on the second comparison image corresponding to the second reference coordinate point based on the similarity relationship;
A second three-dimensional information calculation step of calculating three-dimensional information of the captured image based on a search result of the second corresponding point search step;
It is characterized by being processed from.

Furthermore, the stereo image processing apparatus according to the present invention is
An imaging device for imaging an object;
An image acquisition unit that acquires a plurality of captured images obtained by capturing the same object from a plurality of directions in the imaging device;
One of the low-resolution images is a first reference image, the other is a first comparison image, and a first reference area including a first reference coordinate point at a first search window size determined in advance from the first reference image. And selecting a plurality of first comparison areas from the first comparison image at the first search window size, obtaining a similarity relationship between the first reference area and the first comparison area, and based on the similarity relation A first corresponding point search unit for searching for a first corresponding coordinate point on the first comparison image corresponding to the first reference coordinate point;
A first three-dimensional information calculation unit that calculates three-dimensional information of the low-resolution image based on a search result of the first corresponding point search unit;
A distance range determination unit that determines a distance range from the imaging device that calculates the three-dimensional information of the captured image from the three-dimensional information of the low-resolution image;
A window size determining unit that determines a second search window size based on the distance range;
A search range limiting unit that limits a search range of the second corresponding coordinate point based on a parallax range determined by the distance range;
One of the captured images is a second reference image, the other is a second comparison image, and a second reference coordinate point is included in the second search window size from an area within the distance range of the second reference image. A reference area is selected, a plurality of second comparison areas are selected from the area within the search range of the second comparison image with the second search window size, and the second reference area and the second comparison area A second corresponding point search unit that obtains a similarity relationship and searches for the second corresponding coordinate point on the second comparison image corresponding to the second reference coordinate point based on the similarity relationship;
A second three-dimensional information calculation unit that calculates three-dimensional information of the captured image based on a search result of the second corresponding point search unit;
It is characterized by having.

  According to the stereo image processing method and the stereo image processing apparatus according to the present invention, the corresponding point search is performed by performing the corresponding point search using the window size corresponding to the distance from the imaging device of the target for acquiring the three-dimensional data. Dimensional data can be acquired at high speed and with high accuracy.

1 is a schematic configuration diagram of an embodiment of a stereo image processing apparatus according to the present invention. It is a figure explaining the search principle of a corresponding point. It is a principle figure of a parallelization process. It is explanatory drawing of matching of a high resolution image and a low resolution image. It is a figure explaining the corresponding point search using a window. It is a figure which shows the difference of the search range by the presence or absence of a search range limitation part. It is a figure showing the positional relationship of an imaging device, a conveyor, and a target object. It is the image which imaged the state of FIG. 7 with the imaging device. It is a figure showing the positional relationship of an imaging device and the piled target object. It is a flowchart of the stereo image processing apparatus which concerns on embodiment of this invention. It is a figure which shows the distance range for recognizing the target object in the uppermost stage, when this embodiment is applied to the stacked target object. It is the figure which showed the case where the distance range was divided | segmented into plurality when this embodiment is applied to the piled target object. It is the schematic explaining the principle of a stereo method. This is a distance image generation device described in Patent Document 1.

Hereinafter, embodiments of a stereo image processing method and a stereo image processing apparatus according to the present invention will be described with reference to the drawings.
FIG. 1 is a schematic configuration diagram of an embodiment of a stereo image processing apparatus according to the present invention.

  As illustrated in FIG. 1, the stereo image processing apparatus 100 includes two imaging devices 201 and 202 that capture an image of an object, and an image processing unit 300. Furthermore, a keyboard 401, a mouse 402, and a display 403 are connected to the image processing unit 300 as an interface unit between the stereo image processing apparatus and the operator.

  The imaging devices 201 and 202 acquire image data obtained by imaging the same object from two different viewpoints. The imaging devices 201 and 202 have grasped imaging device parameters such as a viewpoint position, a line-of-sight direction, and a lens distortion correction value. Note that the imaging device parameter can be obtained by various known methods as a calibration method of the stereo camera and is not a main part of the present invention, and thus the description of the imaging device parameter acquisition method is omitted.

  An image captured by the imaging apparatus 201 is set as a reference image, and an image captured by the imaging apparatus 202 is set as a comparison image to be compared. In this embodiment, two imaging devices are used. However, if the imaging device parameters when the reference image and the comparison image are captured can be grasped, the reference image and the reference image can be obtained from two different viewpoints using one imaging device. A comparative image may be acquired. Furthermore, if the imaging parameters can be grasped, one reference image and two or more comparison images may be acquired. In such a case, three or more imaging devices may be used, or imaging may be performed from three or more different viewpoints with one imaging device.

  The image processing unit 300 is a computer that includes a processing device such as a CPU, MPU, or DSP and a storage device such as a ROM, RAM, or HDD. The image processing unit 300 implements each unit described in detail below by executing a program stored in the storage device by the processing device.

Hereinafter, processing contents of the image processing unit 300 will be described.
The image acquisition unit 310 stores an image captured by the image capturing apparatus 201 as a reference image, and stores an image captured by the image capturing apparatus 202 as a comparative image.

  Note that the reference image and the comparison image can be images that have been subjected to parallelization processing, not the images themselves captured by the imaging devices 201 and 202. The corresponding point search unit 330 described later utilizes the fact that the corresponding point (corresponding pixel) on the comparison image corresponding to a certain point (pixel) on the reference image is on a line called an epipolar line as shown in FIG. The most similar point (pixel) on the epipolar line is searched as a corresponding point (corresponding pixel). Here, as shown in FIG. 3, the reference image and the comparison image are respectively projected onto the plane when the parallel camera is photographed so that the epipolar line is horizontal, and the virtual reference image and the virtual comparison image are generated. Thereby, since a search range can be limited to a horizontal direction, the search of a corresponding point (corresponding pixel) can be made easy. This is image parallelization processing.

  The low resolution image generation unit 320 performs image compression processing to reduce the image data size for the reference image and the comparison image acquired by the image acquisition unit 310, and generates a low resolution image. Here, the reference image and the comparison image acquired by the image acquisition unit 310 are referred to as a high resolution image. For example, if the resolution of a high-resolution image is 1200 × 1600, an image having a resolution of 300 × 400 is generated. At this time, the resolution of the low resolution image is not limited.

  Furthermore, it is assumed that the pixels of the high resolution image and the low resolution image are associated with each other. FIG. 4 is an explanatory diagram of association between a high resolution image and a low resolution image. As shown in FIG. 4, when a 300 × 400 low resolution image is generated from a 1200 × 1600 high resolution image, 4 × 4 pixels of the high resolution image are associated with one pixel of the low resolution image. The pixel (m, n) of the image is associated with 16 pixels of the pixel (4m-3,4n-3)... (4m, 4n) of the high-resolution image.

  The corresponding point search unit 330 uses images of the same resolution, and a reference region (hereinafter, referred to as a reference pixel) including a reference point (reference pixel) at a predetermined size (hereinafter referred to as a window size) from the reference image as shown in FIG. The search window may be referred to), and a similarity relationship with a region of the same window size (hereinafter also referred to as a window) on the epipolar line of the reference point (reference pixel) in the comparison image is obtained, and the similarity The corresponding point (corresponding pixel) of the reference point (reference pixel) is obtained from the relationship. Here, the similarity relationship can be obtained by using a well-known SAD operation method, POC operation method, or the like. For example, the correlation value indicating the similarity between the window of the reference image and the window of the comparison image is obtained, and the center point (pixel) between the windows having the correlation value with the highest similarity is the reference point (reference pixel) and the corresponding point ( Corresponding pixel). Of course, other methods for obtaining corresponding points can be used.

  The corresponding point search unit 330 includes a first corresponding point search unit 331 and a second corresponding point search unit 332. The first corresponding point search unit 331 performs corresponding point search in the low resolution image, and the second corresponding point search unit 332 performs corresponding point search in the high resolution image.

  The first corresponding point search unit 331 selects a search window for the entire region of the reference image using a predetermined window size, and from the similarity between the search window and the window on the epipolar line of the comparison image of the search window, A reference point (reference pixel) and a corresponding point (corresponding pixel) are obtained. Note that the point (pixel) in the vicinity of the outer peripheral edge of the reference image and the comparison image theoretically has no corresponding point (corresponding pixel) corresponding to the reference point (reference pixel). For this reason, the first corresponding point search unit 331 obtains a correspondence relationship of all images (all pixels) excluding a region where there is no corresponding point (corresponding pixel) corresponding to the reference point (reference pixel). However, when the image area where the object is present is limited in advance, the corresponding point (corresponding pixel) of the reference point (reference pixel) of the limited area may be obtained.

  The second corresponding point search unit 332 performs basically the same processing as the first corresponding point search unit 332, but has a function of limiting the reference point (reference pixel) to be searched for and the point that the image to be handled is a high-resolution image. This is different in that it also has a function of limiting the search range of the window in the comparison image, and also has a function of changing the search window size.

  The second corresponding point search unit 332 uses the distance range output from the distance range determination unit 350, which will be described in detail later, with the function of limiting the reference point (reference pixel) to be searched, and the distance range in the high-resolution image. Corresponding point search is performed only for the pixels located at.

  Furthermore, the second corresponding point search unit 332 uses the parallax range output from the search range limiting unit 370 described in detail later by the function of limiting the search range of the window in the comparison image, and the parallax in the high-resolution image. The corresponding point search is performed by calculating the similarity only for the window in the range. Specifically, as shown in FIG. 6, a difference occurs in the search range depending on the presence or absence of the search range limiting unit.

  Still further, the second corresponding point search unit 332 is used when a corresponding point search is performed using the window size output from the window size determination unit 360, which will be described in detail later, with the function of changing the search window size. .

  The three-dimensional information calculation unit 340 calculates the principle of triangulation for each point (pixel) based on the positional deviation amount between the reference point (reference pixel) and the corresponding point (corresponding pixel) obtained by the corresponding point search unit 330. Three-dimensional data is obtained by the stereo method used. Note that the three-dimensional information calculation unit 340 includes a first three-dimensional information calculation unit 341 that calculates three-dimensional data of a low-resolution image and a second three-dimensional information calculation unit 342 that calculates three-dimensional data of a high-resolution image. Have. Note that the first three-dimensional information calculation unit 341 and the second three-dimensional information calculation unit 342 differ only in the handled images, and the processing contents are exactly the same.

  The distance range determination unit 350 grasps the distance range of the object from the three-dimensional data obtained from the low resolution image, and determines the distance range of the reference point (reference pixel) for searching for the corresponding point in the high resolution image. Here, the distance range is a range in the depth direction of the image (hereinafter referred to as the Z direction). In the following description, the Z direction is described as being smaller in the near direction and larger in the far direction, but of course the reverse relationship may be used.

  In order to describe the processing contents of the distance range determination unit 350 in more detail, a description will be given using specific examples of FIGS. 7 and 8. FIG. 7 is a diagram illustrating a positional relationship among the imaging devices 201 and 202, the conveyor, and the objects a, b, and c. FIG. 8 is an image obtained by imaging the state of FIG. Note that three-dimensional data for each point (pixel) is obtained in the image of FIG. 8, the objects a, b, and c are cylinders of the same shape, and the diameter D is grasped in advance. In such a case, the area of the conveyor (pixel area) is excluded by comparing with an image captured in advance without the object on the conveyor, and the areas of the objects a, b, and c are extracted. From the minimum values Za, Zb, Zc in the Z direction for each region of the objects a, b, c and the diameter D, the distance ranges for the objects a, b, c are Zas-Zae, Zbs-Zbe, Zcs-Zce. And decide. Each range is preferably a range larger than D because of the problem of accuracy of the three-dimensional data by the low-resolution image, and Zas <Za, Zbs <Zb, Zcs <Zc. In the specific example, the distance range output by the distance range determination unit 350 outputs one distance range that satisfies a predetermined condition, for example, the distance ranges Zas to Zae of the object closest to the imaging devices 201 and 202. Alternatively, the distance ranges Zas to Zae, Zbs to Zbe, and Zcs to Zce may be sequentially output. In any case, various modifications are possible as long as the object region is specified from the three-dimensional data obtained based on the low-resolution image and the distance range of the specified object is determined.

  The window size determination unit 360 determines the window size used for the search process of the corresponding point search unit 330 according to the distance range determined by the distance range determination unit 350. The window size in the corresponding point search should be determined according to the size and texture of the object in the image. However, it is not necessary to change the window size as long as the object exists within a predetermined distance range. However, since the distance range of the objects on the conveyor in FIG. 7 and the distance range of the stacked objects in FIG. 9 vary greatly for each object, the window size is appropriate if the window size is constant. It is possible to search for corresponding points with high accuracy in the distance range, but in the distance range where the window size is inadequate, the corresponding point search accuracy is remarkably deteriorated or the corresponding points are searched in more detail than originally intended. This increases the search time. Therefore, the window size is changed and determined in accordance with the distance in the Z direction where the corresponding point search is performed.

  Various methods can be adopted as the direction of determining the window size. For example, the Z direction is divided into a plurality of ranges, and a window size is determined for each divided range. The window size is preferably determined so that the window size increases as the Z direction decreases. The window size used by the corresponding point search unit 330 can be a window size corresponding to the distance range determined by the distance range determination unit 350. Alternatively, the distance range determined by the distance range determination unit 350 may be associated with the window size by a function, and the window size may be output when the distance range determined by the distance range determination unit 350 is input. For example, the distance range and the window size can be related by a proportional function. Of course, the function is not limited to proportional, but various functions can be used.

  The search range limiting unit 370 obtains a range of parallax that occurs in the reference image and the comparison image from the distance range determined by the distance range determination unit 350, and in the comparison image of the corresponding point search unit 330 based on the obtained parallax range. Limit the search range of the window.

  In order to explain the processing contents of the search range limiting unit 370 in detail, a specific example will be used. For example, consider a corresponding point search only within a certain distance range. Here, when the reference image and the comparison image are images that have been subjected to parallelization processing, the difference between the x coordinates of the corresponding points becomes the parallax d. When the distance in the Z direction at the point (pixel) P (u, v) on the image is ZP, the value of the parallax d (ZP) is uniquely determined from the imaging device parameters, and the corresponding point (corresponding pixel) is P ′. (u + d (ZP), v). That is, for the reference image point (pixel) P (u, v) where the distance range is Zmin <Z <Zmax, the parallax d is d (Zmin) <d <d (Zmax). When the corresponding point (corresponding pixel) of the comparison image corresponding to the reference point (reference pixel) P (u, v) of the reference image is P ′ (u ′, v), the corresponding point (corresponding pixel) P ′ is the x coordinate. The value can be limited to the range of u + d (Zmin) <u ′ <u + d (Zmax). That is, in the conventional corresponding point search, as shown in FIG. 6, the comparison image is searched from end to end, but in the present embodiment, the corresponding point search range can be limited to the range grasped from the parallax from the distance range. .

Next, a processing flow of the stereo image processing apparatus will be described with reference to FIG.
First, in the image acquisition step S10, a reference image and a comparative image are acquired using the imaging devices 201 and 202. The image acquisition step S10 is a process performed by the image acquisition unit 310 in the stereo image processing apparatus 100. In the image acquisition step S10, the images captured by the imaging devices 201 and 202 are parallelized to generate a virtual reference image and a virtual comparison image, and the virtual reference image is stored as a reference image and the virtual comparison image is stored as a comparison image. You may do it.

  In the low resolution image generation step S20, an image compression process is performed from the reference image and the comparison image acquired in the image acquisition step S10 to generate a low resolution image. Note that a low-resolution image is generated for each of the reference image and the comparison image. The low-resolution image generation step S20 is a process performed by the low-resolution image generation unit 320 in the stereo image processing apparatus 100. Hereinafter, the image acquired in the image acquisition step S10 is referred to as a high resolution image. Note that when generating the low resolution image, the pixels of the high resolution image are associated with the pixels of the low resolution image.

  The first corresponding point search step S30 uses the low resolution reference image generated in the low resolution image generation step S20 and the low resolution comparison image to perform a low resolution comparison corresponding to the reference point (reference pixel) of the low resolution reference image. A corresponding point (corresponding pixel) of the image is obtained. The first corresponding point searching step S30 is a process performed by the first corresponding point searching unit 331 in the stereo image processing apparatus 100.

  In the first corresponding point search step S30, in principle, corresponding points are searched for all the pixels of the low-resolution reference image, and the search range of the window in the comparative image is in principle all that is on the epipolar line. The range is set, and the window size is a predetermined size. This is because three-dimensional data is not acquired in the low resolution image.

  In the first three-dimensional information calculation step S40, the amount of positional deviation between the reference point (reference pixel) of the low resolution image obtained in the first corresponding point search step S30 and the corresponding point (corresponding pixel) of the low resolution image. 3D data is obtained for each point (pixel) by the stereo method using the principle of triangulation. Thereby, three-dimensional data for each pixel of the low resolution image can be acquired. The first three-dimensional information calculation step S40 is a process performed by the first three-dimensional information calculation unit 341 in the stereo image processing apparatus 100.

  The distance range determination step S50 grasps the distance range of the object from the three-dimensional data of the low-resolution image obtained in the first three-dimensional information calculation step S40, and searches for corresponding points in the high-resolution image ( The distance range of the reference pixel) is determined. The distance range determination step S50 is a process performed by the distance range determination unit 350 in the stereo image processing apparatus 100.

  In the window size determining step S60, the window size used for the search process in the second corresponding point searching step S80 is determined according to the distance range determined in the distance range determining step S50. The window size determination step S60 is a process performed by the window size determination unit 360 in the stereo image processing apparatus 100.

  The search range limiting step S70 obtains a range of parallax generated in the high resolution reference image and the high resolution comparison image from the distance range determined in the distance range determination step S50, and in the second corresponding point search step S80. The search range of the window is limited by the high-resolution comparison image. The search range limiting step S70 is a process performed by the search range limiting unit 370 in the stereo image processing apparatus 100.

  The second corresponding point search step S80 selects a window centered on the reference point (reference pixel) of the high-resolution reference image, searches for the most similar window on the epipolar line of the reference point in the high-resolution comparison image, The center point (pixel) of the most similar window is obtained as the corresponding point (corresponding pixel) of the reference point (reference pixel). However, the reference point (reference pixel) of the high-resolution reference image is only the reference point (pixel) corresponding to the distance range obtained in the distance range determination step S50. As the window size, the window size obtained in the window determination step S60 is used. Further, the window search range of the high resolution comparison image is the range obtained in the search range limiting step S70. The second corresponding point search step S80 is a process performed by the second corresponding point search unit 332 in the stereo image processing apparatus 100.

  In the second three-dimensional information calculation step S90, the position between the reference point (reference pixel) of the high-resolution reference image obtained in the second corresponding point search step S80 and the corresponding point (corresponding pixel) of the high-resolution comparison image. Based on the deviation amount, three-dimensional data is obtained for each point (pixel) by a stereo method using the principle of triangulation. Thereby, three-dimensional data for each pixel of the high-resolution image can be acquired. The second three-dimensional information calculation step S90 is a process performed by the second three-dimensional information calculation unit 341 in the stereo image processing apparatus 100.

  For example, when the present embodiment is applied to stacked objects, the Z-direction distance ZT of the object at the top of the stack is obtained from three-dimensional data based on a low-resolution image as shown in FIG. The distance ranges Zs to Ze are determined based on the distance and the size of the object, and the three-dimensional data of the object can be obtained with high accuracy only for the object near the uppermost stage based on the distance range.

  Further, as shown in FIG. 12, the Z-direction distance ZT of the object at the top of the stack and the Z-direction distance ZB of the bottom of the stack are obtained from the three-dimensional data by the low resolution image, and the Z-direction distance ZT, Divide the area piled up based on ZB and the size of the object into distance ranges Z1 to Z2, Z2 to Z3, Z3 to Z4, determine the distance range to search for corresponding points, and only the objects in the determined distance range The three-dimensional data of the object can be acquired with high accuracy.

  As described above, in this embodiment, after obtaining a rough distance in the Z direction from the low-resolution image, a distance range of a specific object of the low-resolution image is obtained, and only the reference point (reference pixel) corresponding to the distance range is obtained. Since the search for the corresponding points (corresponding pixels) is limited to the search range corresponding to the distance range, high-speed processing is possible, and the search window is made the distance range by using an appropriate window size according to the distance. High-accuracy search for corresponding points that is not dependent on one another has been made possible.

DESCRIPTION OF SYMBOLS 100 Stereo image processing apparatus 201,202 Imaging device 300 Image processing part 401 Keyboard 402 Mouse 403 Display

Claims (7)

An image acquisition step of acquiring a plurality of captured images obtained by imaging the same object from a plurality of directions;
A low-resolution image generation step of generating a low-resolution image from the captured image;
One of the low-resolution images is a first reference image, the other is a first comparison image, and a first reference area including a first reference coordinate point at a first search window size determined in advance from the first reference image. And selecting a plurality of first comparison areas from the first comparison image at the first search window size, obtaining a similarity relationship between the first reference area and the first comparison area, and based on the similarity relation A first corresponding point searching step of searching for a first corresponding coordinate point on the first comparison image corresponding to the first reference coordinate point;
A first three-dimensional information calculation step of calculating three-dimensional information of the low-resolution image based on a search result of the first corresponding point search step;
A distance range determining step for determining a distance range from the imaging device that calculates the three-dimensional information of the captured image from the three-dimensional information of the low-resolution image;
A window size determining step for determining a second search window size based on the distance range;
A search range limiting step of limiting the search range of the second corresponding coordinate point based on the parallax range determined by the distance range;
One of the captured images is a second reference image, the other is a second comparison image, and a second reference coordinate point is included in the second search window size from an area within the distance range of the second reference image. A reference area is selected, a plurality of second comparison areas are selected from the area within the search range of the second comparison image with the second search window size, and the second reference area and the second comparison area A second corresponding point searching step for obtaining a similarity relationship and searching for the second corresponding coordinate point on the second comparison image corresponding to the second reference coordinate point based on the similarity relationship;
A second three-dimensional information calculation step of calculating three-dimensional information of the captured image based on a search result of the second corresponding point search step;
A stereo image processing method comprising: An imaging device for imaging an object;
An image acquisition unit that acquires a plurality of captured images obtained by capturing the same object from a plurality of directions in the imaging device;
One of the low-resolution images is a first reference image, the other is a first comparison image, and a first reference area including a first reference coordinate point at a first search window size determined in advance from the first reference image. And selecting a plurality of first comparison areas from the first comparison image at the first search window size, obtaining a similarity relationship between the first reference area and the first comparison area, and based on the similarity relation A first corresponding point search unit for searching for a first corresponding coordinate point on the first comparison image corresponding to the first reference coordinate point;
A first three-dimensional information calculation unit that calculates three-dimensional information of the low-resolution image based on a search result of the first corresponding point search unit;
A distance range determination unit that determines a distance range from the imaging device that calculates the three-dimensional information of the captured image from the three-dimensional information of the low-resolution image;
A window size determining unit that determines a second search window size based on the distance range;
A search range limiting unit that limits a search range of the second corresponding coordinate point based on a parallax range determined by the distance range;
One of the captured images is a second reference image, the other is a second comparison image, and a second reference coordinate point is included in the second search window size from an area within the distance range of the second reference image. A reference area is selected, a plurality of second comparison areas are selected from the area within the search range of the second comparison image with the second search window size, and the second reference area and the second comparison area A second corresponding point search unit that obtains a similarity relationship and searches for the second corresponding coordinate point on the second comparison image corresponding to the second reference coordinate point based on the similarity relationship;
A second three-dimensional information calculation unit that calculates three-dimensional information of the captured image based on a search result of the second corresponding point search unit;
Stereo image processing apparatus with   The stereo image processing device according to claim 2, wherein the window size determination unit decreases the search window size as the distance range is farther from the imaging device.   The stereo image processing device according to claim 2, wherein the distance range determination unit determines one or more distance ranges from the three-dimensional information of the low-resolution image. The distance range determination unit specifies the distances of the plurality of objects from the three-dimensional information of the low-resolution image, and determines the plurality of distance ranges from the size of the object for each distance of the specified target parts. To
The stereo image processing apparatus according to claim 4. The distance range determination unit specifies the distance of the object from the three-dimensional information of the low-resolution image, and determines the distance range from the distance of the specified object and the size of the object.
The stereo image processing apparatus according to claim 4. The distance range determining unit specifies the distance of the target object that is closest in distance from the three-dimensional information of the low-resolution image, and determines the distance range from the distance of the specified target object and the size of the target object. To
The stereo image processing apparatus according to claim 6.