JP5582572B2 - Image processing method, image processing program, computer-readable storage medium storing the same, and image processing apparatus - Google Patents

Description

  The present invention relates to an image processing method, an image processing program, a computer-readable storage medium storing the image processing program, and an image processing apparatus.

  Image recognition technology is prevalent in the FA field and the like. One of the image recognition techniques is the stereo vision method. The stereo vision method is a technology that restores three-dimensional information using images taken from a plurality of cameras with different viewpoints, and can measure the three-dimensional shape, dimensions, position and orientation of a measurement object with high accuracy. It has functions that cannot be realized by other image recognition technologies. For this reason, it has a wide range of applications in the industry, such as robot arm picking for randomly loaded parts, and special hardware is required because image information only needs to be acquired from different viewpoints. Because it is a technology that can be realized at low cost without the need for wear, the position and orientation measurement by the stereo vision method is a technology that is strongly expected to be put to practical use.

なお、画像の大きさをM×N、一方の画像の位置（ｉ，ｊ）における画素値をＴ（ｉ，ｊ）、もう一方の画像の位置（ｉ，ｊ）における画素値をＩ（ｉ，ｊ）とする。
これらSADやSSDの値が最も小さくなるような２次元位置を、基準画像に対する参照画像の対応点としている。そして、非特許文献１に開示されているように、再帰相関法を用いることによって、その計算量を大幅に低減することができ、ひいては、処理時間を短縮することが可能である。 In this stereo vision method, the three-dimensional position is calculated using triangulation. In order to calculate the three-dimensional position, the position of a point where a certain spatial point is projected on two images is calculated. Need to be identified. Specifically, it is necessary to search for the same point in the reference image for a pixel at a specific position in the standard image, with one image as the standard image and the other image as the reference image. In order to search for a corresponding point in the reference image corresponding to a pixel at a specific position in the reference image, for example, a sum of absolute differences (SAD) or a sum of squared differences (SAD) expressed by the following Equation 1 is used. Corresponding point search processing is executed using SSD (Sum of Squared Difference).

Note that the size of the image is M × N, the pixel value at the position (i, j) of one image is T (i, j), and the pixel value at the position (i, j) of the other image is I (i , J).
The two-dimensional position where the values of these SAD and SSD are the smallest is taken as the corresponding point of the reference image with respect to the standard image. Then, as disclosed in Non-Patent Document 1, by using the recursive correlation method, the amount of calculation can be greatly reduced, and thus the processing time can be shortened.

Satoshi Okada, Misaki Kaga, Masayuki Inaba, Hiroki Inoue, Recursive Correlation Method and Fast Optical Flow Calculation Method with Multimedia Instruction, Computer Vision and Image Media, 115-17, pp. 127-132, March 1999

しかし、このNCCを用いた対応点探索処理は、SADなどを用いた対応点探索処理のように再帰相関法を使用することができないため、計算量が多く、処理速度が遅くなるという問題があった。 However, in the corresponding point search process using the above-described difference absolute value sum or difference sum of squares, when the luminance difference between the standard image and the reference image is large, or when the contrast in the standard image or the reference image is low Due to the frequent occurrence of erroneous correspondence, there arises a problem that distance measurement cannot be performed normally. For this reason, when the luminance difference between the standard image and the reference image is large, or when the contrast in the standard image or the reference image is low, the normalized cross-correlation (NCC) shown in Equation 2 is used. It was necessary to perform corresponding point search processing using.

However, the corresponding point search process using NCC cannot use the recursive correlation method like the corresponding point search process using SAD or the like, and thus has a problem that the calculation amount is large and the processing speed is slow. It was.

  Therefore, the present invention appropriately performs the corresponding point search process without reducing the processing speed even when the luminance difference between the standard image and the reference image is large or when the contrast in the standard image or the reference image is low. An object is to provide an image processing method, an image processing program, a computer-readable storage medium storing the image processing program, and an image processing apparatus.

  The image processing method according to the present invention includes a step of acquiring two images of an object captured from different viewpoints as a standard image and a reference image, and a normalization process for the luminance of each pixel in the standard image and the reference image. Normalization processing step, each pixel in the reference window set on the normalized reference image, and a reference window of the same size as the reference window set on the normalized reference image A corresponding point search processing step of searching for a corresponding point on the reference image corresponding to the target point on the base image by calculating a correlation with each pixel of the reference image, and the normalization processing step includes The luminance I of the pixel located at the center in the normalization processing window set on each of the reference image and the reference image is set in each normalization processing window. A value obtained by subtracting the average value μ of the prime luminance is divided by the standard deviation σ of the luminance of each pixel in the normalization processing window, and this normalization processing is performed on the reference image and the reference image. This is done for every pixel.

  In this image processing method, the luminance of each pixel in the base image and the reference image is normalized by the above-described method before the corresponding point search processing step. For this reason, when the brightness difference between the standard image and the reference image is large, or when the contrast in the standard image or the reference image is low, even if SAD or SSD is used as the corresponding point search processing step, there is an erroneous response. It does not occur, and distance measurement can be performed appropriately. In order to reduce the problem of occlusion, the number of images of the object can be three or more. The normalization formula is (I−μ) / σ.

  Note that the normalization window is preferably set to the same size as the reference window.

  Further, the corresponding point search processing step includes a sum of absolute differences between a pixel value of each pixel in the reference window set on the reference image and a pixel value of each pixel in the reference window set on the reference image. It is preferable to calculate (SAD: Sum of Absolute Difference) by using the recursive correlation method from the viewpoint of realizing rapid image processing. In addition, the corresponding point search processing step includes a sum of squares of differences (SSD: Sum of Squared) between the pixel value of each pixel in the reference window and the pixel value of each pixel in the reference window set on the reference image. Difference) can also be calculated by using a recursive correlation method.

  In addition, the corresponding point search processing step can search by searching for the corresponding points in the reference image along the epipolar line so that the search range can be narrowed down to the epipolar line, and the amount of calculation required for the corresponding point search is reduced. It becomes possible to do.

  An image processing program according to the present invention causes a computer to execute any of the steps described above.

  A computer-readable storage medium according to the present invention stores the image processing program.

  The image processing apparatus according to the present invention also includes an imaging unit that captures two images from different viewpoints as a standard image and a reference image, and the standard image and the reference image captured by the imaging unit. An arithmetic processing unit that performs image processing, and the arithmetic processing unit calculates a luminance I of a pixel located at a center in a normalization processing window set on each of the standard image and the reference image, and performs the normalization processing A value obtained by subtracting the average value μ of the luminance of each pixel in the window is divided by a standard deviation σ of the luminance of each pixel in the normalization processing window, and this normalization processing is performed on the reference image. And each pixel in the reference window set on the normalized reference image, and the reference image set on the normalized reference image. By calculating the correlation with each pixel in the reference window having the same size as the quasi-window, the corresponding point on the reference image corresponding to the attention point on the reference image is searched.

  According to this image processing apparatus, the arithmetic processing unit first normalizes the luminance of each pixel in the standard image and the reference image as described above, and then performs the corresponding point search process. When there is a large luminance difference from the image or when the contrast in the base image or the reference image is low, even if SAD or SSD is used as the corresponding point search processing step, no miscorrespondence occurs, and an appropriate distance is obtained. Measurement can be performed. The normalization processing window is preferably the same size as the reference window.

  According to the present invention, even when the luminance difference between the standard image and the reference image is large or when the contrast in the standard image or the reference image is low, the corresponding point search process is appropriately performed without reducing the processing speed. It can be carried out.

FIG. 1 is a block diagram showing a schematic configuration of an image processing apparatus according to the present embodiment. FIG. 2 is a flowchart showing an image processing method according to this embodiment. FIG. 3 is an explanatory diagram showing an outline of normalization processing steps according to the present embodiment. FIG. 4 is an explanatory diagram showing corresponding point search processing steps according to the present embodiment. FIG. 5 is an explanatory diagram showing an outline of the recursive correlation method in the corresponding point search processing step according to the present embodiment. FIG. 6 is an image showing a standard image (a) and a reference image (b) in the embodiment. FIG. 7 is a graph showing the result of three-dimensional restoration in Example 1. FIG. 8 is a graph showing the result of three-dimensional restoration in Comparative Example 1.

  Hereinafter, embodiments according to the present invention will be described with reference to the drawings.

  FIG. 1 shows a basic configuration of a stereo vision image processing apparatus 1 according to an embodiment of the present invention. The apparatus 1 includes an arithmetic processing unit (hereinafter referred to as CPU) 2, a recording unit 3 for recording data, a storage unit (hereinafter referred to as memory) 4, an interface unit 5, and data (control signal) between these units. The main body 10 is composed of an internal bus 6 for exchanging. The operation unit 7, the display unit 8, and the first and second imaging units C <b> 1 and C <b> 2 are connected to the main body unit 10 through the interface unit 5.

  The CPU 2 reads a predetermined program from the recording unit 3 into the memory 4 and executes predetermined data processing using a part of the memory 4 as a work area. For example, the operation unit via the interface unit 5 In response to an instruction or data input from 7, the instructed process is executed while the input data is recorded in the recording unit 3, and the corresponding process is executed. In addition, the CPU 2 appropriately records a calculation result during execution and a final result after completion of processing in the recording unit 3 or displays predetermined information on the display unit 8 via the interface unit 5. For example, a graphical user interface image for accepting input from the operation unit 7 is displayed on the display unit 8. This apparatus can be realized by a computer. In this case, for example, a computer keyboard or mouse can be used as the operation unit 7, and a CRT display, a liquid crystal display, or the like can be used as the display unit 8.

  The first and second imaging units C <b> 1 and C <b> 2 are arranged at predetermined positions at predetermined intervals, capture the imaging target T, and transmit the captured image data to the main body unit 10. In the main body unit 10, the image data from the imaging units C <b> 1 and C <b> 2 acquired via the interface unit 5 can be distinguished from each other, for example, with different file names corresponding to the imaging units C <b> 1 and C <b> 2. The image captured by the unit C1 is recorded in the recording unit 3 as a standard image, and the image captured by the second imaging unit C2 is recorded as a reference image. Note that an image captured by the first imaging unit C1 may be a reference image, and an image captured by the second imaging unit C2 may be a reference image.

  When the output signals from the first and second imaging units C1 and C2 are analog signals, the main body unit 10 is provided with an AD (analog-digital) conversion unit (not shown), and samples the analog signals input at predetermined time intervals. To convert it into digital data. If the output signals from the first and second imaging units C1 and C2 are digital data, the AD conversion unit is unnecessary. The first and second imaging units C1 and C2 may be any device that can capture at least a still image. For example, a digital camera or a digital or analog video camera can be used.

  Hereinafter, processing performed by the apparatus 1 will be described based on the flowchart of FIG. In the following description, unless otherwise specified, the processing performed by the CPU 2, that is, the processing performed by the CPU 2 by controlling each unit is described. In addition, the first and second imaging units C1 and C2 are arranged with a predetermined direction facing a predetermined position so that the measurement object T can be imaged. Information on the position and the imaging direction is stored in the recording unit 3. It is recorded. Further, the internal and external parameters of the first and second imaging units C1 and C2 are obtained in advance by a calibration experiment and recorded in the recording unit 3.

  First, in step S1, initial setting is performed. This initial setting is a process necessary for performing the processes after step S2. For example, a data transmission path and a control protocol with the first and second imaging units C1 and C2 are established, and the first and second imaging are performed. The parts C1 and C2 can be controlled.

  Next, in step S2, the target T is imaged by the first and second imaging units C1 and C2, and a standard image and a reference image are acquired. The images captured by the first and second imaging units C1 and C2 are transmitted to the main body unit 10 and recorded in the recording unit 3 with a predetermined file name. As a result, two two-dimensional image data captured from different positions and different directions are recorded in the recording unit 3 as the standard image and the reference image. Either of the two pieces of two-dimensional image data may be used as the reference image.

Subsequently, in step S3, normalization processing is performed on the standard image and the reference image. The method for normalizing the reference image will be described in detail. As shown in FIG. 3A, a normalization processing window W3 having the same size as the reference window W1 described later is provided at a specific position on the reference image A. Set. Then, the average value μ ₁ and the standard deviation σ ₁ of the luminance of each pixel in the normalization processing window W3 at this position are calculated. Then, using the average luminance value μ ₁ and standard deviation σ ₁ of the pixels in the normalization processing window W3 at this position, the luminance I ₁ of the pixel C ₁ located in the center of the normalization processing window W3 is expressed by the following equation: Normalize based on. In FIG. 3, the normalized pixels are indicated by diagonal lines.
(I ₁ -μ ₁ ) / σ ₁
Next, as shown in FIG. 3B, the normalization processing window W3 is shifted to the right by one pixel, and the average value μ ₂ and standard deviation of the luminance of each pixel in the normalization processing window W3 at this position. σ ₂ is calculated, and similarly to the above, the luminance I ₂ of the pixel C ₂ located at the center of the normalization processing window W 3 at this position is normalized based on the following expression.
(I ₂ −μ ₂ ) / σ ₂
By scanning the normalization processing window W3 on the reference image A so as to repeat the above operation, the normalization processing is executed for all the pixels constituting the reference image A. For the reference image B, the normalization process is performed on all pixels in the same manner.

  Corresponding point search processing is performed between the standard image A and the reference image B that have been normalized as described above (step S4). Referring to FIG. 4 in detail, first, a certain pixel of the reference image A is set as the attention point P, and a reference window W1 is set on the reference image A around the attention point P. Note that, by raster scanning the reference window W1, all the pixels on the reference image A or a plurality of pixels with a certain interval are sequentially set as the attention point P. Similarly, a reference window W2 centered on a certain pixel is set on the reference image B. The reference window W2 has the same size as the standard window W1, and searches for a region having the highest similarity with the standard window W1 set for the standard image A while scanning the reference image B. Here, from the viewpoint of reducing errors in matching and the amount of calculation, the reference window W2 is set on the corresponding epipolar line on the reference image B, and the reference window W2 is set on the reference image A while scanning on the epipolar line. It is preferable to search for a region having the highest degree of similarity with the reference window W1. As a result of this search, the position having the highest similarity in the reference image B is specified as the corresponding point of the attention point P of the standard image A. Note that the sizes of the standard window W1 and the reference window W2 are not particularly limited, and are determined by the size, density change, and image density of the object.

  In the above-described corresponding point search, the sum of absolute differences between each pixel value in the standard window W1 set on the standard image A and each pixel value in the reference window W2 set on the reference image B (SAD: Sum of Absolute Difference) or the sum of squared differences between each pixel value in the standard window W1 set on the standard image A and each pixel value in the reference window W2 set on the reference image B ( The degree of similarity can be evaluated using SSD (Sum of Squared Difference). In order to reduce the amount of calculation of the sum of absolute differences and the sum of squared differences, it is preferable to use a recursive correlation method. And a corresponding point is specified by making the point with the smallest difference absolute value sum or the sum of squared differences the point with the highest similarity.

とすると、
以下のＳＡＤの式

（ただし、画素（ｘ、ｙ）での輝度値をI₁(x、y)、I₂(x、y)とし、ｘ、ｙの領域を０≦ｘ、ｙ＜Ｎとする。また、相関演算のウィンドウサイズをＷ×Ｗとし、その値域を０≦ｉ，ｊ＜Wとする。さらに、探索範囲ｋ、ｌの値域を０≦ｋ、ｌ＜Ｄとする。）
の相関演算は、

と書ける。ここで、

とすると、

と再帰的に計算することができる。さらにQ1(x、y、k、l)やQ2(x、y、k、l)も再帰的に計算することができる。 The recursive correlation method will be described with reference to FIG.

Then,
The following SAD equation

(However, the luminance values at the pixel (x, y) are I ₁ (x, y) and I ₂ (x, y), and the x and y regions are 0 ≦ x and y <N. (The window size of the operation is W × W, and the range is 0 ≦ i, j <W. Further, the range of the search ranges k and l is 0 ≦ k and l <D.)
The correlation operation of

Can be written. here,

Then,

And can be calculated recursively. Furthermore, Q1 (x, y, k, l) and Q2 (x, y, k, l) can also be calculated recursively.

  When the corresponding points are specified as described above, a distance image is generated based on the principle of triangulation. A distance image is generated based on the corresponding points searched as described above (step S5).

  Then, the distance image generated in step S5 is recorded in the recording unit 3 (step S6), and the image processing ends.

  Each step in the block diagram shown in FIG. 2 can be realized by causing a computer to execute a predetermined program. This program can be recorded and distributed on a computer-readable recording medium such as a CD-ROM, but can also be distributed via the Internet or the like.

  As mentioned above, although embodiment of this invention was described, this invention is not limited to these, A various change is possible unless it deviates from the meaning of this invention. For example, in the above embodiment, the distance image is generated from two images, but a distance image can be generated from three or more images. In this case, the number of imaging units is increased by that amount, one base image A and a plurality of reference images B, and corresponding points corresponding to the attention point P of the base image A are determined from each reference image B as in the above embodiment. Search by the method.

  The present invention will be described more specifically with reference to the following examples and comparative examples. In addition, this invention is not limited to the following Example.

  Using two digital cameras, the half body of the half-decompressed cod that is the object T was imaged from different viewpoints, and a reference image (FIG. 6A) and a reference image (FIG. 6B) were acquired. . As a condition, the standard camera is installed 610 mm above the installation surface of the object T, and the reference camera is installed at a position 150 mm to the right of the standard camera. The image size is 640 × 480 pixels, the image capture rate is 15 fps, and the illumination is four 20 W DC fluorescent lamps. The standard camera is a camera that captures a standard image, and the reference camera is a camera that captures a reference image.

  With respect to the above two images, the luminance of each pixel is normalized as described above in the normalization window W3 of 15 × 15 pixels, and then the recursive correlation method is performed in the reference window W1 and the reference window W2 of 15 × 15 pixels. FIG. 7 is a graph showing the result of performing the corresponding point search process using the SAD using the three-dimensional reconstruction (Example 1). Further, the result of three-dimensional restoration by performing the corresponding point search process using the SAD using the recursive correlation method for the standard image and the reference image without performing the normalization process is the graph of FIG. Example 1). The X and Y axis values of each graph indicate pixel coordinates (pixels) of the reference image, and the Z axis value indicates parallax.

  As can be seen from the graphs of FIGS. 7 and 8, in Comparative Example 1, there are many linear noises that do not actually exist and holes in which distance measurement is impossible due to mishandling (see FIG. 8). In Example 1, it can be seen that such erroneous correspondence hardly occurs (see FIG. 7).

1 Image processing device 2 CPU (arithmetic processing unit)
A reference image B reference image P attention point T object W1 normalization processing window W2 reference window W3 reference window

Claims (8)

Obtaining two images of the object imaged from different viewpoints as a standard image and a reference image;
Normalization processing steps for normalizing the luminance of each pixel in the standard image and the reference image;
Correlation between each pixel in the reference window set on the normalized reference image and each pixel in the reference window having the same size as the reference window set on the normalized reference image A corresponding point search processing step of searching for a corresponding point on the reference image corresponding to the target point on the base image by calculating;
Including
In the normalization processing step, the luminance I of the pixel located at the center in the normalization processing window set on each of the standard image and the reference image is calculated as an average value of the luminance of each pixel in the normalization processing window. What is subtracted by μ is normalized by dividing by the standard deviation σ of the luminance of each pixel in the normalization processing window, and this normalization processing is performed for every pixel of the reference image and the reference image. Image processing method.   The image processing method according to claim 1, wherein the normalization processing window is set to have the same size as the reference window.   The corresponding point search processing step includes a difference absolute value sum between a pixel value of each pixel in the reference window set on the reference image and a pixel value of each pixel in the reference window set on the reference image The image processing method of Claim 1 or 2 performed by calculating (SAD: Sum of Absolute Difference) using a recursive correlation method.   The image processing method according to claim 1, wherein the corresponding point search processing step searches for a corresponding point in the reference image along an epipolar line. An image processing program for causing a computer to execute the image processing method according to claim 1.   A computer-readable storage medium storing the image processing program according to claim 5. An imaging unit that captures two images from different viewpoints as a base image and a reference image;
An arithmetic processing unit that performs image processing on the reference image and the reference image captured by the imaging unit,
The arithmetic processing unit includes:
What is obtained by subtracting the luminance I of the pixel located in the center of the normalization processing window set on each of the standard image and the reference image by the average value μ of the luminance of each pixel in the normalization processing window, Normalization processing is performed by dividing by the standard deviation σ of luminance of each pixel in the normalization processing window, and this normalization processing is performed for every pixel of the standard image and the reference image,
Correlation between each pixel in the reference window set on the normalized reference image and each pixel in the reference window having the same size as the reference window set on the normalized reference image An image processing apparatus that searches for a corresponding point on the reference image corresponding to a point of interest on the base image by calculating.   The image processing apparatus according to claim 7, wherein the normalization processing window is set to have the same size as the reference window.

Images