JP2007132715A - Three-dimensional measuring method, measuring device, restoring method, and restoring device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method and device capable of removing ambiguity of matching, and measuring a three-dimensional shape easily and with high accuracy. <P>SOLUTION: In this method, an object 26 is photographed by a photographic means 22 capable of photographing two images simultaneously, a shadow 23a is moved on the object 26, and a plurality of objects 26 are photographed by the photographic means 22 so that the shadow 23a is included. A plurality of sets of subtracted image are created, by subtracting the luminance value of each region of each image that does not include shadows from the luminance value of each region of each image including the shadow 23a and objects 26, and each part of the shadow region included in a second subtracted image, photographed and subtracted from the other side that corresponds to the shadow region included in a first subtracted image photographed and subtracted from one side of the photographic means is retrieved. A three-dimensional coordinate position is calculated, by using the two-dimensional coordinate position data of each part of the shadow region included in the first subtracted image and two-dimensional coordinate position data of each corresponding part included in the second subtracted image. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention provides a method and apparatus capable of measuring the shape of an object with high accuracy by stereo viewing using a difference area that is optically different, and measuring the shape of an object with high accuracy by stereo viewing using a difference area. The present invention also relates to a method and apparatus for restoring the three-dimensional shape as a three-dimensional model on a display screen.

  In the survey of mineral resources on planets, planetary exploration using rover has attracted attention from the viewpoint of cost and safety. The rover is equipped with a measuring device for measuring the three-dimensional shape of the mineral resource. When measuring accuracy of a measuring device is increased, its weight increases and power consumption increases, so there is a limit to the measuring device that can be mounted on a rover. For this reason, the method of measuring a three-dimensional shape or the like using a measuring device mounted on a rover has a limit in measurement accuracy.

  In view of this problem, in order to measure with high accuracy, for example, a method of mounting a manipulator on a rover, collecting a mineral resource using the manipulator, and taking it home can be considered. As for manipulators, a lot of research on remote control using joysticks and slave arms has been conducted, but the planet has a large spatial distance from the earth and causes a problem of communication time delay, so it accurately grasps the target object. It is difficult. For this reason, it is thought that it is difficult to extract a target mineral resource using a manipulator.

As a method for measuring a three-dimensional shape, there are a method using stereo vision, an optical radar method, an active stereo method, and the like. In the stereo vision method, two cameras are used to observe the same object from two different viewpoints, and three-dimensional information of the object is obtained from the difference in projection position on each image. . Although this method has the advantage that it can be easily measured with a small number of devices, there is a problem that it cannot be measured with high accuracy. The optical radar method is a method of obtaining a distance image by applying light to an object to be measured and measuring the time until the light returns. The active stereo method is a method in which one camera is replaced with a machine that generates light instead of using two cameras. Although these methods have the advantage of being able to obtain highly accurate measurement results compared to methods based on stereo vision, a device that emits energy such as light and ultrasonic waves is necessary, and the device is large, There is a problem that the weight increases and the power consumption also increases. In particular, it is difficult to mount on the rover.

  In view of this, a method and apparatus for restoring a highly accurate three-dimensional shape using the method based on stereo vision has been proposed (see, for example, Patent Documents 1 to 3). The apparatus described in Patent Document 1 proposed here generates three-dimensional shape data of an object existing in a range that can be photographed by a camera by stereo measurement processing, and obtains the points and points where the three-dimensional position data is obtained. The data missing part is extracted by connecting the boundaries with the points that were not found in a chain, and extracting the connected closed curve, and the average value of the height data of the points on the closed curve and each point in the data missing part The three-dimensional position data of each point of the data missing part is calculated from the position data and set. In this apparatus, since three-dimensional shape data having no data missing point is generated, a highly accurate three-dimensional model can be restored.

  The system described in Patent Document 2 extracts feature data indicating an edge of a measurement object from a plurality of pieces of image data obtained by photographing the measurement object from different angles. A perspective projection calculated from the length data of one location in the given image data, the feature data, and the constraint condition The three-dimensional shape is restored from the transformation matrix. In this system, it is possible to prevent miscorrespondence and association between feature data with low accuracy, improve the accuracy of association, and restore a highly accurate three-dimensional shape.

  The apparatus described in Patent Document 3 detects an edge from two-dimensional luminance image data obtained by photographing an object, calculates a parameter indicating the smoothness of the surface of the object from the edge, and uses this parameter as a reflection map as shadow information. The parameters are determined, and image data for restoring the three-dimensional shape is generated using the reflection map parameters.

  These devices and systems are simple devices equipped with a photographing means such as a camera and a processing means for processing image data, and can restore a highly accurate three-dimensional shape, but generate and extract a closed curve. It is necessary to calculate three-dimensional position data, giving the data of the length of one place in the image data, extracting the feature data and calculating the constraint condition, and calculating the perspective projection transformation matrix from the feature data and constraint condition There is a problem in that it is necessary to calculate, and it is necessary to calculate a parameter indicating the smoothness of the surface of the object, and to determine a reflection map parameter as shadow information using this parameter, which requires a calculation time. In order to shorten the calculation time, a computer having high calculation processing performance can be used. However, when the weight, power consumption, etc. of a device that can be mounted like the rover are limited, the processing performance of the computer is also limited. Therefore, an apparatus and a method that can restore a three-dimensional shape as easily as possible, with high accuracy, and in a short calculation time in view of power consumption are desired.

In the correspondence between images in stereo vision, ambiguity is an important problem that always occurs. In order to solve this problem, many methods for associating images with local density patterns (local support) as clues have been proposed. For example, a window is set around an attention point in one of the left and right images, and the epipolar line of the other image (a straight line connecting one imaging means and the attention point projected on the other imaging means using the window as a template) ) Matching method within the preset search range above, using local support with Gaussian weighting according to the distance from the center of the window, distribution of density and parallax in the window A method of using variable local support in a window considering the above has been proposed. Since these methods using local support are based on many assumptions and assumptions, this problem of ambiguity cannot be essentially solved. There has been a problem that fine changes and sudden changes are suppressed, resulting in an overall smooth shape.
JP 2001-283200 A JP 2002-109518 A JP-A-5-181980

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a method and apparatus capable of removing the ambiguity of correspondence and measuring a three-dimensional shape easily and with high accuracy. . Another object of the present invention is to provide a method and an apparatus for measuring a three-dimensional shape and restoring the three-dimensional shape as a three-dimensional model on a display screen.

  The inventors of the present invention have made extensive studies in view of the above problems. As a result, the focal lengths are equal, the calibration is performed accurately, the optical axes are parallel to each other, and the image planes are on the same plane. In this way, a camera with lenses arranged on the left and right sides is arranged, and shadows are used as the difference areas that are optically different. First, a shadowless image is acquired by the camera, and then the shadows are projected onto the ground. Slide a stick, etc., and shoot multiple shots in succession so that the shadow is reflected.The brightness value of each region of the multiple images with the shadow is calculated for each image of the shadowless image at the same coordinate position as each region. By subtracting the brightness value of the area and generating an image with brightness only in the shadow area, the search range when matching two images captured by the camera can be limited to improve the matching accuracy. This removes the ambiguity of the mapping It can easily and found that can measure a highly accurate three-dimensional shape. The above-mentioned subject is achieved by providing the three-dimensional measuring method, measuring device, three-dimensional shape restoration method, and restoration device of the present invention.

The three-dimensional measurement method of the present invention includes a step of photographing an object by a photographing unit capable of photographing two images at the same time, a step of generating a difference region that is optically different, and moving the difference region on the object; , A step of photographing a plurality of objects so as to include a different area by the photographing means, a luminance value of each area of a plurality of images including the different area and the object, and a different area at the same two-dimensional coordinate position as each of the areas A difference between the brightness value of each area of the image not including the image and generating a plurality of subtracted images having brightness in the difference area, and a plurality of sets of subtracted images are captured from one side of the imaging means Each of the difference areas included in the plurality of second subtraction images captured from the other side of the imaging unit and subtracted, corresponding to each portion of the difference areas included in the plurality of first subtraction images subjected to the subtraction process. Part A step of searching based on the shape portion having the characteristic of the difference area, a two-dimensional coordinate position data of each part of the difference area included in the first subtraction image, and a search of the difference area included in the second subtraction image. And calculating the three-dimensional coordinate position of each portion using the corresponding two-dimensional coordinate position data of each portion.

In areas other than the difference area where there is almost no change in brightness, and areas where the change in brightness is small within the difference area, the exact matching point (the other subtraction image of the other subtracted image) is affected by geometric distortion and random noise caused by projections with different viewpoints. It is difficult to find each corresponding part). Since the edge (edge) of the difference area has a characteristic that the change in luminance is large, ambiguity during the matching process is reduced, and an accurate matching point can be easily found. For this reason, when there are a plurality of first subtraction processing images having luminance values at the same two-dimensional coordinate position, a region-based corresponding point search method that performs matching based on the similarity of the peripheral region of the pixel to be matched is performed. It is preferable to use the first subtraction processing image in which the edge of the difference area enters the window for designating the predetermined area including the peripheral pixels for all the two-dimensional coordinate positions to be matched. It is preferable to perform matching using the 1 subtraction processing image and the corresponding second subtraction processing image.

It is preferable to include a step of searching for a first subtraction-processed image (for example, a left image in the case of photographing means capable of photographing right and left simultaneously) that maximizes the dispersion of luminance values in the window. In order to shorten the search time, it is particularly preferable to search for the first subtraction processing image in which the average value of the dispersion in the direction parallel to the epipolar line is maximized. The first subtraction processing image in which the variance or the average value of the variance is maximized has a high probability that the edge of the difference area is included, and the edge can easily find an exact matching point as described above. is there. The searching step includes a step of searching for one first subtraction image that maximizes the variance of the luminance values of pixels in the window while maintaining the two-dimensional coordinate position of the window, and a step of moving the window. And a step of searching for another first subtraction image having the maximum variance at the moved two-dimensional coordinate position, a step of moving, and a step of searching for another first subtraction image. Can do. In the step of moving the window, the window can be moved along the epipolar line. Further, the window can be moved so that the center pixel located at the center in the window moves rightward with respect to the paper surface by one pixel.

The present invention also provides an apparatus for performing the three-dimensional measurement method. The apparatus includes a photographing unit capable of photographing two images at the same time, a unit that generates a difference area that is optically different, a movable unit that moves the difference area on the object, and the difference area and the object. The difference between the luminance value of each area of the plurality of images and the luminance value of each area of the image not including the difference area at the same two-dimensional coordinate position as each of the areas is obtained, and a plurality of subtraction processes having luminance in the difference area Generating the image, and the other of the imaging means corresponding to each part of the difference area included in the plurality of first subtraction images captured and subtracted from one side of the imaging means of the plurality of sets of subtraction processing images Each of the different areas included in the plurality of second subtraction-processed images photographed from the side and subtracted is searched based on the shape part having the characteristics of the different areas, and the different areas included in the first subtraction-processed image 2D coordinate position of each part of Using the data and the two-dimensional coordinate position data of the searched corresponding respective portions of the difference region in the second subtraction image, and a calculation processing unit for calculating a three-dimensional coordinate positions of the respective parts. This calculation processing means can process a subtraction process, a search for matching (association), and a calculation of a three-dimensional coordinate position by executing a program. Specifically, the program is recorded. A recording medium and a processor that reads and executes the program can be configured.

The three-dimensional shape restoration method of the present invention uses the three-dimensional coordinate position data in each part of the difference area calculated for each set of subtraction images in addition to each step in the measurement method, Each of the shapes is generated, the three-dimensional shapes are arranged in order of photographing time, and displayed on the display device as a three-dimensional model.

The restoration apparatus according to the present invention includes a display device that displays a three-dimensional model of an object in the measurement device, and the calculation processing means of the measurement device uses a difference area calculated for each set of subtraction processing images. Using the three-dimensional coordinate position data, three-dimensional shapes of the difference areas are generated, the three-dimensional shapes are arranged in order of photographing times, and displayed on the display device as a three-dimensional model.

  By providing the three-dimensional measurement method and the measurement apparatus of the present invention, the ambiguity of correspondence can be removed, and a three-dimensional shape can be measured easily and with high accuracy. In addition, by providing the three-dimensional shape restoration method and restoration device of the present invention, the three-dimensional shape of an object at a remote place can be accurately restored on the display screen. Since the apparatus of the present invention requires only a small number of apparatuses and can shorten the matching process time, the apparatus weight can be reduced and the power consumption required for the calculation process can be reduced. It is suitable for a rover where a stackable device is limited.

  The present invention will be described below with reference to specific embodiments shown in the drawings, but the present invention is not limited to the embodiments described below. In addition, the present invention will be described in detail using a shadow as a difference area that is optically different. However, as this difference area, an area irradiated with light can be used, for example, a laser light irradiation area, a slit, or the like. For example, a red light irradiation region or a blue light irradiation region other than the light irradiation region and the background color can be used. First, FIG. 1 illustrates a rover for observing mineral resources. FIG. 1 shows the manipulator 10 being moved in order to perform three-dimensional measurement. The manipulator 10 is a device that works on behalf of a human by remotely operating a machine having a hand structure. The manipulator 10 can be used to collect mineral resources and take them home.

  In order to observe various mineral resources, the rover shown in FIG. 1 has a photographing unit 11 capable of photographing two images at the same time so that at least a part of a photographing target region overlaps, a wheel 12 as a moving unit, and a photographing unit. A calculation processing means (not shown) for calculating the processed image data. The manipulator 10 can be used as a means for generating a shadow (difference area that is optically different), and is connected to a movable means (not shown), and is represented by an arrow A in FIG. 1 by this movable means. Can be moved to. The shadow 13 of the manipulator 10 is generated by a light source 14 such as a halogen lamp, and moves as the manipulator 10 moves.

The photographing means 11 provided in the rover can photograph the mineral resource 15 on the ground, and when the shadow 13 of the manipulator 10 is generated on the mineral resource 15, the mineral resource 15 and the shadow 13 are photographed. be able to. The photographing unit 11 can photograph two images simultaneously at a time interval of several tens to several hundreds of milliseconds, for example. Moreover, the imaging | photography means 11 image | photographs the specific position on the ground, unless the wheel 12 of a rover is rotated. In the embodiment shown in FIG. 1, since only the manipulator 10 is moving, the captured image is an image in which only the position of the shadow 13 is different.

The captured image is sent as image data to calculation processing means (not shown). In the calculation processing means (not shown), the luminance value of each region of the plurality of images including the shadow 13 and the mineral resource 15 that is the object is calculated for each region of the image that does not include the shadow at the same two-dimensional coordinate position as each region. Subtract the luminance value to generate a plurality of subtraction processing images having luminance in the shadow region, and capture each part of the shadow region included in the plurality of sets of subtraction processing images photographed by the photographing unit 11 and subtracted. A search is performed based on the shape portion having the above feature, and the three-dimensional coordinate position of each part is calculated using the two-dimensional coordinate position data of each corresponding part. This calculation process will be described in detail below.

  In the embodiment shown in FIG. 1, the three-dimensional measurement apparatus includes the photographing unit 11, a manipulator 10 and a light source 14 that generate a shadow, a movable unit that makes the manipulator 10 movable, and a calculation processing unit. ing.

FIG. 2 is a diagram showing in detail the three-dimensional measuring apparatus when a shadow is used as a different area that is optically different. The apparatus shown in FIG. 2 includes a photographing means 22 capable of photographing two images simultaneously with the optical axes 20 and 21 being parallel and the image plane being on the same plane, a rod-shaped member 24 and a light source 25 for generating a shadow 23. A calculation process for calculating a three-dimensional coordinate position using a plurality of captured images, and a movable means 27 for connecting the rod-shaped member 24 and moving the rod-shaped member 24 so that the shadow 23a crosses the object 26. And means 28. Here, the optical axes 20 and 21 are parallel, but in the present invention, it is only necessary to be able to photograph so that at least a part of the photographing target region overlaps.

  The imaging means 22 shown in FIG. 2 can be a stereo camera that has the same focal length, known camera parameters, is accurately calibrated, and can capture two images simultaneously. Further, as described above, the photographing unit 22 can continuously photograph at a time interval such as several tens to several hundred milliseconds.

  The rod-shaped member 24 can be a simple manipulator in addition to a simple rod. The light source 25 may be sunlight, but can be an illumination device such as a halogen lamp. The movable means 27 can be slid in a state where the rod-like member 24 such as a rod or a manipulator is connected and the shadow 23 is projected on the ground. Any member can be used as long as it can generate the shadow 23. However, in the left and right images simultaneously captured by the photographing means 22, for example, when the left image is matched with the right image. In order to limit the search range, it is preferable that the rod-shaped member 24 can generate a shadow having a narrow width. In the present invention, as described above, light can also be used, and examples of means for generating the difference area include laser light irradiation means, slit light irradiation means, red light, and blue light. The movable means can move the irradiation area of the irradiation means or the light, and can be, for example, a turntable or the like on which the irradiation means or the light can be placed.

  Here, the two images photographed by the photographing means 22 are a left image and a right image, respectively, and a plurality of images photographed are shown in FIG. FIG. 3A shows a left image, and FIG. 3B shows a right image. The image indicated by number 1 is a shadowless image that is captured by pressing the first shutter when no shadow is generated. The image indicated by the number 2 is an image which is generated by generating a shadow and starting the movement of the shadow from the right side to the left side with respect to the paper surface and then pressing the second shutter, and is indicated by the number k. Is an image shot by pressing the kth shutter, and the image indicated by the number n is an image shot by pressing the nth shutter. Since the orientation and position of the photographing means 22 are not changed, as shown in FIG. 3A, the brightness values of the ground and the object shown in the left image of number 1 and the left images of numbers 2 to n are shown. The brightness values of the ground and the object are almost the same except for the shadow area.

The calculation processing means 28 shown in FIG. 2 subtracts the left images of numbers 2 to n with the left image of number 1. Specifically, processing as shown in FIG. 4 is performed. FIG. 4A is a diagram showing a subtraction process for the left image of number k, and FIG. 4B is a diagram showing a subtraction process for the right image of number k. . The luminance value of the left image and the right image of number 1 is set to I ₁ (X, Y), the luminance value I _k (X, Y) of the left image and the right image of number k, and the luminance value of the subtracted image subjected to the subtraction process If I ′ _k (X, Y), subtraction processing can be performed using the following equation. However, when an area irradiated with light is used as the difference area, I ₁ (X, Y) is subtracted from I _k (X, Y).

As shown in FIGS. 4A and 4B, the left image and the right image after the subtraction process are images having luminance only in the difference area (here, the shadow area). Since the shadow area photographed by the photographing means has a luminance value of 0, the luminance value of the shadow area after the subtraction process is equal to I ₁ . The spatial positions of the shadow areas of the left image and the right image taken by the photographing means are equal, and the luminance value is 0 for all areas other than the shadow areas. Therefore, the search range when matching the left image and the right image can be limited to a portion where the shadow region exists, and thereby matching can be facilitated and matching accuracy can be improved. Details of the matching process will be described later.

After the matching between the left image and the right image is completed, the three-dimensional coordinate position (X, Y, Z) of the point P in the object is calculated using the following formula. First, the projection point (x ₁ , y ₁ ) of the point P on the left image and the projection point (x ₂ , y ₂ ) of the point P on the right coordinate are extracted. Specifically, since the projection point (x ₁ , y ₁ ) corresponding to the point P in the left image is selected and the matching is completed by the matching process, the projection point (x ₂ , y ₂ ) of the right image is completed. Is derived from the projection point (x ₁ , y ₁ ) of the left image. Since the focal length f and the base line length (inter-focal distance) b of the photographing means including two lenses are set in advance and are already known, next, using the projection point coordinates and the following formula, Calculate the 3D coordinate position.

D (= x ₁ −x ₂ ) in Equations 2 to 4 is parallax. These calculations and processes can be performed by executing the program as described above, and the calculation processing means is a computer including a recording medium on which the program is recorded and a processor that reads and executes the program. be able to.

  Here, the processing by the calculation processing means will be described with reference to the flowchart shown in FIG. The calculation processing means receives the image data photographed by the photographing means in order of photographing time and stores them in a storage device such as a memory or a hard disk (step 500). For example, the data is stored separately for the left image and the right image together with the numbers 1 to n. Each image data includes luminance value data in each pixel. Next, the left image data and right image data of number 1 are read (step 510). The left image data and the right image data of number 2 are read (step 520) and subtracted (step 530). The number is incremented by 1 (step 540), and it is determined whether the number has reached n + 1 (step 550). n is the number of times of photographing, and can be determined as, for example, the number of left image data stored in the storage device. The calculation processing means can count the number of received data and set the number of data as n. If the number has not reached n + 1, the process returns to step 530 to perform subtraction processing. When the number reaches n + 1, since there is no image to be subtracted, the process proceeds to step 560, where each part of the shadow area included in the right image corresponding to each part of the shadow area included in the left image is searched. . That is, the matching point of the right image corresponding to the left image is searched. After the matching point is searched, the three-dimensional coordinate position at each point of the object is calculated using the above formulas (2) to (4) (step 570).

  In the present invention, the edge of the difference area can be used to facilitate finding an exact matching point. The edge is an edge portion of the different area, and is a portion where the luminance value greatly changes. The portion where the luminance value changes greatly has a large feature in shape.

As shown in FIG. 6, the two-dimensional coordinate position of the window 60 that designates a predetermined area is fixed, and the left of the plurality of left images photographed by the photographing means has the maximum variance of luminance values in the window 60. It is preferable to search for an image. This is because there is a high probability that the left image with the maximum variance includes the edge of the shadow region 61. This search is not limited to the left image, but may be the right image. The window 60 includes a predetermined number of pixels, and after searching for a left image having the maximum dispersion at the position, the center pixel located at the center in the window 60 moves to the right by about one pixel with respect to the paper surface. To be moved. That is, the left image having the maximum variance is searched for at the position of the next window 60. The search range can be further narrowed down by moving the window 60 along the epipolar line and searching only the range indicated by B.

FIG. 7 is a diagram showing a state where the window 60 is fixed at a predetermined position and a left image having the maximum variance is searched for from among a plurality of left images. In FIG. 7, the window is fixed at the same two-dimensional coordinate position (x, y) of each left image, and several pixels (pixels) are designated. The number assigned to each left image is an image number assigned in the shooting order. FIG. 7 shows that the edge of the shadow area is included in the window of the left image numbered 7 and 11.

FIG. 8 is a diagram showing the relationship between the image number and the evaluation value Ci (x, y) that is the average value of dispersion in the direction parallel to the epipolar line in the window centered on the pixel (x, y). . The evaluation value Ci (x, y) in the image number i of the pixel (x, y) is calculated using the following evaluation function.

  2W is the width of the window, and 2h is the height of the window. I (x, y) is a luminance value at the two-dimensional coordinate position (x, y). In FIG. 8, when there is no pixel having a luminance value in the window, that is, when the numbers are 1 to 6 and 13 to 32, the evaluation value is 0, and when the shadow area is included in the window, that is, the numbers 7 to 12 In the case of (2), it has a luminance value, and since the luminance values of the pixels in the window have dispersion, it has an evaluation value. In particular, the numbers 7 and 11 including the edges of the shadow region have a large variance of the luminance values, and thus the evaluation value C at the numbers 7 and 11 is larger than the numbers 8 to 10 and 12. The left image with the maximum evaluation value is an image represented by the number 7, and the left and right images at this window position are easily matched by using the left image of the number 7 and the corresponding right image. be able to. Actually, the matching can be performed by associating the pixel in the window of the left image with the number 7 with the corresponding pixel of the right image.

  With reference to FIG. 9, the process of searching for the edge of the shadow area will be described in detail. Two-dimensional array MaxC (array that holds the maximum evaluation value for each two-dimensional coordinate) that can hold the same number of values as the number of pixels in the image and MaxCImage (image number that gives the maximum evaluation value for each two-dimensional coordinate) To be prepared). For example, these arrays can be held as a table in the storage device. In step 900, the left image of number i is read from the storage device. The left image with the number i is an i-th image captured by the imaging means. A window having a width of 2W and a height of 2h centered on the pixel at the two-dimensional coordinate position (x, y) is generated with x = 0 and y = 0 as initial values (if the window protrudes from the image, the presence of the pixel) The area to be used is a window) (step 910). It is determined whether or not the center pixel (x, y) in the window has a luminance value (step 920). If it has a luminance value, a shadow area is included in the window, and in step 930, the evaluation value Ci (x, y) is calculated using Equations 5 and 6 above. Next, it is determined whether or not the evaluation value Ci (x, y) is larger than the maximum value MaxC (x, y) of the evaluation values of the same two-dimensional coordinate position calculated so far ( Step 940). If larger, the value of MaxC (x, y) is updated as the maximum evaluation value, and the image number i is stored in MaxCImage (x, y) (step 950). If it is determined in step 920 that there is no luminance value, or after storing in step 950, the process moves to the next image (step 960), and it is determined whether or not all the pixels in the image i have been evaluated. (Step 970). If the evaluation has been completed, the image number i is incremented by 1 (step 980), and it is determined whether or not the increased number is equal to n + 1 (step 990). If it is not equal to n + 1, the process returns to step 900 to read the image with that number. When n + 1 is reached, there is no more image, so the search for an image that maximizes the variance of the luminance value in the window centered on the pixel at the two-dimensional coordinate position (W, h) is terminated (step 1000).

  In FIG. 10, after searching for the left image with the maximum evaluation value, MaxCImage (x, y) of the left image with the maximum evaluation value C at the two-dimensional coordinate position (x, y) and the right corresponding to it. It is the figure which read the image and calculated | required the window where the luminance distribution in a window corresponds most (high degree of coincidence) from the right image with respect to the window centering on MaxCImage (x, y) of the left image. The method of obtaining will be described later. As shown in FIG. 10, when the edge of the shadow region enters the window, matching can be performed accurately. Further, by associating one image with one two-dimensional position coordinate, even when there are a plurality of subtracted images having luminance values at the same two-dimensional coordinate position, the number of times of matching is only one ( That is, since the number of acquired images increases, the number of matching operations can be reduced and the matching process can be speeded up.

When the pixel (x _R , y _R ) of the right image corresponding to the pixel (x _L , y _L ) of the left image is detected, the pixel of the right image corresponding to the pixel on the right side of x _{L in} the left image is x By using a constraint condition that exists on the right side of _R, it is possible to limit the search range in matching and further speed up the matching process. However, the search method using this constraint condition may cause error propagation, and this method is used only when the left and right images having the same image number are used with consecutive pixels, and are used at adjacent two-dimensional coordinate positions. Even for a certain pixel, when searching using left and right images having different image numbers, matching is performed without using this constraint condition. Thereby, the matching process can be speeded up while suppressing the propagation of errors.

In the present invention, a restoration method including each step of the measurement method and a restoration device including the measurement device can be provided. In the method of restoring an object as a three-dimensional model, for each set of subtraction images, after calculating the three-dimensional coordinate position data in each part of the difference area, the three-dimensional coordinate position data is used to calculate the three-dimensional difference area. Each shape is generated. Next, the generated three-dimensional shapes are arranged in order of photographing time and displayed on the display device. Thereby, the three-dimensional shape of the object can be restored as a three-dimensional model. As described above, since the three-dimensional coordinate position of each point of the object can be measured with high accuracy, the three-dimensional shape can be restored with high accuracy.

The restoration device includes the measurement device including a display device that displays a three-dimensional model of the object. However, the calculation processing means included in the measurement device generates the three-dimensional shape of the difference area using the three-dimensional coordinate position data in each part of the difference area calculated for each set of subtraction processing images, and The original shape is arranged in order of photographing time and further has a function of displaying it as a three-dimensional model on the display device. This can also be processed by executing the program, and can be included in the recording medium as a program. A display can be used as the display device.

  Here, as a means for generating a shadow as a difference area that is optically different from a stereo camera, a halogen lamp and a rod-shaped member are used, and a personal computer is used as an image processing means to restore a three-dimensional shape. The results obtained using the method according to the invention and the conventional stereo vision method are shown below.

  A square residual method (SSD), which is representative in region-based matching and capable of high-speed computation, is widely used for matching left and right images simultaneously captured by a stereo camera. In this SSD, the difference between the luminance value of the window of the left image and the luminance value of the window of the right image is set as the evaluation value R (x, y), and the position where the evaluation value is minimum is set as the best matching point. The evaluation value R (x, y) can be calculated by the following formula.

However, since the SSD compares the luminance differences in the windows as they are, the degree of coincidence is high in the portion where the luminance change is small, but the degree of coincidence is low in the edge portion where the luminance value changes rapidly. Therefore, in the present invention, the ZNCC method is used in which the association is performed based on the similarity of dispersion. By adopting this ZNCC method, it is possible to accurately correspond to a sudden change in luminance value. The ZNCC method is a method in which a pixel having a maximum correlation coefficient R _ZNCC obtained by the following equation is used as a corresponding point.

Similarly to the above, 2W is the width of the window, 2h is the height of the window, and I (x, y) is a luminance value at the two-dimensional coordinate position (x, y). Luminance value I _L is the left image, I _R indicates the luminance value of the right image.

With reference to FIG. 11, the process of searching for corresponding points will be described in detail. In step 1100, the left and right images of number i are read from the storage device. Whether or not the element MaxCImage (x _L , y _L ) of the stored array is equal to the current image number i (the pixel (x _L , y _L ) of the current image is the two-dimensional coordinate position (x _L , y _L )) is judged (whether it has the maximum evaluation value MaxC (x _L , y _L )) (step 1110). If equal, the process proceeds to step 1120, where x _L = 0 and y _L = 0 are initial values, and the left image has a window of width 2W and height 2h centered on the pixel at the two-dimensional coordinate position (x _L , y _L ) Is generated (step 1120). Further, a window having a width 2W and a height 2h centered on the two-dimensional coordinate position (x _R , y _R ) is generated in the right image with x _R = 0 and y _R = y _L as initial values (step 1130). ). If they are not equal, it indicates that there is an image having the possibility of including an edge other than the read image, and matching is not performed from this image. In step 1140, it is determined whether there is a luminance value in the window of the right image. If there is a luminance value, the process proceeds to step 1150, and the evaluation value _RZNCC is calculated using the above equations 8 to 13 using the left and right image windows. When there is no luminance value, it indicates that there is an image having the possibility of including an edge other than the read image as described above, and matching is not performed from this image. In step 1160, it is determined whether or not the evaluation value R _ZNCC is greater than the maximum evaluation value MaxR _ZNCC stored in the storage device. If it is larger, the calculated evaluation value R _ZNCC is stored as the maximum evaluation value MaxR _ZNCC , and the two-dimensional coordinate position (x _R , y _R ) is stored as the corresponding point of (x _L , y _L ) (step 1170). ). If smaller, or after storing in step 1170, the x-axis coordinate of the two-dimensional coordinate position of the right image is incremented by 1 (step 1180), and the process returns to step 1110. In this way, the corresponding point on the x-axis coordinate is searched. Since the corresponding point is on the epipolar line, it is only necessary to search on the x-axis coordinate. That is, the search target pixel of the left image is moved by one (step 1190). After performing the processing of Steps 11010 to 1190 for all the pixels of the image of number i (Step 1200), the image number is incremented by 1 (Step 1210), and the above processing is repeated. When the image number reaches n + 1 (step 1220), all matching processes are terminated (step 1230).

In three-dimensional measurement by the conventional stereo method, two images without shadows (left image indicated by number 1 in FIG. 3A and numbers in FIG. 3B) simultaneously photographed by the photographing means 22 shown in FIG. FIG. 12A shows the result of performing the matching process using the ZNCC method from the right image 1).

  According to the method of the present invention, two images without shadow and a plurality of shadowed images (a plurality of left images indicated by numbers 1 to n in FIG. 3A), which are simultaneously photographed by the photographing means 22 shown in FIG. A plurality of subtracted images with a limited search range are obtained from a plurality of right images indicated by numbers 1 to n in FIG. 3B, and the result of performing a matching process using shadow edges is shown in FIG. Shown in b).

  In the conventional stereo method, matching processing can be performed by using only one set of left and right images. However, in the method of the present invention, it is necessary to perform matching processing using a plurality of sets of images, for example, 32 sets of images. The number of images increases. However, the search range of each image is limited, and a useless search, that is, a subtraction-processed image with a luminance value of 0 can be skipped, so that the calculation cost can be suppressed. In addition, the three-dimensional shape model restored by the method of the present invention synthesizes a screen, which is difficult to perform by the matching method according to the conventional method, as can be seen by comparing FIG. 12 (a) and FIG. 12 (b). It was found that the method is also effective for a region having a small amount of texture, which is an image used as a background.

  Table 1 below shows the number of matching points, the number of mismatch points (number of isolated points), and the calculation time when matching is performed by the method of the present invention and when matching is performed by the conventional stereo method. Calculations were performed for 320 × 240 pixel images and 640 × 480 pixel images. In addition, the window size used was to specify an area of 11 × 7 pixels and 21 × 13 pixels, respectively. The number of corresponding points was the number of pixels that were not determined to be isolated from the matched pixels, and the isolated points had a difference of 1 cm or more from the average value of the surrounding 48 pixels in the z-axis direction.

As shown in Table 1, it can be said that when the method of the present invention is used, the number of isolated points can be greatly reduced as compared with the conventional stereo method, and mismatching in matching processing is reduced. The average value of the correlation coefficient _RZNCC is larger than that of the conventional stereo method, and a high degree of coincidence is obtained for many pixels. Further, it has been found that the method of the present invention can greatly reduce the calculation time compared to the conventional stereo method, thereby improving the matching accuracy and reducing the calculation cost. It was.

  Since the measuring device of the present invention is a simple device, it can be made small and light, and the calculation time can be shortened. Therefore, power consumption can be reduced. Moreover, since three-dimensional measurement can be performed with high accuracy, it is suitable for mounting on a rover.

The figure which illustrated the rover. The figure which illustrated the three-dimensional measuring apparatus of this invention. The figure which showed the image image | photographed by the imaging | photography means. The figure which showed the place which is performing the subtraction process. The figure which showed the flow of the process by a calculation process means. The figure which showed one of the left images which have a brightness | luminance only in the shadow area | region after a subtraction process. The figure which showed the place which is searching for the left image from which the dispersion | distribution is the maximum among several left images, fixing a window to a predetermined position. The figure which showed the relationship between the image number and evaluation value Ci (x, y) which is an average value of dispersion | distribution within a window. The figure which showed the flow of the process which searches the edge of a shadow area | region. The figure which illustrated the place which matched the pixel in the window of the left image, and the pixel of the right image corresponding to it, after searching the left image for which an evaluation value becomes the maximum. The figure which showed the flow of the process which searches a corresponding point. The image which restored the three-dimensional model by the method by the conventional stereo vision, and the image which restored the three-dimensional model by the method of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Manipulator, 11 ... Imaging | photography means, 12 ... Wheel, 13 ... Shadow, 14 ... Light source, 15 ... Mineral resource, 20, 21 ... Optical axis, 22 ... Imaging means, 23, 23a ... Shadow, 24 ... Rod-shaped member, 25 ... light source, 26 ... object, 27 ... movable means, 28 ... calculation processing means, 60 ... window, 61 ... shadow region

Claims (8)

A method for measuring the three-dimensional shape of an object,
Photographing the object by photographing means capable of photographing two images at the same time;
Generating a difference area that is optically different, and moving the difference area on the object;
Photographing a plurality of the objects so that the difference area is included by the photographing means;
The brightness value of each area of each image including the difference area and the object is subtracted by the brightness value of each area of the image not including the difference area at the same two-dimensional coordinate position as the area, Generating a plurality of sets of subtracted images having luminance;
From the other side of the imaging unit corresponding to each part of the difference area included in the plurality of first subtraction processing images captured and subtracted from one side of the imaging unit of the plurality of sets of subtraction processing images. Searching for each part of the difference area included in the plurality of second subtraction-processed images that have been photographed and subtracted based on the shape part having the characteristics of the difference area;
Using the two-dimensional coordinate position data of each part of the difference area included in the first subtraction image and the two-dimensional coordinate position data of each corresponding part searched for the difference area included in the second subtraction image. And calculating the three-dimensional coordinate position of each portion. The method of claim 1, wherein the searching includes searching for an edge of the difference area. The step of searching includes a step of searching for a first subtraction image that maximizes the variance of the luminance values of pixels in a window that designates a predetermined region of the image from the plurality of first subtraction images. The method of claim 1. The searching step includes a step of searching for one first subtraction processing image that maximizes the variance of luminance values of pixels in the window while maintaining the two-dimensional coordinate position of the window, and moving the window. A step of searching for another first subtraction image having the maximum variance at the moved two-dimensional coordinate position, a step of repeating the step of searching, and a step of searching for the other first subtraction image. The method of claim 1 comprising: The method of claim 4, wherein moving the window comprises moving the window along an epipolar line. An apparatus for measuring the three-dimensional shape of an object,
Photographing means capable of photographing two images at the same time;
Means for generating a difference area that is optically different;
Movable means for moving the difference area on the object;
The difference area is obtained by subtracting the luminance value of each area of each image including the different area and the object by the luminance value of each area of the image not including the different area at the same two-dimensional coordinate position as each of the areas. A plurality of sets of subtracted images having luminance, and the difference regions included in the plurality of first subtracted images captured from the one side of the photographing unit and subtracted. Search for each part of the difference area included in the plurality of second subtraction-processed images photographed from the other side of the photographing unit and corresponding to each part based on the shape part having the characteristic of the difference area. Two-dimensional coordinate position data of each part of the difference area included in the first subtraction processing image and two-dimensional coordinate position data of each corresponding part searched for the difference area included in the second subtraction processing image. Use the tertiary of each part And a calculation processing unit for calculating a coordinate position, the measuring apparatus. A method for measuring a three-dimensional shape of an object and restoring the three-dimensional shape,
Photographing the object by photographing means capable of photographing two images at the same time;
Generating a difference area that is optically different, and moving the difference area on the object;
Photographing a plurality of the objects so that the different areas are included by the two photographing means;
The brightness value of each area of each image including the difference area and the object is subtracted by the brightness value of each area of the image not including the difference area at the same two-dimensional coordinate position as the area, Generating a plurality of sets of subtracted images having luminance;
From the other side of the imaging unit corresponding to each part of the difference area included in the plurality of first subtraction processing images captured and subtracted from one side of the imaging unit of the plurality of sets of subtraction processing images. Searching for each part of the difference area included in the plurality of second subtraction-processed images that have been photographed and subtracted based on the shape part having the characteristics of the difference area;
Using the two-dimensional coordinate position data of each part of the difference area included in the first subtraction image and the two-dimensional coordinate position data of each corresponding part searched for the difference area included in the second subtraction image. Generating a three-dimensional shape of each of the different areas, arranging the three-dimensional shapes in order of photographing time, and displaying the three-dimensional shape on a display device as a three-dimensional model. An apparatus for measuring the three-dimensional shape of an object and restoring the three-dimensional shape,
Photographing means capable of photographing two images at the same time;
Means for generating a difference area that is optically different;
Movable means for moving the difference area on the object;
A display device for displaying a three-dimensional model of the object;
The difference area is obtained by subtracting the luminance value of each area of each image including the different area and the object by the luminance value of each area of the image not including the different area at the same two-dimensional coordinate position as each of the areas. A plurality of sets of subtracted images having luminance, and the difference regions included in the plurality of first subtracted images captured from the one side of the photographing unit and subtracted. Search for each part of the difference area included in the plurality of second subtraction-processed images photographed from the other side of the photographing means and corresponding to each part based on the shape part having the characteristic of the difference area. Two-dimensional coordinate position data of each part of the difference area included in the first subtraction processing image and two-dimensional coordinate position data of each corresponding part searched for the difference area included in the second subtraction processing image. Use the tertiary of each part A coordinate position is calculated, and using the three-dimensional coordinate position data in each part of the difference area calculated for each set of the subtraction processing images, a three-dimensional shape of the difference area is generated, and the three-dimensional shape is calculated. And a calculation processing unit arranged in order of photographing time and displayed on the display device as the three-dimensional model.

Images