JP4243692B2 - Three-dimensional shape detection apparatus and three-dimensional shape detection method - Google Patents

Description

  The present invention relates to an apparatus and method for detecting a three-dimensional shape, and in particular, an edge image having distance information is generated by minutely moving the optical axis of a camera, and a reference point is associated using this to generate a three-dimensional shape. The present invention relates to a three-dimensional shape detection apparatus and a three-dimensional shape detection method.

  Conventionally, various methods for detecting a three-dimensional shape based on two images using parallax have been developed. Usually, using the principle of triangulation, the target object is photographed by two cameras arranged at two known positions, the reference point (corresponding point) of the target object is determined, and each camera The distance from the camera to each edge of the target object can be detected by matching the reference points of the captured target object and obtaining the parallax of other parts.

  Here, in order to obtain the parallax, it is necessary to determine a reference point indicating the same position of the target object from the image of the target object captured by each camera. If the reference point is set at a different position on the target object, the three-dimensional shape is mistaken and the correct shape cannot be detected. This is a very important problem, and various solutions to this problem have been proposed. ing. For example, assuming that the relative difference in the position in the x-axis direction between each pixel of another edge with respect to each pixel of an edge corresponds to parallax, the edges of two edge images correspond to edges on the same line of the target object In this case, there is a method using the principle that the parallax of the pixels on the edge is almost uniform at any position in the y-axis direction.

  Furthermore, as described in Patent Document 1, a relative difference is obtained for each pixel constituting the edge of two edge images, and weighting is further performed based on partial information on the density related to each pixel. Some have attempted to determine the corresponding reference point.

JP-A-6-180218

  However, in the conventional method, even if the corresponding reference point can be obtained accurately, a large amount of calculation is required for this, and processing time is required. Therefore, it is difficult to detect a three-dimensional shape in real time by photographing a moving target object using two cameras because of processing time.

  In view of such circumstances, the present invention intends to provide a three-dimensional shape detection device and a three-dimensional shape detection method capable of performing real-time processing by accurately associating reference points in an edge image with simple processing. .

  In order to achieve the above-described object of the present invention, a three-dimensional shape detection apparatus according to the present invention includes an imaging element and a lens for photographing a target object from at least two predetermined photographing positions having different lines of sight to the target object. At least one camera means, visual axis fine movement means for finely moving the optical axis of the camera means in a direction connecting the at least two predetermined photographing positions, and the optical axis finely moved by the visual axis fine movement means Image processing is performed on the plurality of images from the camera means to generate edge images in which the edges of the target object at the at least two predetermined shooting positions are extracted, and the respective edge images are compared. Image processing means for detecting a corresponding edge of the target object.

  Here, the predetermined photographing position may be arranged along at least one direction of the horizontal direction, the vertical direction, and the oblique direction, or a plurality of these directions.

  Further, the camera unit may have a moving unit capable of moving to a predetermined photographing position.

  The camera means may be composed of at least two cameras, and each camera is arranged in advance at a predetermined shooting position.

  The at least two camera units may be synchronized with each other, and may be simultaneously photographed with the same minute movement.

  Further, the visual axis fine moving means may move the lens minutely or move the camera means minutely.

  The visual axis fine movement means may be composed of any one of a piezoelectric element, a giant magnetostrictive element, an electrostatic actuator, a hydraulic actuator, a pneumatic actuator, an electromagnet, an optical actuator, a shape memory alloy, and a polymer actuator.

  Here, the fine movement by the visual axis fine movement means may be a translational movement in the normal direction of the optical axis of the camera means.

  Further, the fine movement by the visual axis fine movement means may be a rotational movement around a predetermined point. The rotational angle of the rotational motion at that time may be an angle such that the moving width of the target object close to infinity on the image sensor is 0 to 3 pixels wide.

  Furthermore, the three-dimensional shape detection method according to the present invention includes a step of photographing a target object with at least one camera means having an image sensor and a lens from at least two predetermined photographing positions having different lines of sight to the target object, and the photographing During the process of moving the optical axis of the camera means in a direction connecting the at least two predetermined photographing positions, and a plurality of images from the camera means in which the optical axis is moved finely The image processing is performed to generate an edge image in which the edges of the target object are extracted at the at least two predetermined shooting positions, and the corresponding edge of the target object is detected by comparing the respective edge images. The process of comprising.

  Here, in the process of generating the edge image, normalization may be performed by differentially processing the gray value or the saturation value of each corresponding pixel of the plurality of images. Further, binarization processing may be added.

  In the process of detecting the corresponding edge, the edge of the edge image at the at least two predetermined photographing positions may be associated with each other based on the width and length of the edge of the edge image.

  Furthermore, the process of detecting the corresponding edge may be performed by a pattern matching method.

  In the process of detecting the corresponding edge, the direction of the line of sight to the target object may be used as a reference. In this case, the edges of the generated edge image may be converted into a predetermined reference width based on the direction of the line of sight to the target object, and the edge images may be compared.

  In the three-dimensional shape detection device and the three-dimensional shape detection method of the present invention, an edge image having distance information is generated, so that the corresponding edge can be easily detected by focusing only on the edge width and length of the edge image. There is an advantage of being. In addition, since it can be easily detected, the processing time is short, and there is an advantage that real-time processing is possible.

  The human eyeball is constantly unconscious of micro-vibrations such as tremor, which is very small high-frequency vibration, drift, which is small smooth movement, and microsaccade, which is linear motion. It is done in. Many studies have been conducted on the effects of such microvibration on visual recognition from both physiological and engineering perspectives, and it is considered to be a movement for detecting the depth of visual objects. . Among such human eyeball movements, focusing on the movement of the microsaccade, an apparatus and method have been developed that can detect a three-dimensional shape with high accuracy and high speed using the same movement as the microsaccade.

  The best mode for carrying out the present invention will be described below with reference to the drawings. FIG. 1 is a schematic diagram for explaining details of a camera unit used in the three-dimensional shape detection apparatus of the present invention. As illustrated, the camera unit 10 includes a lens 1 and an image sensor 2. The lens may be a short focus lens or a zoom lens, and the angle of view may be wide or narrow. These may be variously changed or selected depending on the purpose of use of the three-dimensional shape detection apparatus, the distance to the target object, and the like. The image sensor 2 is a photoelectric conversion element such as a CCD or CMOS, and converts an image formed on the image sensor by a lens into an electric signal. Then, the image processing unit 3 connected to the image sensor 2 generates an image using an electrical signal from the image sensor.

  The visual axis fine movement device 4, which is a characteristic part of the present invention, is connected to the camera unit 10. The visual axis fine movement device 4 causes the camera unit to vibrate laterally with respect to the optical axis, and the optical axis of the camera unit is slightly moved laterally. Although the vibration frequency is not particularly limited, for example, it may be appropriately vibrated at about several Hz to several tens Hz. This area can be variously set in consideration of the followability to the moving speed of the target object. Then, a plurality of target objects are photographed by the camera unit 10 while the optical axis of the camera unit is vibrating. It should be noted that since it is sufficient for the images to have at least two images at the positions of the left and right end portions that have moved the most, the timing of photographing by the visual axis fine movement device 4 and the camera unit 10 may be set, or a high-speed shutter. A large number of images may be generated using. However, as will be described later, when generating an edge image, it is preferable to generate a plurality of images because a clearer edge image is obtained when there are not only the left and right ends but also a plurality of images between them. This area may be appropriately adjusted according to the processing capability of the image processing unit 3 and the required three-dimensional shape detection speed. The image processing unit 3 may be an electronic computer such as a personal computer, but may be realized by a dedicated chip such as a DSP.

  The visual axis fine movement device 4 may be any device as long as the optical axis of the camera unit 10 can be vibrated minutely in a predetermined direction, but is not limited thereto. For example, piezoelectric elements, giant magnetostrictive elements, electrostatic actuators, hydraulic actuators, pneumatic actuators, electromagnets, optical actuators, shape memory alloys, polymer actuators, and the like can be used.

  Further, the visual axis fine movement device 4 may be a device that moves the camera unit 10 minutely, or a device that moves the lens 1 minutely. When the lens 1 is moved minutely, for example, a lens portion may be configured as shown in FIG. FIG. 2 is a schematic front view for explaining an example of the structure of the lens portion when the lens is moved minutely. As shown in the drawing, a visual axis fine movement device 4, for example, a piezoelectric element, more specifically, a piezoelectric ceramic or the like is disposed around the lens 1, and the piezoelectric ceramic 4 is connected to the lens 1 by an appropriate member. The illustrated piezoelectric ceramic 4 is divided into four parts. By controlling each part, the lens 1 can be moved in any direction in the x-axis direction and the y-axis direction. However, when the movement only to the left and right is sufficient, the piezoelectric ceramic does not need to be divided into four parts, and may be divided into two parts. The same configuration can be used when moving the camera unit 10 itself.

Referring to FIG. 1 again, the relationship between the distance from the target object to the lens, the distance from the lens to the image sensor, and the amount of change in the image formation point on the image sensor as the optical axis of the camera unit 10 changes will be described. . As shown in the figure, when the distance from the target object 20 to the lens 1 is d, the distance from the lens 1 to the image sensor 2 is h, and the moving distance of the camera unit is l, the imaging point p1 before the movement of the camera unit is The distance s between the image forming point P2 after the movement is expressed by the following equation.
s = hl / d (1)

  From Equation 1, it can be seen that the smaller d is, the larger s is, and vice versa. That is, the shorter the distance from the lens to the target object, the larger the moving width s of the imaging point. Therefore, the distance to the target object can be known to some extent by the moving width s of the image formation point. That is, it can be said that the movement width s includes distance information to the target object. Accordingly, if a plurality of images taken by moving the camera unit are combined, a plurality of lines of a nearby target object having a large movement width s will be gathered together, and the line of a remote target object will appear thick. Since s is small, it is not fat.

  Generation of an edge image in the image processing unit 3 will be described with reference to FIG. The edge image is an image in which the edge of the target object is emphasized in the image obtained by photographing the target object. FIG. 3A is an edge image according to the prior art when the visual axis fine movement device is not used. For example, the edge is emphasized by differentiating the gray level difference of the sharply changing portion of the gray level in the image. No matter how many images of the target object are taken from the same position, the edge image will not change, and the line width of the edge will be the same if the original edge is the same width, and it will not contain any information about distance .

  However, an edge image obtained by processing a plurality of images from the camera unit using the visual axis fine movement device according to the present invention by the image processing unit 3 is as shown in FIG. The near edge is thick and the far edge is thin. Therefore, it can be said that distance information is included in the edge width of this edge image, and the edges of the target object located at the same distance will have the same width. Will be located. FIG. 3B is an edge image obtained when the optical axis of the camera unit is moved horizontally in the horizontal direction in the drawing. By moving the optical axis to the left and right, the vertical direction according to the movement width s described above. The edges are emphasized. That is, distance information is included in the vertical edge, but distance information is not included in the horizontal edge. Therefore, if the edge is completely horizontal, the width of the edge does not increase, or it is not displayed at all by image processing that takes a difference.

  As described above, the image processing in the image processing unit 3 may be performed by differentiating the shade differences of the images and superimposing them, or by taking the differences of the shade values and saturation values of the corresponding pixels of the images. It may be normalized. Here, normalization refers to processing such as taking an absolute value of the difference or moving the entire value in the plus direction because a negative value is output when the difference between the images is taken. Further, the obtained image may be simplified by binarization. Further, if necessary, an exaggeration process or a thinning process may be added. Such image processing for generating an edge image includes conventional methods and all types of processing that can be developed in the future.

  A configuration in the case where the camera unit configured as described above is used for detection of a three-dimensional shape will be described below. FIG. 4 is a schematic block diagram for explaining the configuration of the three-dimensional shape detection apparatus of the present invention. As shown in the figure, the camera unit 10 is arranged in parallel on the left and right, and the respective outputs are input to the image processing unit 3. In this specification, the term “line of sight” means a line connecting the camera unit 10 and the target object 20, and the term “optical axis” means the optical center axis of the camera unit 10. To do.

  In the following, as shown in FIG. 4, an example in which the camera unit 10 is arranged at two predetermined positions and the target object 20 is photographed from different line-of-sight positions will be mainly described. However, the present invention is not limited to this. A single camera unit 10 may be moved to the left and right to a predetermined shooting position so as to shoot from two places with different lines of sight. In this case, although not limited to this, for example, the camera unit 10 may be placed on a rail or the like, and a moving mechanism that enables movement on the rail may be provided. Furthermore, the photographing position is not limited to two places. If it is a binocular type, two locations are sufficient. However, when it is desired to detect the surface shape or the like of a three-dimensional object more precisely and with high accuracy, the camera units may be arranged at a plurality of locations. That is, the camera units 10 can be arranged not only in the x-axis direction but also in the y-axis direction. Note that, from the viewpoint of ease of processing and the like, it is preferable that the imaging elements of each camera are arranged on the same plane.

  In the camera unit 10 arranged in this way, the visual axis fine movement device 4 slightly moves the optical axis of the camera unit in a direction connecting two predetermined photographing positions. The minute movement may be a translational movement in the normal direction of the optical axis of the camera unit 10. Moreover, not only a translational motion but a rotational motion may be sufficient. In the case of rotational movement, the center of rotation is other than the optical center of the lens, and is preferably on or near the optical axis. That is, any movement is possible as long as the optical axis of the camera unit 10 changes. For example, since the human eyeball is finely moved by rotational movement, when the three-dimensional shape detection device of the present invention is applied to the eyes of a humanoid robot, the actuator for causing the normal eye movement or the like is used as it is. Can also be used. Note that the rotation centers of the cameras are preferably arranged on the same plane from the viewpoint of ease of processing and the like. In addition, the rotation angle of the camera unit is preferably adjusted so that the movement width s of the target object close to infinity on the image sensor is about 0 to 3 pixels. As a result, the edge of the target object that is far from infinity is as thin as possible (for example, a width corresponding to two pixels) or is not captured at all.

  Thus, in this specification, “movement in a direction connecting two predetermined photographing positions” is not only translational movement in the normal direction of the optical axis of the camera unit, but also between camera units (lenses). It also includes a rotational movement around a predetermined point in a direction in which the two approach or move away from each other.

  The moving direction of the visual axis fine movement device 4 according to the present invention will be described with reference to FIG. FIG. 5 is a diagram for explaining an example in which the optical axes of the plurality of camera units 10 are moved in a direction connecting them, and is a schematic front view of the camera units viewed from the front. FIG. 5A shows an example in which there are two camera units, and the visual axis fine movement device finely moves the camera units 10a and 10b to the left and right in the direction of ab. When there are three camera units, for example, as shown in FIG. 5 (b), the camera unit is arranged at the position of the apex of the triangle, and is finely moved in the direction of ab during shooting with the camera units 10a and 10b. Then, the shooting position is switched and the camera units 10a and 10c are finely moved in the direction of ac during shooting. Further, the shooting position is switched so that the camera units 10b and 10c are finely moved in the direction of bc during shooting. Thus, by finely moving the optical axis of the camera unit in a plurality of directions, it becomes possible to detect the distance between the edges in the plurality of directions more accurately. Further, as shown in FIG. 5 (c), it is of course possible to make fine movements in the vertical and horizontal directions using the four camera units 10a to 10d. In the three-dimensional shape detection apparatus according to the present invention, since the optical axis only needs to be moved in the direction connecting the shooting positions, the three-dimensional shape detection apparatus is finely moved in a plurality of directions such as a combination of FIG. 5B and FIG. But of course.

  Furthermore, it is of course possible to slightly move the optical axis in a direction other than the direction connecting the camera units shown in FIG. By finely moving the optical axis in a plurality of directions, it is possible to execute movements corresponding to tremor and drift movements other than microsaccade, such as human eyeball movements, and to complement the edge image.

  Further, when using a plurality of camera units, it is preferable to synchronize each camera unit. That is, the minute movements of the respective camera units are the same movements, and at the same time, the minute movements are similarly performed with the same width in the same direction. Also, photographing is performed at the same time. This makes it possible to obtain a good edge image even for a moving target object.

  A method for associating the reference points of each edge image using the edge image obtained in this way will be described below. In general, a pattern matching method is used to obtain corresponding points of images taken using two camera units. Of course, the pattern matching method can also be used in the present invention. Further, for example, the edge image can be simplified so that high-speed processing can be performed by the method described below.

  FIG. 6 is a diagram for explaining each edge image obtained when a target object is photographed using two camera units having the visual axis fine movement device according to the present invention. Since the optical axis of the camera unit is finely moved left and right by the visual axis fine movement device, an image in which the vertical edge is emphasized is generated, but this is further simplified and only the vertical edge is extracted. The result of processing is shown in the figure. As described above, since the edge information of the edge image according to the present invention includes distance information, those at the same distance become edges having the same width, so the width of the edge and the length of the edge are used as a reference. In addition, the edges of the edge images from the two camera units can be associated with each other. That is, for the edges Ra, Rb, Rc of the left edge image and the edges La, Lb, Lc of the right edge image in FIG. 6, for example, an edge corresponding to the edge Rb of the left edge image is determined from the right edge image. In this case, an edge having the same edge width and length as the edge width and length of the edge Rb is detected from the right edge image. Then, the edge Lb having the same width and the same length is detected. Therefore, it is possible to associate the left and right edge images with the upper left vertex of these edges Rb and Lb as a reference point.

  When the edge width and length of the edge image are not constant with a complicated shape, a pattern matching method for associating by finding similar patterns in the left and right images may be used.

  Then, by obtaining the parallax using the edge image in which the reference points are associated, it is possible to detect the three-dimensional shape.

  As described above, according to the present invention, the edge image can be greatly simplified, and the reference point can be associated simply by using the edge length and width of the edge image as a reference. Can be greatly reduced. If necessary, edge images from two or more shooting positions can be added to detect a finer three-dimensional shape. Furthermore, after constructing the stereoscopic image, the smooth surface shape of the target object in the part surrounded by the edge can be obtained by analyzing, for example, the gray value and pasted to the stereoscopic image as the stereoscopic information of the surface of the target object. Is possible.

  Note that if there is a target object at a position close to the central axis between the left and right camera units, it can be associated from the edge image obtained as described above, but the target object is the central axis between the left and right camera units. If the position is far from the distance from the camera unit, the distance from each camera unit may change greatly even at the same edge, and the width of the edge may change despite the corresponding edge. In such a case, a corresponding edge may be detected in addition to the direction of the line of sight to the target object. For example, in the left edge image, since the direction of the line of sight can be determined by measuring how far the edge is from the center of the edge image, the edge in the direction of the left line of sight from the right edge image is also taken into account. It is possible to associate edges having the same length from among the two. Furthermore, it is also possible to convert each edge of the obtained edge image into a predetermined reference width based on the direction of the line of sight to the target object. That is, by converting the direction of the line of sight that is shifted to the width (reference width) when moved on the optical axis, it is possible to simply compare the width and length of the edge without considering the direction of the line of sight. It is also possible to do so.

  Note that the three-dimensional shape detection apparatus and the three-dimensional shape detection method of the present invention are not limited to the illustrated examples described above, and it is needless to say that various modifications can be made without departing from the scope of the present invention.

FIG. 1 is a schematic diagram for explaining details of a camera unit used in the three-dimensional shape detection apparatus of the present invention. FIG. 2 is a schematic front view for explaining the visual axis fine movement device of the lens unit when the lens is moved minutely. FIG. 3 is a diagram for explaining an edge image. FIG. 3A shows an edge image when the visual axis fine movement device is not used, and FIG. 3B shows an edge when the visual axis fine movement device is used. It is an image. FIG. 4 is a schematic block diagram for explaining the configuration of the three-dimensional shape detection apparatus of the present invention. FIG. 5 is a schematic front view of the camera unit for explaining an example in which the optical axes of a plurality of camera units are moved in a direction connecting the camera units. FIG. 6 is a diagram for explaining each edge image obtained when a target object is photographed using two camera units having the visual axis fine movement device according to the present invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Lens 2 Image pick-up element 3 Image processing part 4 Visual axis fine movement apparatus 10 Camera part 20 Target object

Claims (18)

A three-dimensional shape detection device for photographing a target object and detecting a three-dimensional shape thereof,
At least one camera means having an image sensor and a lens for photographing the target object from at least two predetermined photographing positions having different lines of sight to the target object;
Visual axis fine movement means for minutely moving the optical axis of the camera means in a direction connecting the at least two predetermined photographing positions;
By performing image processing on a plurality of images from the camera unit whose optical axis is slightly moved by the visual axis fine moving unit, edge images obtained by extracting the edges of the target object at the at least two predetermined photographing positions are respectively obtained. Image processing means for generating and detecting corresponding edges of the target object by comparing the respective edge images;
A three-dimensional shape detection apparatus comprising:   The three-dimensional shape detection apparatus according to claim 1, wherein the predetermined photographing position is arranged along at least one direction of a horizontal direction, a vertical direction, an oblique direction, or a plurality of these directions. Shape detection device.   3. The three-dimensional shape detection apparatus according to claim 1, further comprising a moving unit capable of moving the camera unit to a predetermined photographing position.   3. The three-dimensional shape detection apparatus according to claim 1, wherein the camera means includes at least two cameras, and each camera is arranged in advance at a predetermined photographing position.   5. The three-dimensional shape detection apparatus according to claim 4, wherein the at least two camera units are synchronized and photograph at the same time with the same minute movement.   6. The three-dimensional shape detection apparatus according to claim 1, wherein the visual axis fine movement unit slightly moves the lens.   6. The three-dimensional shape detection apparatus according to claim 1, wherein the visual axis fine movement unit slightly moves the camera unit.   8. The three-dimensional shape detection apparatus according to claim 1, wherein the visual axis fine movement means includes a piezoelectric element, a giant magnetostrictive element, an electrostatic actuator, a hydraulic actuator, a pneumatic actuator, an electromagnet, an optical actuator, and a shape. A three-dimensional shape detection apparatus comprising a memory alloy or a polymer actuator.   9. The three-dimensional shape detection apparatus according to claim 1, wherein the fine movement by the visual axis fine movement means is a translational movement in the normal direction of the optical axis of the camera means. Solid shape detection device.   9. The three-dimensional shape detection apparatus according to claim 1, wherein the fine movement by the visual axis fine movement means is a rotational movement about a predetermined point.   The three-dimensional shape detection apparatus according to claim 10, wherein the rotational angle of the rotational motion is an angle such that a moving width of the target object on the imaging device close to infinity is a width of 0 to 3 pixels. A three-dimensional shape detecting device characterized by the above. A three-dimensional shape detection method for photographing a target object and detecting the three-dimensional shape, the method comprising:
A process of photographing the target object from at least two predetermined photographing positions having different lines of sight to the target object with at least one camera means having an image sensor and a lens;
A step of minutely moving the optical axis of the camera means in a direction connecting the at least two predetermined photographing positions during the photographing step;
A process of generating edge images in which edges of the target object at the at least two predetermined photographing positions are extracted by performing image processing on a plurality of images from the camera unit in which the optical axis is moved minutely,
Comparing the respective edge images to detect corresponding edges of the target object;
A three-dimensional shape detection method comprising:   13. The three-dimensional shape detection method according to claim 12, wherein in the process of generating the edge image, a grayscale value or a saturation value of each corresponding pixel of the plurality of images is subjected to a differential process to perform normalization. 3D shape detection method.   The three-dimensional shape detection method according to claim 13, wherein the step of generating the edge image further performs binarization processing.   15. The three-dimensional shape detection method according to claim 12, wherein the step of detecting the corresponding edge includes the at least two predetermined edges based on the width and length of the edge of the edge image. A three-dimensional shape detection method characterized by associating edges of an edge image at a photographing position.   15. The three-dimensional shape detection method according to claim 12, wherein the step of detecting the corresponding edge associates edges of the edge image at the at least two predetermined photographing positions by a pattern matching method. A solid shape detection method characterized by the above.   17. The solid shape detection method according to claim 12, wherein the step of detecting the corresponding edge further uses a direction of the line of sight to the target object as a reference. Method. The three-dimensional shape detection method according to claim 17, wherein in the process of detecting the corresponding edge, the edge of the generated edge image is converted into a predetermined reference width based on the direction of the line of sight to the target object. A three-dimensional shape detection method comprising comparing respective edge images.