JP2009210520A - Distance measuring instrument and distance measuring method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a small information measuring instrument for measuring distance information on an object in a short time. <P>SOLUTION: This distance measuring instrument includes an image sensor camera 1, an image capture board 2, a memory 3 for storing image data, an image processing part 4 for calculating distance information to the object from the image data stored in the memory 3, and a display monitor 5 for displaying the distance information calculated by the image processing part 4. The image processing part 4 includes a reconstruction part 4a, a luminance information calculation part 4b, a distance information calculation part 4c, and a region selecting part 4d. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a technique for measuring a distance to an object based on image data of the object acquired by an imaging element through an imaging lens and a microlens array.

  In recent years, in order to construct a system that automates production lines using industrial robots and various FA (Factory Automation) devices, or to make intelligent robots intelligent, there has been a visual sensor for acquiring distance information of objects. It has become important. In particular, in a system using a robot arm, an image sensor camera is often used as a visual sensor for measuring the position, posture, shape, and the like of a workpiece.

  Also, in the manufacture of mounting boards for semiconductor devices and circuit components, there is an increasing need to measure height information of minute devices arranged in two dimensions such as solder bumps and gold bumps in order to control quality. Yes.

  In order to meet such needs, a conventional method for detecting height and posture information from an image of a two-dimensional image sensor camera, and a stereo three-dimensional information measurement system using a plurality of image sensor cameras. Etc. are used.

  As a method for measuring the height information of a relatively small object, a height measurement method using the principle of a confocal microscope has been proposed. For example, Japanese Patent Laid-Open No. 2003-75119 (Patent Document 1) describes a height information measuring device using the principle of a confocal microscope.

  This height information measuring apparatus will be described with reference to FIGS. FIG. 17 is a diagram illustrating a configuration of a height information measuring device using the principle of a confocal microscope. FIG. 18 is a diagram for explaining an algorithm for detecting the height information of the sample by the height information measuring device using the principle of the confocal microscope. As shown in FIG. 17, the confocal optical system 101 provided with the light source 103, the objective lens 104, and the two-dimensional image camera 102 moves the sample 106 in the height direction while moving the sample 106 in the height direction. It has the structure which each images the confocal image of a horizontal surface in several height positions (FIG. 18 (a) (b)). This height information measuring device compares the luminance information for each pixel of a plurality of confocal images to obtain the maximum luminance, and performs particle analysis using confocal pixel data including the pixel having the maximum luminance to extract a specific region. To do. Then, representative values of luminance and height in the extracted region are calculated to obtain sample height information (FIG. 18C).

Also, in a camera system using an image sensor, a plenoptic camera technology that can synthesize acquired image data by digital processing and move the focus to the front or back as desired later is “Light Field Photography with a Hand”. -Held Plenoptic Camera ", Stanford Tech Report CTSR 2005-02 (Non-Patent Document 1). The plenoptic camera described in Non-Patent Document 1 has an imaging lens similar to an ordinary camera lens, but is different from an ordinary camera in that a microlens array is accurately arranged on an image plane, and moreover than a microlens array. The difference is that an image sensor array (imaging device) having many imaging pixels is placed immediately behind the microlens array. Although the number of pixels in the final image is determined by the number of lenses in the microlens array, the direction and intensity of light incident on the microlens can be recorded by a large number of sensor pixels assigned to a single microlens. Using that data, an image focused on a predetermined distance can be reconstructed.
JP 2003-75119 A “Light Field Photography with a Hand-held Plenoptic Camera”, Stanford Tech Report CTSR 2005-02

  There is a problem that the depth distance and height information cannot be directly detected by the method of detecting height and posture information in a pseudo manner from the image of the two-dimensional image sensor camera. Further, distance measurement using a stereo three-dimensional information measurement system using a plurality of image sensor cameras has a problem that a plurality of cameras and complicated image processing are required.

  In addition, when acquiring a plurality of confocal images using the principle of the confocal microscope described in Patent Document 1, an operation of scanning in the surface direction of the stage and moving in the height direction is necessary. There is a problem that the moving mechanism system including the focus mechanism becomes large. In addition, in order to acquire two-dimensional image data at a plurality of times with different heights, there is a problem that the time required is long because of movement of the stage, stationary, auto-focusing of the objective lens, and the like.

  In the plenoptic camera technique described in Non-Patent Document 1, for example, when an operator inputs a focus position (distance) to acquired image data, an image focused on that position is arranged in data. Although it can be created by replacement (reconstruction), the distance information of the object cannot be calculated using it.

  The present invention has been made paying attention to such a problem, and an object of the present invention is to provide a small information measuring apparatus capable of measuring distance information of an object in a short time.

  The present invention according to one aspect is a distance measuring device for measuring a distance to an object, and an imaging lens that collects object light from the object and object light that has passed through the imaging lens are incident thereon. A camera including a condensing array composed of a plurality of concentrators and an image sensor that acquires image data of object light that has passed through the condensing array, the image sensor includes a plurality of pixels, An area selection means for extracting an edge from the image data and selecting an area including at least a part of the edge, which is divided into a plurality of pixel groups for detecting object light that has passed through one of the condensers; In this area, the plurality of pixels are rearranged based on the incident direction of the object light to the light collector determined by the position of the pixel that detected the object light in each pixel group, and the image sensor is positioned at each of the plurality of virtual positions. Should be obtained when you do Reconstructing means for generating a reconstructed image of the selected region, difference value calculating means for calculating a difference value of luminance between each pixel and pixels in the vicinity of the pixel for each of the plurality of reconstructed images, For each pixel position common to a plurality of reconstructed images, a virtual position corresponding to a reconstructed image having a pixel having a maximum difference value is specified, and a virtual position specified based on a parameter of a camera optical system is determined. Distance calculating means for converting and calculating the distance.

  Preferably, the region selection unit determines a processing region from the image data and selects a region including an edge in the processing region.

  More preferably, the region selection means determines a region including a continuous edge as a processing region.

  More preferably, an area including the entire edge included in the image data is determined as the processing area.

  More preferably, the region selection means selects an edge in the processing region and a region near the edge in the processing region.

  More preferably, the region selection means determines a region whose distance from the edge is equal to or less than a set value as a neighboring region.

  The present invention according to another aspect is a distance measurement method for measuring a distance to an object, in which object light from an object that has passed through an imaging lens and a condensing array composed of a plurality of concentrators. From the image data, the image data has a plurality of pixels, and the plurality of pixels are acquired by an image sensor that is divided into a plurality of pixel groups that detect object light that has passed through one of the condensers; Extracting an edge; selecting a region including an edge portion; and, in the selected region, based on an incident direction of the object light to the collector determined by the position of the pixel that detected the object light in each pixel group. Rearranging a plurality of pixels to generate a reconstructed image of a selected region to be obtained when the image sensor is located at each of a plurality of virtual positions; for each of the reconstructed images, Near pixel A step of calculating a difference value of luminance between the pixels and a virtual position corresponding to a reconstructed image having a pixel having a maximum difference value for each pixel position common to a plurality of reconstructed images And a step of calculating a distance by converting a virtual position specified based on a parameter of the optical system of the camera.

  According to the present invention, a plurality of pieces of image data in different focusing states can be generated from imaged data once by image processing, and distance information to an object can be obtained from the image data. Accordingly, a camera lens focusing mechanism and a moving mechanism such as a stage are not required, and a small distance information measuring apparatus can be realized. In addition, since information can be acquired with a single imaging operation, the time required to move the camera or object multiple times, the shooting time, or the matching operation of each image associated with the movement is not required, and distance measurement can be performed in a short time. .

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In addition, this invention is not limited to the following embodiment.

(1. Configuration of the present invention)
The configuration of the distance information detection apparatus according to the present invention will be described with reference to FIG. FIG. 1 is a schematic configuration diagram of a distance information detection apparatus according to an embodiment of the present invention.

  The distance information detection apparatus includes an image sensor camera 1, an image capture board 2 that acquires image data of an object photographed by the image sensor camera 1, a memory 3 that stores image data, and an image stored in the memory 3. An image processing unit 4 that calculates distance information from the data to the object, and a display monitor 5 that displays the distance information calculated by the image processing unit 4 are provided.

  The image sensor camera 1 acquires image data of an object. The image sensor camera 1 includes an imaging lens 11 (hereinafter referred to as a main lens), a microlens array 12 including a plurality of microlenses, and an imaging element 13. As the imaging device 13, for example, a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) sensor can be used.

  Object light from the object is incident on the main lens 11. The main lens 11 condenses object light from the incident object. Object light that has passed through the main lens 11 enters the microlens array 12. Each of the plurality of microlenses included in the microlens array 12 collects object light. Note that the microlens array 12 is an example of a light collection array including a plurality of light collectors. For example, instead of the microlens array 12, a pinhole array composed of a plurality of pinholes may be used.

  The image sensor 13 acquires image data of object light that has passed through the microlens array 12. The image sensor 13 has a plurality of pixels. The pixels of the image sensor 13 and the image data acquired by the image sensor 13 will be described in detail later.

  The image capture board 2 captures the image data acquired by the image sensor 13 and converts it into a form that allows subsequent processing. The memory 3 stores the image data transferred from the image capture board 2.

  The image processing unit 4 reads image data from the memory 3 and performs a process of calculating a distance to the object based on the read image data. The image processing unit 4 obtains the region selection unit 4d that selects a partial region of the image data and the pixel data included in the selection region, and the image sensor 13 is located at each of a plurality of different virtual positions. A reconstruction unit 4a that generates a reconstructed image to be performed, a luminance information calculation unit 4b that calculates luminance information in each of the plurality of reconstructed images, and a distance information calculation unit 4c that calculates a distance to an object. Become. Details of the processing performed by the image processing unit 4 will be described later.

(2. Features of image sensor camera)
In the present invention, there is a feature in the configuration of the optical system of the image sensor camera 1, and the feature will be described with reference to FIG. FIG. 2 is a diagram illustrating an optical system of the image sensor camera 1. For simplicity, FIG. 2 shows a case where the object 14 is a point light source.

  The microlens array 12 of the image sensor camera 1 is disposed at a position where the object 14 is substantially imaged by the main lens 12 when the object 14 is at the position A. The position A at this time is called a focal point position, the side closer to the main lens 11 than the focal point position is called a near point position, and the side farther from the main lens 11 than the focal point position is called a far point position. The image pickup device 13 is disposed almost at the focal position of the microlens array 12. The light of the object enters the image sensor 13 through the microlens. In addition, it is assumed that the optical system of the image sensor camera 1 is adjusted so that light that has passed through each microlens enters the image sensor 13 without overlapping each other.

  Next, an image captured by the image sensor camera 1 will be described with reference to FIGS.

  The imaging of the object 14 will be described with reference to FIG. FIG. 3 is a diagram illustrating a state of light collection when the object 14 is at the near point position B or the far point position C. When the object 14 is at the position B, the light from the object 14 is condensed (imaged) on the surface D by the main lens 11. Further, when it is at the position C, it is condensed (imaged) on the surface F. The surface E is a light condensing surface with respect to the in-focus position A and substantially corresponds to the position of the microlens array 12.

  Next, what kind of image is picked up by the image pickup device 13 when the object 14 is at the positions A, B, and C in FIG. 3 will be described with reference to FIGS. 4 and 5. FIG. 4 is a diagram showing the shape of the incident light 15 on the microlens array 12, and FIG. 5 is a diagram showing the shape of the incident light 16 on the image sensor 13.

  First, the shape of the incident light 15 incident on the microlens array 12 when the object 14 is at positions A, B, and C in FIG. 3 will be described with reference to FIG. As shown in FIG. 4, in the present embodiment, the microlens array 12 is composed of circular microlenses arranged in a two-dimensional plane. Each microlens is given a number M (i, j) for identifying them. 4 (a), 4 (b), and 4 (c) respectively show the case where the object 14 is at the far point position C, the in-focus position A, and the near point position B. It is a figure which shows the shape of the incident light 15 on the micro lens array 12. FIG.

  The light from the object 14 at the in-focus position A is collected almost on the microlens M (2, 2) as shown in FIG.

  When the object 14 is located at the far point position C, the light from the object 14 is once condensed through the main lens 11, then further defocused and spread on the microlens 12. Therefore, as shown in FIG. 4A, the incident light 15 on the microlens array 12 is transmitted from the peripheral lenses M (1,2), M (2,1), M (M (2,2) of the microlens M (2,2). 2, 3) and M (3, 2).

  When the object 14 is at the near point position B, light in a defocused state before focusing is incident on the microlens 12. Therefore, as shown in FIG. 4B, the incident light 15 on the microlens array 12 is transmitted to the peripheral lenses M (1,2), M (2,1), M (M (2,2) of the microlens M (2,2). 2, 3) and M (3, 2).

  The incident light 16 on the image sensor 13 will be described with reference to FIG. As shown in FIG. 5, the image sensor 13 includes a plurality of pixel groups. Each pixel group is assigned a number T (i, j). Each pixel group detects object light that has passed through one of the microlenses. In other words, the light that has passed through the microlens M (i, j) enters the pixel group T (i, j). In addition, although the figure has shown the case where a pixel group is comprised by the pixel of 10x10, the number of the pixels which comprise a pixel group is not restricted to this.

  When the object 14 is at the in-focus position A, the light condensed on the microlenses M (2, 2) as shown in FIG. Therefore, the light incident on the image sensor 13 spreads over the entire surface of the pixel group T (2, 2) of the image sensor 14 as shown in FIG.

  When the object 14 is located at the far point position C, the peripheral lenses M (1,2), M (2,1), M (2,3), as shown in FIG. Light that has spread to M (3, 2) also enters the image sensor 13. As shown in FIG. 5A, the light incident on these peripheral lenses is pixel groups T (1,2), T (2,1), T corresponding to the respective microlenses, as shown in FIG. Incident on a part of (2, 3) and T (3, 2).

  FIG. 5C is a diagram illustrating the shape of light incident on the image sensor 13 when the object 14 is located at the near point position B. FIG. Similarly to the case where the object is located at the far point position C, light is also emitted to a part of the pixel groups T (1,2), T (2,1), T (2,3), and T (3,2). Incident.

(3. Image reconstruction)
Next, image reconstruction on each virtual plane, which is a basic operation for calculating distance information in this apparatus, will be described.

  When reconstructing an image, it is important that information on the incident direction of the object light on the microlens can be obtained from the image data acquired by the image sensor camera 1. The fact that information on the incident direction can be obtained from the image data will be described with reference to FIG. 5 again. For example, as shown in FIG. 5A, the light from the object 14 at the far-point position C is centered on the pixel group T (2, 2), and the surrounding pixel groups T (1, 2), T The incident light is incident on some fixed pixels outside (2,1), T (2,3), and T (3,2). On the other hand, as shown in FIG. 5C, the light from the object 14 at the near point position B is centered on the pixel group T (2, 2), and the surrounding pixel group T (1, 2). ), T (2,1), T (2,3), and T (3,2) are incident on some fixed pixels inside. Thus, information on the incident direction of the object light to the microlens installed at the position corresponding to each pixel group can be obtained depending on which pixel among the pixels included in each pixel group is incident.

  Based on the information on the incident direction, the position of the virtual plane (virtual position), and the parameters of the optical system (such as the distance between the main lens 11 and the microlens array 12), the pixel data acquired by the image sensor 13 is rearranged. Thus, it is possible to acquire image data to be acquired when the image sensor 13 is in a virtual position.

  A specific example of rearrangement will be described with reference to FIG. FIG. 6 is a diagram showing a reconstructed image obtained by converting data of a captured image of the object 14. For example, FIG. 6A shows the distribution of light on the virtual plane F reconstructed from the image data shown in FIG. 5A, and the light beam at the in-focus position in FIG. 5B. It becomes almost the same as the state. Only the light incident on the surrounding fixed pixels, that is, the pixel output is collected and added to the pixel output in the vicinity of the center of the pixel group T (2, 2) to be the output of the pixel group T (2, 2) again. Thus, an image on the virtual plane F can be created. On the other hand, for the object (point image) at the near point position B, the spread (division state) on the image sensor 13 is different from the far point state as shown in FIG. Similarly, the reconstructed image in FIG. 6C can be generated by adding together the pixel outputs inside the surrounding pixel group.

  Although an example in which the object is a point image is shown here, an image can be reconstructed by a similar operation even in the case of an object having a size. However, in the case of an image having a size, since light is incident on all the microlenses, the same operation is performed on the pixel groups T (1,1) to T (3,3) corresponding to each microlens. There is a need.

  A reconstructed image in the case where the object 14 is not a point image will be described with reference to FIGS. 7 and 8. FIG. 7 is a diagram illustrating a state of light collection when the arrow-shaped object 14 is at the near point position B, and FIG. 8 is a diagram illustrating an image acquired on the virtual plane illustrated in FIG. is there.

  Since the object 14 is at the near point position B, the acquired image on the virtual plane E almost at the position of the image sensor 13 is a blurred image as shown in FIG. However, when the virtual plane is changed to D1, D2, and D3, the respective reconstructed images are as shown in FIGS. 8B, 8C, and 8D. When changing to D3, since D3 is a virtual plane with respect to position B, a focused image of the object at position B is obtained.

  8 corresponds to an image acquired through the microlens array 12, and when the image is enlarged, as shown in FIG. 9, the pixel group for each microlens is an image as one unit. . FIG. 9 is a diagram showing an enlarged image of a portion surrounded by a frame in FIG.

  As described above, by performing image processing on imaging data obtained by one imaging, images on a plurality of virtual planes can be reconstructed.

(4. Determination of selection area)
Next, selection of a region to be processed, which is a feature of the present invention, will be described. As can be seen from FIGS. 8 and 9, when the virtual plane is changed, a change appears in the pixel in the edge portion (a portion where the luminance change on the image is large), but the image changes almost in the region other than the edge portion. Does not occur.

  Thus, an appropriate reconstructed image can be obtained even when only the pixels included in the edge portion of the captured image or in the vicinity region of the edge portion are rearranged. In this way, if the images are rearranged using the pixel data of a part of the area, the processing speed is greatly increased compared to the case where the image is reconstructed for the entire area, and the image necessary for the processing is also obtained. A small amount of memory can be used. The present invention focuses on this point.

  Now, a method for detecting an edge portion on a captured image will be described with reference to FIG. FIG. 10 is a diagram for explaining detection of an edge portion. Assume that the captured image is as shown in FIG. The area selection unit 4d applies a spatial filter for detecting edges to this image. As a spatial filter for edge detection, there is a first-order differential filter that calculates a difference from adjacent pixels. For example, a sobel filter and a Roberts filter have been proposed. The result of applying the primary filter to FIG. 10 (a) is shown in FIG. 10 (b). In this figure, the darker the color, the greater the difference between adjacent pixels. Here, FIG. 10C shows a distribution of pixel values in a portion indicated by a chain line on the image of FIG. The edge portion of the large arrow is almost in focus and the difference value is large because the contrast with the background is large. The edge portion of the small arrow is slightly out of focus, and the difference value is slightly smaller because the contrast with the background is not so large. In addition, the round object is blurred as a whole, and the difference value is small because there is not much contrast with the background.

  Since the difference value indicates a distribution correlated with the edge in this way, an edge region can be extracted by providing a certain threshold value and performing threshold value processing on the difference value image. For example, a portion having a value larger than the threshold value is regarded as an edge. The threshold value may be appropriately determined according to the purpose, and the threshold value may be set large to detect only edges with high contrast, and conversely, the threshold value should be set small to detect edges with low contrast. FIG. 10D shows the edges detected by examining the distribution of edges for all the captured images.

In order to perform image reconstruction, it is preferable that the selected region includes a region near the edge in addition to the detected edge. This is because the region centered on the edge includes the light distribution by far points and near points. Since this region varies depending on the optical system, it is necessary to set it appropriately. Usually, however, when the pixel group for each microlens is used as one unit, it is a region of several pixel groups centered on the edge. For example, the position coordinates of the edge _{(x edge, y edge),} x -direction, _x the length of the pixel group in the y-direction, respectively p, as _{y p, x edge -sx p ≦} x ≦ x edge + tx p, y edge A set of (x, y) satisfying −uy _p ≦ x ≦ y _edge + by _p is set as a selection region. Here, s, t, u, and v are positive integers. In general, s = t = u = v.

  The shaded portion in FIG. 10E is an area selected corresponding to the edge in FIG. A selected region image as shown in FIG. 10F can be obtained by cutting out the region portion of FIG. 10E on FIG.

However, FIG. 10 (e) and FIG. 10 (f) are diagrams only showing an image of the selection region. Strictly speaking, the shape of the selection region is not as shown in these drawings. It becomes like this. FIG. 11 is a diagram showing the relationship between edges and selected areas. This figure selection area _corresponds to the case where a set of _{x edge -x p ≦ x ≦ x} edge + x p, satisfies the _{y edge -y p ≦ x ≦ y} edge + y p (x, y). A portion obtained by combining the vicinity region and the edge portion in FIG. 11 becomes the selection region.

  In the above description, the selection area is determined from the entire image data. However, the selection area may be determined from a set area (hereinafter referred to as a processing area) in the image data. The processing area is determined according to a user instruction, for example. According to this configuration, when the user confirms the image data displayed on the monitor 5 and determines the processing area, the amount of calculation can be reduced and the processing speed can be increased.

  When selecting a processing region, it is not preferable to select a connection edge. The reason for this will be described with reference to FIG. FIG. 12 is a diagram for explaining division of a single edge. Assume that there is an image as shown in FIG. Assume that the image shown in FIG. 12B is obtained as a result of detecting the edge of the image. FIG. 12C shows an enlarged view of a part of the edge (the part surrounded by a square) on the image.

  Assume that when selecting an area for the image in FIG. 12B, a processing area R for dividing the edge of the image as shown in FIG. 12D is selected. FIG. 12E shows an enlarged view of a portion surrounded by a square in FIG. When the processing region R is selected, information on the region including the light distribution in the diagonally lower right direction and the downward direction is lost for the pixel group in the G portion in FIG. Therefore, it is not preferable for reconstruction to select a region where the edge is divided. As described above, when selecting a region, it is preferable that the region includes a continuous edge without dividing the edge in the middle. In order to select a region including a continuous edge, for example, a region may be selected so as to include all detected edge regions.

  However, if an area where distance information needs to be detected is determined in advance, an area that divides the edge may be selected. If it demonstrates using FIG.12 (e), if the distance information of the part of G is unnecessary, you may select the process area | region R. FIG. By performing such region selection and processing only necessary regions, the entire processing time can be shortened.

The series of processing described here is performed by the region selection unit 4d.
(5. How to get distance information)
Next, a method for reconstructing an image of a virtual surface for a region selected by the above method and further acquiring distance information (three-dimensional information) of an object in the image will be described with reference to FIG. To do. FIG. 13 is a diagram for explaining image data generated in each process of obtaining distance information.

  In acquiring distance information, first, image data of the object is acquired by the image sensor camera 1. The acquired image data is stored in the memory 3 via the image capture board 2. Then, the area selection unit 4d included in the image processing unit 4 determines the selection area by the method described above. This image data is assumed to be f (i, j) as shown in FIG.

  Thereafter, the image processing unit 4 processes the image data f (i, j) to obtain distance information of the object. This process is largely divided into “reconstruction step”, “luminance information calculation step”, and “distance information calculation step”. Hereinafter, each processing will be described.

(1) Reconstruction Step The reconstruction unit 4a included in the image processing unit 4 rearranges the image data f (i, j), and reconstructs on N virtual planes as shown in FIG. 13B. Image data gn (i, j) is generated. Here, n (n = 1 to N) indicates the order of the virtual surfaces. This rearrangement is expressed as gn (i, j) = An (f (i, j)) using the focal plane conversion coefficient An (n = 1 to N). Here, An is determined based on the incident direction of the object light on each microlens determined by the position of the pixel that detected the object light in each pixel group, the virtual position, and the parameters of the optical system.

  Note that, for example, the reconstruction unit 4a is not particularly limited, but in the present embodiment, the reconstruction unit 4a acquires reconstruction image data for a plurality of virtual positions arranged at predetermined intervals. In addition, when the in-focus virtual position is calculated by the method described below, the reconstruction unit 4a further narrows the interval between the virtual positions near the in-focus virtual position so that the virtual position is narrow. On the other hand, a reconstructed image may be acquired. According to this configuration, the accuracy of distance detection can be increased.

In this step, one image data f (i, j) is converted into N pieces of reconstructed image data as shown in FIG. 13B, that is, g ₀₁ (i, j) = An (f (i, j j)), g ₀₂ (i, j) = An (f (i, j)),..., gn (i, j) = An (f (i, j)),..., gN (i, j) = An (f (i, j)) is obtained.

(2) Luminance information calculation step The luminance information calculation unit 4b calculates luminance information related to the luminance from the reconstructed image data gn (i, j). Here, the luminance information is specifically information related to a luminance difference value. The calculation of information relating to luminance is performed according to the following procedure.

  First, the luminance information calculation unit 4b converts image data hn () obtained by converting each pixel group as shown in FIG. 13C into pixels having uniform luminance for each reconstructed image data gn (i, j). i ′, j ′). Here, the uniform luminance may be a luminance corresponding to the luminance of the pixels included in each pixel group. For example, the luminance information calculation unit 4b calculates the average value of the luminance of the pixels included in each pixel group or the total luminance as the uniform luminance.

  In this embodiment, it is assumed that a pixel group composed of m × m pixels is converted (averaged) into one pixel having uniform luminance. This conversion is expressed as hn (i ′, j ′) = Bn (gn (i, j)) where Bn is an averaging coefficient.

  A specific example of the averaging process will be described with reference to FIG. FIG. 14 is a diagram showing the luminance distribution of the pixel group when the averaging process is performed on the image shown in FIG. FIGS. 14A to 14D are obtained by averaging the luminance distributions of FIGS. 9A to 9D in the pixel group.

With this step, the amount of data to be processed can be reduced to 1 / m ² . Therefore, the image processing speed of the subsequent processing is greatly improved. However, this process is not essential.

Next, the luminance information calculation unit 4c calculates a luminance difference value between pixels for each reconstructed image. When the above-described averaging process is performed, the luminance information calculation unit 4c calculates a luminance difference value between each pixel group and the adjacent pixel group in each image data hn (i ′, j ′). Difference value image data kn (i ′, j ′) as shown in 13 (d) is created. The relationship between hn and kn is expressed as kn (i ′, j ′) = Cn (h (i ′, j ′)), where Cn is the difference value conversion coefficient. Here, the difference value conversion coefficient Cn is, for example, in the x direction with respect to the pixel group of (i ′, j ′) when using a generally used edge detection method called the Roberts method. Differential value dx = h (i ′, j ′) − h (i ′ + 1, j ′ + 1), y-direction differential value dy = h (i ′ + 1, j ′) − h (i ′, j ′ + 1) And (dx ² + dy ² ) ^1/2 . However, the difference value conversion coefficient is not limited to this, and various filter windows such as a simple differentiation method, a Sobel method, and a Laplacian method using a secondary differentiation can be set. As shown in FIG. 13D, N difference value images are created.

  When the above-described averaging process is not performed, the luminance information calculation unit 4b simply calculates a difference in luminance value between adjacent pixels for each pixel of each reconstructed image, and calculates a difference value image. Get the data.

(3) Distance information calculation step The distance information calculation unit 4c calculates the distance information of the object in the following procedure based on the information obtained in the luminance information calculation step.

  First, the distance information calculation unit 4c detects the maximum value of the difference value intensity at each pixel position p = (i ′, j ′) common to a plurality of reconstructed images based on the difference value image data, and the maximum An n value (hereinafter referred to as a representative value) from which a value is detected is obtained, and representative value image data s (i ′, j ′) as shown in FIG. 13E is generated. That is, the value of n that maximizes kn (i ′, j ′) to kN (i ′, j ′) at each p = (i ′, j ′) is set as s (i ′, j ′). As shown in FIG. 9 or FIG. 14, the pixel group luminance change at the edge portion is the largest in the image at the in-focus position. Therefore, it can be determined that the position of the virtual surface having the maximum edge strength value indicates the in-focus state. Using this fact, the distance to the object can be calculated.

  Alternatively, the representative value may be obtained from an approximate curve of the difference value distribution. This finding method will be described with reference to FIG. FIG. 15 is a diagram for explaining a method of obtaining the representative value from the approximate curve of the difference value distribution. FIG. 15A is a diagram showing a reconstructed image of two objects arranged at different distances. FIG. 15B is a plot of the difference value intensity at the pixel position (14A, 15B) near the edge of each object shown in FIG. 15A with respect to the position (n value) of the virtual plane. . For each object, an approximate curve of difference value intensities plotted with respect to the n value is obtained at the pixel position (i ′, j ′), and the n value of the peak of the approximate curve (n1, n2 in FIG. 15B). ) As a representative value s (i ′, j ′). This representative value indicates the position of the virtual plane in focus on the image of the object.

  Lastly, the distance information calculation unit 4c uses a distance information detection device to obtain a representative value image s (i ′, j ′) from which the maximum difference value intensity is obtained at each pixel position using magnification data of a known optical system. Is converted into an absolute distance d (i ′, j ′) from the object to the object to generate final distance image data d (i ′, j ′) as shown in FIG.

  The flow of processing performed by the present invention will be described with reference to FIG. FIG. 16 is a flowchart for explaining processing performed by the present invention.

  First, the image processing apparatus 4 acquires image data in step S101. This image data is obtained by photographing the object once by the image sensor 1.

  Next, in step S102, the image processing apparatus 4 extracts an edge region from the image data.

  In step S103, the image processing apparatus 4 determines a selection area including an edge area from the image data.

  Subsequently, in step S104, the image processing device 4 rearranges the image data in the selected region, and creates reconstructed images for a plurality of virtual positions.

  In step S105, the image processing apparatus 4 averages each pixel group. That is, each pixel group is converted into a pixel having uniform luminance. However, this step is not mandatory.

  In step S106, the image processing apparatus 4 creates difference value image data from the reconstructed image data.

  Further, in step S107, the image processing device 4 creates representative value data from the difference value image data.

  In step S108, the image processing apparatus 4 converts the representative value data into the distance data of the object based on the magnification of the optical system, and creates distance data.

  With the distance information calculation algorithm, a plurality of images in different focusing states (virtual planes) can be generated from a single image data by image processing, and a distance image of the object can be obtained from the data. It is also possible to obtain the tilt and posture from the positional relationship before and after in the depth direction and the distance information of different edges of the object.

  As a conventional plenoptic camera as described in Non-Patent Document 1, an image focused on a set distance can be reconstructed, but distance information of the object of the image cannot be obtained. . On the other hand, distance information can be calculated in the present invention.

  Further, as in Patent Document 1, after autofocusing is performed by changing the height of an object on a stage or the like, a plurality of images are acquired, and height information is obtained based on the luminance information. In contrast, the method of the present invention eliminates the need for a camera lens focusing mechanism and a moving mechanism such as a stage, and is therefore small and low-cost.

  In addition, since distance data can be calculated from image data acquired by a single imaging operation, the time required to move the camera or object multiple times, the shooting time, or the matching operation of each image associated with the movement is not required, making significant measurements. Time can also be shortened. The operation for generating a plurality of reconstructed images from one acquired image can be combined in parallel without the need for serial processing, so that a plurality of shooting times can be omitted, which is advantageous for high-speed measurement.

  In particular, according to the present invention, since the edge is extracted from the image data and the distance data is obtained for the region including the edge, the processing can be greatly speeded up. Also, the image memory required for processing can be reduced with a small capacity.

  Furthermore, since this optical system is configured to incorporate a microlens array in the immediate vicinity of the image sensor, it is almost the same size as a conventional image sensor camera, so it is widely used in visual sensors such as robots for FA applications. Because it fits in the same outer diameter size as the image sensor camera, compatibility (replacement) with conventional cameras is high.

  In addition, in this method, the number of calculations and processing for calculating distance information are performed by using luminance information in units of pixels composed of a plurality of pixels corresponding to one microlens instead of in units of pixels of the image sensor. Time can be shortened.

  The embodiments disclosed herein are illustrative and non-restrictive in every respect. The scope of the present invention is shown not by the above-described embodiment but by the scope of claims, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

  The present invention relates to distance measurement using a camera, and is used, for example, for measuring a three-dimensional position, posture, and shape of a machine part in a production line or the like, or for inspecting an outer shape of a product.

It is a schematic block diagram of the distance information detection apparatus which concerns on embodiment of this invention. 1 is a diagram illustrating an optical system of an image sensor camera 1. FIG. It is a figure which shows the mode of condensing in case the target object 14 exists in the near point position B or the far point position C. It is a figure which shows the shape of the incident light 15 on the micro lens array 12. FIG. It is a figure which shows the shape of the incident light 16 on the image pick-up element 13. FIG. It is a figure which shows the reconstruction image obtained by converting the data of the picked-up image of the target object. It is a figure which shows the mode of condensing in case the arrow-shaped target object 14 exists in the near point side B. FIG. It is a figure which shows the image acquired by the virtual surface shown in FIG. It is a figure which shows the image which expanded the part enclosed with the frame in FIG. It is a figure for demonstrating the detection of an edge part. It is the figure which showed the relationship between an edge and a selection area. It is a figure for demonstrating the division | segmentation of one edge. It is a figure for demonstrating the image data produced | generated at each process of acquisition of distance information. It is a figure which shows the luminance distribution of a pixel group at the time of performing the averaging process with respect to the image shown in FIG. It is a figure for demonstrating the method of calculating | requiring a representative value from the approximated curve of difference value distribution. It is a flowchart for demonstrating the process which this invention performs. It is a figure which shows the structure of the height information measuring apparatus using the principle of a confocal microscope. It is a figure explaining the detection algorithm of the sample height information by the height information measuring device using the principle of a confocal microscope.

Explanation of symbols

  1 image sensor camera, 2 image capture board, 3 memory, 4 image processing unit, 4a reconstruction unit, 4b luminance information calculation unit, 4c distance information calculation unit, 4d region selection unit, 5 display monitor, 11 imaging lens, 12 micro Lens array, 13 Image sensor, 14 Object, 15 Incident light on microlens array 12, 16 Incident light on image sensor 13

Claims (7)

A distance measuring device for measuring a distance to an object,
An imaging lens that condenses object light from a target, a condensing array that includes a plurality of condensers on which the object light that has passed through the imaging lens is incident, and an image of the object light that has passed through the condensing array A camera including an image sensor for acquiring data;
The imaging device has a plurality of pixels, and the plurality of pixels are divided into a plurality of pixel groups that detect the object light that has passed through one of the condensers,
An area selecting means for extracting an edge from the image data and selecting an area including at least a part of the edge;
In the selected region, the plurality of pixels are rearranged based on the incident direction of the object light to the condenser determined by the position of the pixel that detected the object light in each pixel group, and the imaging element Reconstructing means for generating a reconstructed image of the selected region to be obtained when is located at each of a plurality of virtual positions;
For each of the plurality of reconstructed images, difference value calculating means for calculating a difference value of the luminance between each of the pixels and pixels in the vicinity of the pixel;
For each pixel position common to the plurality of reconstructed images, the virtual position corresponding to the reconstructed image having the pixel having the maximum difference value is specified, and based on the parameters of the optical system of the camera A distance measuring device comprising: distance calculating means for converting the identified virtual position and calculating the distance.   The distance measuring device according to claim 1, wherein the region selecting unit determines a processing region from the image data and selects a region including an edge in the processing region.   The distance measuring device according to claim 2, wherein the region selecting unit determines a region including a continuous edge as the processing region.   The distance measuring device according to claim 3, wherein an area including the entire edge included in the image data is determined as the processing area.   5. The distance measuring device according to claim 2, wherein the region selecting unit selects an edge in the processing region and a region near the edge in the processing region. 6.   The distance measuring device according to claim 5, wherein the region selecting unit determines a region whose distance from the edge is equal to or less than a set value as the neighboring region. A distance measurement method for measuring a distance to an object,
The image data of the object light from the object that has passed through the imaging lens and a light collecting array including a plurality of light collectors has a plurality of pixels, and the plurality of pixels includes one of the light collectors. Acquiring with an imaging device that is divided into a plurality of pixel groups that detect the object light that has passed; and
Extracting an edge from the image data;
Selecting a region including the edge portion;
In the selected region, the plurality of pixels are rearranged based on the incident direction of the object light to the condenser determined by the position of the pixel that detected the object light in each pixel group, and the imaging element Generating a reconstructed image of the selected region to be obtained when is located at each of a plurality of virtual locations;
For each of the reconstructed images, calculating a difference value of the brightness between each of the pixels and a pixel near the pixel;
Identifying the virtual position corresponding to the reconstructed image having the pixel with the largest difference value for each pixel position common to the plurality of reconstructed images;
Converting the identified virtual position based on a parameter of an optical system of the camera and calculating the distance.