JP2009229125A - Distance measuring device and distance measuring method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a small-sized inexpensive distance measuring device capable of measuring accurately distance information. <P>SOLUTION: This distance measuring device is equipped with an image sensor camera 1, an image capture board 2, a memory 3 for preserving image data, an image processing part 4 for calculating distance information to an object from the image data preserved in the memory 3, and a display monitor 5 for displaying the distance information calculated by the image processing part 4. The image processing part 4 comprises a reconstitution part 4a, a brightness information calculation part 4b, a distance information calculation part 4c, and a mask part 4d. The mask part 4d converts a pixel value corresponding to a non-lens part of a microlens array 12 inside the image sensor camera 1 into a prescribed value. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a distance measuring device and a distance measuring method for measuring a distance to an object.

  In recent years, visual sensors have become important for systems that automate production lines using industrial robots and various FA devices and intelligent robots. In particular, in a system using a robot arm, an image sensor camera is often used as a visual sensor for measuring the position, posture, shape, and the like of a workpiece.

  Also, in the manufacture of mounting boards for semiconductor devices and circuit components, there is an increasing need to measure height information of minute devices arranged in two dimensions such as solder bumps and gold bumps in order to control quality. Yes.

  In order to meet such needs, a stereo three-dimensional information measurement system using a plurality of image sensor cameras has been conventionally used. However, this system has a problem that a plurality of cameras and complicated image processing are required.

  As a method for measuring the height information of a relatively small object, a height measurement method using the principle of a confocal microscope has been proposed. For example, a height information measuring device using the principle of a confocal microscope described in Japanese Patent Laid-Open No. 2003-75119 (Patent Document 1) has a light source 103, an objective lens 104, and a two-dimensional image as shown in FIG. A configuration in which a confocal optical system 101 provided with a camera 102 captures a confocal image 107 on a horizontal plane at a plurality of height positions different from each other in the height direction while moving the sample 106 in the height direction. (FIGS. 19A and 19B). Then, the luminance information for each pixel of a plurality of confocal images is compared, particle analysis is performed using the confocal pixel data including the pixel having the maximum luminance, the specific region is extracted, and then the luminance in the extracted region is increased. The representative value of the height is calculated to obtain the height information of the sample (FIG. 19 (c)).

  In addition, in a camera system using an image sensor, a plenoptic camera technology that synthesizes acquired image data by digital processing and can move the focus to the front or back as desired later is “Single Lens Stereo with a Plenoptic”. "Camera", IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 14, NO. 2, FEBRUARY 1992 (Non-patent Document 1) and “Light Field Photographic with a Hand-Held Plenoptic Camera”, Stanford Tech Report CTSR 2005-02 (Non-patent Document 2).

  In addition to ordinary camera lenses that form images in the usual way, plenoptic cameras include microlens arrays that are precisely placed in the image plane of the camera lens. Furthermore, the image sensor array (image sensor array) is provided immediately behind the micro lens array and has more image sensors than the micro lens array.

Although the number of pixels in the final image is determined by the number of lenses in the microlens array, the direction and intensity of light incident on the microlens can be recorded by a large number of sensor pixels assigned to a single microlens. By using the data, an image focused on a predetermined distance can be reconstructed.
JP 2003-75119 A “Single Lens Stereo with a Plenoptic Camera”, IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 14, NO. 2, FEBRUARY 1992 “Light Field Photography with a Hand-held Plenoptic Camera”, Stanford Tech Report CTSR 2005-02

  When a plurality of confocal images are acquired using the principle of a confocal microscope as in Patent Document 1, an operation of scanning in the surface direction of the stage and moving in the height direction is necessary, and an autofocus mechanism is included. Necessary to increase the size of the moving mechanism system and to acquire 2D image data multiple times at different heights (including the time required to move the stage, the time required to stand still, and the auto-focusing time for the objective lens, etc.) There are issues such as becoming.

  Further, in Non-Patent Document 1, for example, when an operator inputs a focus position (distance) to acquired image data, an image focused at that position is created by rearranging (reconstructing) data. Although it is shown, it is not shown that the distance information of the object is calculated using it.

  There is also a problem that noise is generated in the reconstructed image due to non-lens portions existing between or around each microlens of the microlens array. Light (unnecessary light) that has passed through the non-lens portion and entered the imaging element appears as noise in the reconstructed image.

  The present invention has been made paying attention to such a problem, and an object of the present invention is to provide a small and low-cost distance measuring device capable of measuring distance information with high accuracy.

  The present invention according to one aspect is a distance measuring device for measuring a distance to an object, and includes a camera that images object light from the object, and the camera is an imaging lens that condenses the object light. A microlens array composed of a plurality of microlenses on which the object light that has passed through the imaging lens is incident, and an imaging device that detects the object light that has passed through the microlens array and converts it into image data. Each has a plurality of image sensors that detect object light that has passed through the microlens array and converts it into image signals, and each of the plurality of image sensors has a plurality of elements on which the object light that has passed through one microlens is incident. The pixel value of the pixel data corresponding to the image sensor on the outer edge portion that is divided into groups and that is incident on the light transmitted through the non-lens portion of the microlens array is set to a predetermined value. Based on the incident direction of the object light to the microlens determined by the position of the image sensor that detected the object light in each element group, the mask means to be replaced, and the plurality of pixel data included in the image data processed by the mask means Rearrangement means for generating a reconstructed image to be obtained when the imaging device is positioned at each of a plurality of virtual positions, and each of the plurality of reconstructed images in a region common to the plurality of reconstructed images A difference value calculating means for calculating a difference value of luminance between each pixel and a pixel in the vicinity of each pixel, and a pixel having a maximum difference value for each pixel position included in the common area. Distance calculating means for specifying a virtual position corresponding to the constituent image, converting the specified virtual position based on a parameter of the optical system of the camera, and calculating a distance;

  Preferably, the difference value calculating unit calculates uniform luminance based on pixel values of a plurality of pixels included in an area of image data corresponding to each element group, and the plurality of pixels included in each area are uniformly luminance. The image processing apparatus further includes an averaging unit that converts the pixel value into a pixel having a luminance value, and calculates a difference value based on the pixel having a uniform luminance.

  More preferably, the averaging means calculates the uniform luminance based on the pixel values of the pixels included in each region, excluding the pixels at the outer edge.

  More preferably, the averaging means calculates uniform luminance based on pixel values of pixels excluding pixels having a predetermined value among pixels included in each region.

Preferably, the predetermined value is a minimum value that can be taken by the pixel value.
Preferably, the predetermined value is a maximum value that the pixel value can take.

  Preferably, the masking unit sets the predetermined value to a minimum value that can be taken by the image signal when the average value of the pixel values included in the predetermined region of the image data is equal to or greater than a predetermined threshold, and the average value is equal to the predetermined value. When it is less than the threshold, the predetermined value is set to the maximum value that the image signal can take.

  Preferably, the mask means deletes the pixel data of the outer edge portion from the image data, and the reconstruction means rearranges the plurality of pixel data included in the image data from which the pixel data of the outer edge portion is deleted. Is generated.

  The present invention according to another aspect is a distance measuring method for measuring a distance to an object with a camera object that images object light from the object, wherein the camera collects the object light. A microlens array composed of a plurality of microlenses on which the object light that has passed through the imaging lens is incident, and an imaging device that detects the object light that has passed through the microlens array and converts it into image data. Each has a plurality of image sensors that detect object light that has passed through the microlens array and converts it into image signals, and each of the plurality of image sensors has a plurality of elements on which the object light that has passed through one microlens is incident. The pixel value of the pixel data corresponding to the image sensor at the outer edge portion that is divided into groups and that is incident on the light transmitted through the non-lens portion of the microlens array is changed to a predetermined value. And a plurality of pixel data included in the image data obtained by converting the pixel value of the pixel data corresponding to the image sensor at the outer edge to a predetermined value according to the position of the image sensor that detects the object light in each element group. Reordering based on the incident direction of the fixed object light to the microlens to generate a reconstructed image to be obtained when the imaging device is located at each of a plurality of virtual positions, and common to the plurality of reconstructed images For each of a plurality of reconstructed images, a step of calculating a luminance difference value between each pixel and a pixel in the vicinity of each pixel, and a difference value for each pixel position included in the common region Identifying a virtual position corresponding to a reconstructed image having a pixel with the largest value, converting the identified virtual position based on a parameter of the optical system of the camera, and calculating a distance. .

  According to the present invention, it is possible to generate a plurality of images in different focusing states by image processing from a single imaged data, and obtain a distance image from the data. Therefore, a camera lens focusing mechanism and a moving mechanism such as a stage are provided. It is unnecessary. In addition, by masking image information from the image sensor in the area corresponding to the non-lens part and generating a reconstructed image of the virtual position, high-precision distance detection is possible while eliminating the need for a high-precision microlens array. Can be done. Therefore, a small and low-cost highly accurate distance information measuring device can be realized.

  The present invention relates to a distance measuring device that captures an image formed by an imaging lens with an imaging element through a microlens array and acquires three-dimensional information of an object from the image data. Hereinafter, embodiments of the present invention will be described with reference to the drawings. In addition, this invention is not limited to the following embodiment.

(1. Configuration of distance measuring device)
With reference to FIG. 1, the structure of the distance measuring device according to the present invention will be described. FIG. 1 is a schematic configuration diagram of a distance measuring apparatus according to an embodiment of the present invention.

  The distance measuring device includes an image sensor camera 1 that captures object light of an object, an image capture board 2 that acquires image data from an imaging result of the image sensor camera 1, a memory 3 that stores image data, and image data. And an image processing unit 4 that calculates distance information to the object. Moreover, the display monitor 5 which displays the distance information calculated by the image processing part 4 is further provided as needed.

  The image processing unit 4 includes a reconstruction unit 4a, a luminance information calculation unit 4b, a distance information calculation unit 4c, and a mask unit 4d.

  The mask unit 4d converts a pixel value of a part of the image data of the object acquired from the image capture board 2 into a specific value. The reconstruction unit 4a rearranges the pixel data included in the image data, and generates reconstructed images at a plurality of different virtual focus positions. The luminance information calculation unit 4b calculates luminance information of each pixel or a pixel group including a plurality of pixels in the plurality of reconstructed images. The distance information calculation unit 4c calculates the distance to the object based on the luminance information and the parameters of the optical system of the image sensor camera 1.

  The image sensor camera 1 includes an imaging lens (hereinafter referred to as a main lens 11), a microlens array 12 including a plurality of microlenses, and an imaging element array 13 including a plurality of imaging elements. As the image sensor, for example, a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) sensor can be used.

  Object light from the object is incident on the main lens 11. The main lens 11 collects object light. Object light that has passed through the main lens 11 enters the microlens array 12. Each of the plurality of microlenses included in the microlens array 12 collects object light.

  Each of the plurality of image sensors included in the image sensor array 13 converts incident light into an image signal. The image data acquired by the image sensor array 13 will be described in detail later.

(2. Image sensor camera optical system)
The present embodiment is characterized by the configuration of the optical system of the image sensor camera 1. The optical system of the image sensor camera 1 will be described with reference to FIG. FIG. 2 is a diagram illustrating an optical system of the image sensor camera 1. For simplicity, FIG. 2 shows a case where the object 14 is a point light source.

  The microlens array 12 of the image sensor camera 1 is disposed at a position where the object 14 is substantially imaged by the main lens 12 when the object 14 is at the position A. The position A at this time is called a focus position, the side closer to the main lens 11 than the focus position is called a near point position, and the side farther from the main lens 11 than the focus position is called a far point position. The image pickup device array 13 is disposed almost at the focal position of the microlens array 12. The light of the light object condensed on the microlens array 12 is enlarged and incident on the imaging element array 13 through the microlens.

  Also, the parameters of the optical system of the image sensor camera 1 (such as the F number of the main lens 12 and the microlens) are adjusted so that the light that has passed through each microlens is incident on different areas on the image sensor array 13. It is assumed that

  Next, the principle of imaging and distance information detection by the image sensor camera 1 will be described with reference to FIGS.

  With reference to FIG. 3, the imaging of the target 14 will be described. FIG. 3 is a diagram illustrating a state of light collection when the object 14 is at the near point position B or the far point position C. When the object 14 is at the position B, the light from the object 14 is condensed (imaged) on the surface D by the main lens 11. Further, when it is at the position C, it is condensed (imaged) on the surface F. The surface E is a light condensing surface with respect to the in-focus position A and substantially corresponds to the position of the microlens array 12. The imaging planes F, E, and D are called virtual focal planes for C, A, and B, respectively.

  Next, the shape of the incident light 15 incident on the microlens array 12 when the object 14 is at the positions A, B, and C in FIG. 3 will be described with reference to FIG. FIG. 4 is a diagram showing the shape of the incident light 15 on the microlens array 12.

  As shown in FIG. 4, in the present embodiment, the microlens array 12 is made of circular microlenses arranged in a two-dimensional plane. Each microlens is given a number M (i, j) for identifying them. 4 (a), 4 (b), and 4 (c) respectively show the case where the object 14 is at the far point position C, the in-focus position A, and the near point position B. It is a figure which shows the shape of the incident light 15 on the micro lens array 12. FIG.

  The light from the object 14 at the in-focus position A is collected almost on the microlens M (2, 2) as shown in FIG.

  When the object 14 is located at the far point position C, the light from the object 14 is once condensed through the main lens 11, then further defocused and spread on the microlens 12. Therefore, as shown in FIG. 4A, the incident light 15 on the microlens array 12 is transmitted from the peripheral lenses M (1,2), M (2,1), M (M (2,2) of the microlens M (2,2). 2, 3) and M (3, 2).

  When the object 14 is at the near point position B, light in a defocused state before focusing is incident on the microlens 12. Therefore, as shown in FIG. 4B, the incident light 15 on the microlens array 12 is transmitted to the peripheral lenses M (1,2), M (2,1), M (M (2,2) of the microlens M (2,2). 2, 3) and M (3, 2).

  The incident light 16 on the image sensor array 13 will be described with reference to FIG. As shown in FIG. 5, the imaging element array 13 includes a plurality of element groups. Each element group is given a number T (i, j). Each element group detects object light that has passed through one of the microlenses. In other words, the light that has passed through the microlens M (i, j) enters the pixel group T (i, j). In addition, although the figure has shown the case where an element group is comprised with a 10x10 image pick-up element, the number of the elements which comprise an element group is not restricted to this.

  When the object 14 is at the in-focus position A, the light condensed on the micro lens M (2, 2) as shown in FIG. Therefore, the light incident on the image sensor array 13 spreads over the entire surface with respect to the element group T (2, 2) as shown in FIG.

  When the object 14 is located at the far point position C, the peripheral lenses M (1,2), M (2,1), M (2,3), as shown in FIG. The light that has spread to M (3, 2) also enters the image sensor array 13. As shown in FIG. 5A, the light incident on these peripheral lenses is grouped with element groups T (1,2), T (2,1), corresponding to each microlens, as shown in FIG. Incident on part of T (2,3) and T (3,2).

  FIG. 5C is a diagram illustrating the shape of light incident on the image sensor array 13 when the object 14 is located at the near point position B. FIG. Similarly to the case where the object is located at the far point position C, light is also emitted to a part of the element groups T (1,2), T (2,1), T (2,3), and T (3,2). Incident.

(3. Image reconstruction)
Next, image reconstruction on each virtual plane, which is a basic operation for calculating distance information in the present invention, will be described.

  When reconstructing an image, it is important that information on the incident direction of the object light on the microlens can be obtained from the image data acquired by the image sensor camera 1. The fact that information on the incident direction can be obtained from the image data will be described with reference to FIG. 5 again. For example, as shown in FIG. 5A, the light from the object 14 at the far-point position C is centered on the element group T (2, 2), and the surrounding element groups T (1, 2), T The incident light is incident on some fixed pixels outside (2,1), T (2,3), and T (3,2). On the other hand, as shown in FIG. 5C, the light from the object 14 at the near point position B is centered around the element group T (2, 2), and the surrounding pixel group T (1, 2). ), T (2,1), T (2,3), and T (3,2) are incident on some fixed pixels inside. As described above, information on the incident direction of the object light to the microlens installed at a position corresponding to each element group can be obtained depending on which image sensor among the image sensors included in each element group is incident. it can.

  Based on the information on the incident direction, the position of the virtual plane (virtual position), and the parameters of the optical system (focal length of each lens, distance between the main lens 11 and the microlens array 12, etc.) By rearranging the pixel data acquired by 13, it is possible to acquire image data to be acquired when the image sensor array 13 is on a virtual plane.

  A specific example of rearrangement will be described with reference to FIG. FIG. 6 is a diagram showing a reconstructed image obtained by converting data of a captured image of the object 14. For example, FIG. 6A shows the distribution of light on the virtual plane F reconstructed from the image data shown in FIG. 5A, and the light beam at the in-focus position in FIG. 5B. It becomes almost the same as the state. By collecting the outputs of the surrounding imaging elements and adding them together with the output near the center of the element group T (2, 2), the output of the element group T (2, 2) is obtained again. An image can be created.

  For the object (point image) at the near point position B, a reconstructed image as shown in FIG. 6C can be generated by adding the outputs. However, as can be seen from FIG. 5, since the spread (division state) on the image sensor array 13 is different from the far point state, different operations (one inside the surrounding pixel group) are required to obtain a reconstructed image. Part pixel outputs must be added).

  Although an example in which the object is a point image is shown here, an image can be reconstructed by a similar operation even in the case of an object having a size. However, in the case of an image having a size, since light is incident on all the microlenses, the same operation is performed on the pixel groups T (1,1) to T (3,3) corresponding to each microlens. There is a need.

  A reconstructed image in the case where the object 14 is not a point image will be described with reference to FIGS. 7 and 8. FIG. 7 is a diagram illustrating a state of light collection when the arrow-shaped object 14 is at the near point position B, and FIG. 8 is a diagram illustrating an image acquired on the virtual plane illustrated in FIG. is there.

  Since the object 14 is at the near point position B, the acquired image on the virtual plane E almost at the position of the imaging element array 13 is a blurred image as shown in FIG. However, when the virtual plane is changed to D1, D2, and D3, the respective reconstructed images are as shown in FIGS. 8B, 8C, and 8D. When changing to D3, since D3 is a virtual plane with respect to position B, a focused image of the object at position B is obtained.

  8 corresponds to an image acquired through the microlens array 12, and when the image is enlarged, as shown in FIG. 9, the pixel group for each microlens is an image as one unit. . FIG. 9 is a diagram showing an enlarged image of a portion surrounded by a frame in FIG.

  As described above, by performing image processing on imaging data obtained by one imaging, images on a plurality of virtual planes can be reconstructed.

(4. How to obtain distance information)
Next, a basic flow of a method for acquiring distance information (three-dimensional information) of an object will be described with reference to FIG. FIG. 10 is a diagram for explaining image data generated in each process of obtaining distance information.

  In acquiring distance information, first, image data of the object is acquired by the image sensor camera 1. The acquired image data is stored in the memory 3 via the image capture board 2. This image data is assumed to be f (i, j) as shown in FIG.

  Thereafter, the image processing unit 4 processes the image data f (i, j) to obtain distance information of the object. This process is broadly divided into “reconstruction step”, “difference value calculation step”, and “distance information calculation step”. Hereinafter, each processing will be described.

(1) Reconstruction Step The reconstruction unit 4a included in the image processing unit 4 rearranges the image data f (i, j) and reconstructs on N virtual planes as shown in FIG. 10B. Image data gn (i, j) is generated. Here, n (n = 1 to N) indicates the order of the virtual surfaces. This rearrangement is expressed as gn (i, j) = An (f (i, j)) using the focal plane conversion coefficient An (n = 1 to N). Here, An is determined based on the incident direction of the object light on each microlens determined by the position of the pixel that detected the object light in each pixel group, the virtual position, and the parameters of the optical system.

  Note that, for example, the reconstruction unit 4a is not particularly limited, but in the present embodiment, the reconstruction unit 4a acquires reconstruction image data for a plurality of virtual positions arranged at predetermined intervals. In addition, when the in-focus virtual position is calculated by the method described below, the reconstruction unit 4a further narrows the interval between the virtual positions near the in-focus virtual position so that the virtual position is narrow. On the other hand, a reconstructed image may be acquired. According to this configuration, the accuracy of distance detection can be increased.

In this step, one piece of image data f (i, j) is converted into N pieces of reconstructed image data as shown in FIG. 10B, that is, g ₀₁ (i, j) = An (f (i, j j)), g ₀₂ (i, j) = An (f (i, j)),..., gn (i, j) = An (f (i, j)),..., gN (i, j) = An (f (i, j)) is obtained.

(2) Difference Value Calculation Step The luminance information calculation unit 4b calculates the difference value of the luminance of adjacent pixels from the reconstructed image data gn (i, j). The calculation of the difference value is performed according to the following procedure.

  First, the luminance information calculation unit 4b converts image data hn () obtained by converting each pixel group as shown in FIG. 13C into pixels having uniform luminance for each reconstructed image data gn (i, j). i ′, j ′). Here, the uniform luminance may be a luminance corresponding to the luminance of the pixels included in each pixel group. For example, the luminance information calculation unit 4b calculates the average value of the luminance of the pixels included in each pixel group or the total luminance as the uniform luminance.

  In this embodiment, it is assumed that a pixel group composed of m × m pixels is converted (averaged) into one pixel having uniform luminance. This conversion is expressed as hn (i ′, j ′) = Bn (gn (i, j)) where Bn is an averaging coefficient.

  A specific example of the averaging process will be described with reference to FIG. FIG. 11 is a diagram showing the luminance distribution of the pixel group when the averaging process is performed on the image shown in FIG. FIGS. 11A to 11D are obtained by averaging the luminance distributions of FIGS. 9A to 9D in the pixel group.

With this step, the amount of data to be processed can be reduced to 1 / m ² . Therefore, the image processing speed of the subsequent processing is greatly improved. However, this process is not essential.

Next, the luminance information calculation unit 4c calculates a luminance difference value between pixels for each reconstructed image. When the above-described averaging process is performed, the luminance information calculation unit 4c calculates a luminance difference value between each pixel group and the adjacent pixel group in each image data hn (i ′, j ′). Difference value image data kn (i ′, j ′) as shown in FIG. The relationship between hn and kn is expressed as kn (i ′, j ′) = Cn (h (i ′, j ′)), where Cn is the difference value conversion coefficient. Here, the difference value conversion coefficient Cn is, for example, in the x direction with respect to the pixel group of (i ′, j ′) when using a generally used edge detection method called the Roberts method. Differential value dx = h (i ′, j ′) − h (i ′ + 1, j ′ + 1), y-direction differential value dy = h (i ′ + 1, j ′) − h (i ′, j ′ + 1) And (dx ² + dy ² ) ^1/2 . However, the difference value conversion coefficient is not limited to this, and various filter windows such as a simple differentiation method, a Sobel method, and a Laplacian method using a secondary differentiation can be set. As shown in FIG. 10D, N difference value images are created.

  When the above-described averaging process is not performed, the luminance information calculation unit 4b simply calculates a difference in luminance value between adjacent pixels for each pixel of each reconstructed image, and calculates a difference value image. Get the data.

(3) Distance information calculation step The distance information calculation unit 4c calculates the distance information of the object in the following procedure based on the data obtained in the difference value calculation step.

  First, the distance information calculation unit 4c detects the maximum value of the difference value intensity at each pixel position p = (i ′, j ′) common to a plurality of reconstructed images based on the difference value image data, and the maximum An n value (hereinafter referred to as a representative value) from which a value is detected is obtained, and representative value image data s (i ′, j ′) as shown in FIG. That is, the value of n that maximizes kn (i ′, j ′) to kN (i ′, j ′) at each p = (i ′, j ′) is set as s (i ′, j ′). As shown in FIG. 9 or FIG. 11, the luminance change around the edge is the largest in the image at the in-focus position. Therefore, it can be determined that the position of the virtual surface having the maximum difference value indicates the in-focus state. Using this fact, the distance to the object can be calculated.

  Alternatively, the representative value may be obtained from an approximate curve of the difference value distribution. This finding method will be described with reference to FIG. FIG. 12 is a diagram for explaining a method for obtaining the representative value from the approximate curve of the difference value distribution. FIG. 12A shows a reconstructed image of two objects arranged at different distances. FIG. 12B is a plot of the difference value intensities at the pixel positions (14A, 15B) near the edge of each object shown in FIG. 12A with respect to the position (n value) of the virtual plane. . For each object, an approximate curve of difference value intensities plotted with respect to the n value is obtained at the pixel position (i ′, j ′), and the n value of the peak of the approximate curve (n1, n2 in FIG. 12B). ) As a representative value s (i ′, j ′). This representative value indicates the position of the virtual plane in focus on the image of the object.

  Finally, the distance information calculation unit 4c obtains the representative value image s (i ′, j ′) from which the maximum difference value intensity is obtained at each pixel position from the distance measurement device using the magnification data of a known optical system. The absolute distance d (i ′, j ′) to the object is converted into final distance image data d (i ′, j ′) as shown in FIG.

  The above algorithm is for performing luminance information calculation processing and distance information calculation processing for all pixels and obtaining distance data for all pixels, but if it is not necessary to obtain distance data for all pixels, the algorithm processing area May be limited. For example, a common processing area may be set for a plurality of reconstructed images, and the luminance information calculation process and the distance information calculation process may be performed in the set processing area. By limiting the processing area, processing time and calculation load can be reduced.

  Further, by performing threshold processing on the difference value intensity image kn (i ′, j ′), the edge region in the image is extracted, and the calculation of the representative value and the calculation of the distance information are limited to the extracted edge region. May be performed. Thus, according to the structure which calculates | requires distance only about an edge area | region, processing time and a calculation load can be reduced significantly.

  With the distance information calculation algorithm, a plurality of images in different focusing states (virtual planes) can be generated from a single image data by image processing, and a distance image of the object can be obtained from the data.

(5. Mask processing)
Basically, a distance image can be obtained from a captured image by the method described above. However, in the above-described method, the quality of the microlens array 12 affects the quality of the reconstructed image. A specific example will be described with reference to FIGS. FIG. 13 is a diagram for explaining a subject, and FIG. 14 is a diagram showing a captured image and a reconstructed image of the subject shown in FIG. The captured image shown in FIG. 14 is a photograph of the subject shown in FIG. 13 taken in monochrome quality.

  As shown in FIG. 13, the subject is an array of cards (SHA) on which characters (SHA) are drawn at intervals of 15 mm. The position of each character is shifted in the radial direction of the main lens 11 so that all characters can be seen when photographed from the front. The frontmost card is placed at a position 100 mm from the main lens 11, and the position of the second card is taken as the reference focal plane. In other words, the photograph is focused on the surface on which the second card is located.

  FIG. 14A shows a captured image of the subject under the above conditions. Moreover, the enlarged view of the part enclosed with the square of Fig.14 (a) is shown in FIG.14 (d). FIG. 14B shows a reconstructed image on the virtual plane at a position moved by 15 mm from the reference focal plane to the main lens 11 side with respect to the captured image (FIG. 14A). Further, FIG. 14C shows an image obtained by subjecting the reconstructed image of FIG. 14B to pixel group averaging processing.

  In this case, since the virtual plane is set at a position moved 15 mm from the reference focal plane to the main lens 11 side, an image focused on the card in the front row of the subject in FIG. 13 should be obtained. In FIG. 14 (c), the outline of the characters on the front row of cards (the leftmost character in FIG. 14 (c)) is not so clear.

  If the image quality here is poor, then the accuracy of edge detection, that is, maximum difference value intensity detection, which is a subsequent process, is reduced. This is because it is difficult to detect the edge intensity of an image whose outline is not clear, and it is difficult to find the maximum difference value intensity. This means that the distance detection accuracy decreases when the image quality deteriorates.

  The deterioration of the image quality as shown in FIG. 14 is largely due to the non-lens portion of the microlens array 12. In general, in a microlens array, the lens shape is often not maintained between microlenses due to manufacturing problems. Further, the peripheral portion of the microlens does not necessarily function as a lens because processing accuracy is difficult to obtain. As described above, the non-lens portion that exists between the microlenses or around the microlens and does not function as a lens causes deterioration in image quality.

  This will be described with reference to FIG. 14D which is an enlarged view of the captured image. A portion surrounded by a square in FIG. 14D is a portion corresponding to one microlens. There is a flat gap between the microlenses of the microlens array used for imaging in FIG. Therefore, as can be seen from FIG. 14D, the actual microlens is inside the portion enclosed by the square.

  In the calculation for generating the reconstructed image, the pixels between and around the microlenses are considered to be noise. Since an image including noise is reconstructed and averaged, an image whose edges are not sufficiently clear as shown in FIG.

  In order to prevent this image deterioration, it is conceivable to use a highly accurate lens with few non-lens portions. However, such lenses are generally expensive.

  In addition, it is conceivable to perform the correction calculation of the non-lens part in consideration of the optical characteristics of the non-lens part, but this method requires grasping the detailed optical characteristics of the non-lens part, Reflecting this, there is a problem that calculation processing becomes complicated and processing time becomes long.

  In the present invention, in order to solve this problem, mask processing is performed between the microlenses and the microlens peripheral portion of the captured image by the mask unit 4d.

  As the mask processing, for example, the pixel value of the mask target portion is set to a specific value. The specific value to be set is, for example, the minimum value in the range that the pixel value can take. Alternatively, the maximum value of the range that the pixel value can take is set.

  In the present embodiment, it is assumed that mask processing is performed on the mask pixels indicated by the hatched portion in FIG. 15 as pixels corresponding to the non-lens portion of the captured image in FIG. Also, the value after mask processing is set to pixel value = 0 (black).

  An image subjected to this processing is shown in FIG. Moreover, the enlarged view of the part enclosed with the square of Fig.16 (a) is shown in FIG.16 (d). It can be seen from FIG. 16D that the portions between the microlenses and the peripheral portions of the microlenses are masked and blackened.

  FIG. 16B shows a reconstructed image of the image of FIG. 16A on the virtual plane moved 15 mm from the reference focal plane to the main lens side. Further, FIG. 16C shows an image obtained by performing averaging processing on the reconstructed image of FIG.

  Compared to FIG. 14C, the outline of the character (leftmost character) drawn on the card in the front row is clearer in FIG. 15C. In FIG. 15C, it can be seen that the character written on the card in the front row among the three cards is most focused. Thus, by performing mask processing and generating a reconstructed image, it is possible to eliminate the influence of noise between the microlenses and the peripheral portion of the microlenses.

  Note that when averaging is performed including the pixels of the mask portion, the entire image is biased to the pixel values of the mask portion. In the case of this embodiment, the entire image becomes dark. Therefore, it is desirable to exclude the pixels in the mask portion from the objects of averaging.

  For this purpose, pixels satisfying a certain condition may be excluded from the averaging calculation. A condition including the pixel value of the mask portion is set to distinguish the mask pixel from other pixels.

  For example, a mask pixel is distinguished from other pixels using a threshold value. That is, when the converted value of the mask pixel is the minimum value, a pixel having a pixel value equal to or less than a predetermined threshold is regarded as a mask pixel. On the contrary, when the converted value of the mask pixel is the maximum value, a pixel having a pixel value equal to or greater than a predetermined threshold is regarded as a mask pixel. Alternatively, pixels having the pixel value of the mask pixel after conversion may be excluded.

  When extracting mask pixels using pixel values in this way, if the entire image is bright or the object image to be noticed is bright, the pixel value of the mask target portion is set to the minimum value (black), conversely, When the entire image is dark or the object image to be noticed is dark, it is preferable to set the pixel value of the mask target portion to the maximum value (white) because the distinction between necessary pixels and mask pixels is made clear.

  Furthermore, the converted value of the mask pixel may be automatically set depending on the image. For example, whether the entire image or the object image of interest is bright or dark is determined by comparing the average value of the pixel values of the entire image or a part of the image with a predetermined threshold, and the average value is equal to or greater than the threshold. In this case, if the pixel value after mask processing is the minimum value and the average value is less than the threshold value, the pixel value after mask processing is set to the maximum value, so that the distinction between the mask pixel and other pixels can be made clearer.

  As another method for removing the pixels of the mask portion from the averaging target, the pixels included in the mask target portion may be excluded from the averaging target. In this method, since the pixel determination based on the threshold can be omitted in the averaging process, the processing speed can be increased.

  In particular, the mask unit 4d may delete the pixel data of the mask target portion from the image data. When the mask portion is as shown in FIG. 15, the image data after deletion of the target portion has a lattice shape. The reconstruction unit 4a rearranges and averages the pixel data for this image data. By deleting the mask portion data from the image data in this way, the amount of image data is reduced.

(6. Process flow)
A flow of processing in distance measurement according to the present embodiment will be described with reference to FIG. FIG. 17 is a flowchart showing a flow of distance measurement processing according to the present embodiment.

  First, the image processing unit 4 acquires image data in step S101. This image data is obtained by photographing the object once by the image sensor 1.

  Next, in step S102, the image processing unit 4 performs mask processing on the image data. That is, the pixel data of the mask target portion corresponding to the non-lens portion is converted into a predetermined value.

  In step S103, the image processing unit 4 determines a processing area to be subjected to distance measurement from the image data. The processing area may be performed by edge determination or may be set by the user. Further, this step is omitted when distance measurement is performed for the entire area of the image data.

  Subsequently, in step S104, the image processing device 4 rearranges the image data in the selected region, and creates reconstructed images for a plurality of virtual positions.

  In step S105, the image processing unit 4 averages each pixel group. That is, each pixel group is converted into a pixel having uniform luminance. At this time, the pixels of the mask target portion may be excluded from the averaging target.

  In step S106, the image processing unit 4 creates difference value image data from the reconstructed image data.

  Further, in step S107, the image processing unit 4 creates representative value data from the difference value image data.

  In step S108, the image processing unit 4 converts the representative value data into the distance data of the target object based on the magnification of the optical system, and creates the distance data.

(7. Summary)
As described above, according to the present invention, a plurality of images in different focusing states (virtual surfaces) are generated from a single image data by image processing, and a distance image of the object is obtained from the data. Can do. It is also possible to obtain the tilt and posture from the positional relationship before and after in the depth direction and the distance information of different edges of the object.

  A conventional plenoptic camera as described in Non-Patent Document 1 can reconstruct an image focused on a set distance, but cannot obtain distance information of an object of the image. On the other hand, distance information can be calculated in the present invention.

  Further, as in Patent Document 1, after autofocusing is performed by changing the height of an object on a stage or the like, a plurality of images are acquired, and height information is obtained based on the luminance information. In contrast, the method of the present invention eliminates the need for a camera lens focusing mechanism and a moving mechanism such as a stage, and is therefore small and low-cost.

  In addition, since distance data can be calculated from image data acquired by a single imaging operation, the time required to move the camera or object multiple times, the shooting time, or the matching operation of each image associated with the movement is not required, making significant measurements. Time can also be shortened. The operation for generating a plurality of reconstructed images from one acquired image can be combined in parallel without the need for serial processing, so that a plurality of shooting times can be omitted, which is advantageous for high-speed measurement.

  Furthermore, since this optical system is configured to incorporate a microlens array in the immediate vicinity of the image sensor, it is almost the same size as a conventional image sensor camera, so it is widely used in visual sensors such as robots for FA applications. Because it fits in the same outer diameter size as the image sensor camera, compatibility (replacement) with conventional cameras is high.

  In addition, it can accurately detect distances without using a high-precision microlens array by masking image information between non-lens parts around microlenses and around microlenses to generate reconstructed images on virtual surfaces. I can do it.

  The embodiments disclosed herein are illustrative and non-restrictive in every respect. The scope of the present invention is shown not by the above-described embodiment but by the scope of claims, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

It is a schematic block diagram of the distance information detection apparatus which concerns on embodiment of this invention. 1 is a diagram illustrating an optical system of an image sensor camera 1. FIG. It is a figure which shows the mode of condensing in case the target object 14 exists in the near point position B or the far point position C. It is a figure which shows the shape of the incident light 15 on the micro lens array 12. FIG. It is a figure which shows the shape of the incident light 16 on the image pick-up element array. It is a figure which shows the reconstruction image obtained by converting the data of the picked-up image of the target object. It is a figure which shows the mode of condensing in case the arrow-shaped target object 14 exists in the near point side B. FIG. It is a figure which shows the image acquired by the virtual surface shown in FIG. It is a figure which shows the image which expanded the part enclosed with the frame in FIG. It is a figure for demonstrating the image data produced | generated at each process of acquisition of distance information. It is a figure which shows the luminance distribution of a pixel group at the time of performing the averaging process with respect to the image shown in FIG. It is a figure for demonstrating the method of calculating | requiring a representative value from the approximated curve of difference value distribution. It is a figure explaining a to-be-photographed object. FIG. 14 is a diagram illustrating a captured image and a reconstructed image of the subject illustrated in FIG. 13. It is a figure explaining a mask part. It is a figure which shows the reconstructed image of a mask image and a mask image. It is a flowchart which shows the flow of the process of the distance measurement which concerns on this Embodiment. It is a figure which shows the structure of the height information measuring apparatus using the principle of a confocal microscope. It is a figure explaining the detection algorithm of the sample height information by the height information measuring device using the principle of a confocal microscope.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Image sensor camera, 2 Image capture board, 3 Memory, 4 Image processing part, 4a Reconstruction part, 4b Luminance information calculation part, 4c Distance information calculation part, 5 Display monitor, 11 Imaging lens, 12 Micro lens array, 13 Imaging Element array, 14 Object, 15 Incident light on microlens array 12, 16 Incident light on imaging element array 13.

Claims (9)

A distance measuring device for measuring a distance to an object,
Comprising a camera for imaging object light from the object;
The camera
An imaging lens for condensing the object light;
A microlens array comprising a plurality of microlenses on which the object light having passed through the imaging lens is incident;
An imaging device that detects the object light that has passed through the microlens array and converts it into image data;
The imaging device includes a plurality of imaging elements that each detect the object light that has passed through the microlens array and convert it into an image signal.
The plurality of image sensors are divided into a plurality of element groups on which the object light transmitted through one micro lens is incident,
Mask means for converting a pixel value of pixel data corresponding to the imaging element on an outer edge portion where light transmitted through a non-lens portion of the microlens array is incident into a predetermined value;
The plurality of pieces of pixel data included in the image data processed by the masking means are incident in the incident direction of the object light on the microlens determined by the position of the imaging element that detected the object light in each of the element groups. Reconstructing means for rearranging and generating a reconstructed image to be obtained when the imaging device is located at each of a plurality of virtual positions;
Difference value calculation means for calculating a difference value of luminance between each pixel and a pixel in the vicinity of each pixel for each of the plurality of reconstructed images in a region common to the plurality of reconstructed images;
For each pixel position included in the common area, the virtual position corresponding to the reconstructed image having the pixel having the maximum difference value is specified, and the specification is performed based on a parameter of the optical system of the camera A distance measuring device comprising: a distance calculating unit that converts the calculated virtual position and calculates the distance. The difference value calculating means includes:
Uniform luminance is calculated based on pixel values of the plurality of pixels included in the region of the image data corresponding to each element group, and the plurality of pixels included in each of the regions are converted into pixels having the uniform luminance. Further comprising an averaging means for converting,
The distance measuring device according to claim 1, wherein the difference value is calculated based on the pixels having the uniform luminance.   The distance measurement according to claim 2, wherein the averaging unit calculates the uniform luminance based on a pixel value of the pixel excluding the pixel at the outer edge portion of the pixels included in each of the regions. apparatus.   The distance according to claim 3, wherein the averaging means calculates the uniform luminance based on a pixel value of the pixel excluding a pixel having the predetermined value among the pixels included in each of the regions. measuring device.   The distance measuring device according to claim 1, wherein the predetermined value is a minimum value that the pixel value can take.   The distance measuring apparatus according to claim 1, wherein the predetermined value is a maximum value that the pixel value can take.   The mask means sets the predetermined value to a minimum value that can be taken by the image signal when an average value of pixel values included in a predetermined region of the image data is equal to or greater than a predetermined threshold, and the average value is The distance measuring device according to any one of claims 1 to 4, wherein when the value is less than the predetermined threshold, the predetermined value is set to a maximum value that the image signal can take. The mask means deletes the pixel data of the outer edge from the image data;
The distance measuring device according to claim 1, wherein the reconstruction unit rearranges a plurality of the pixel data included in the image data from which the pixel data of the outer edge portion is deleted, and generates the reconstructed image. . A distance measurement method for measuring a distance to an object with a camera object that images object light from the object,
The camera
An imaging lens for condensing the object light;
A microlens array comprising a plurality of microlenses on which the object light having passed through the imaging lens is incident;
An imaging device that detects the object light that has passed through the microlens array and converts it into image data;
The imaging device includes a plurality of imaging elements that each detect the object light that has passed through the microlens array and convert it into an image signal.
The plurality of image sensors are divided into a plurality of element groups on which the object light transmitted through one micro lens is incident,
Converting a pixel value of pixel data corresponding to the imaging element at an outer edge portion where light transmitted through a non-lens portion of the microlens array is incident into a predetermined value;
The imaging in which the object light in each of the element groups is detected from a plurality of the pixel data included in the image data obtained by converting pixel values of pixel data corresponding to the imaging element at the outer edge to the predetermined value Rearranging based on the incident direction of the object light determined by the position of the element to the microlens, and generating a reconstructed image to be obtained when the imaging device is located at each of a plurality of virtual positions;
Calculating a luminance difference value between each pixel and a pixel in the vicinity of each pixel for each of the plurality of reconstructed images in a region common to the plurality of reconstructed images;
For each pixel position included in the common area, the virtual position corresponding to the reconstructed image having the pixel having the maximum difference value is specified, and the specification is performed based on a parameter of the optical system of the camera Converting the virtual position thus calculated and calculating the distance.