JP2009250685A - Distance measuring device and distance measuring method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a small-sized information measuring device capable of measuring distance information to an object in a short time. <P>SOLUTION: This device is equipped with an image sensor camera 1, an image capture board 2, a memory 3 for holding image data, an image processing part 4 for calculating the distance information to the object from the image data held in the memory 3, and a display monitor 5 for displaying the distance information calculated by the image processing part 4. The image sensor camera 1 includes a main lens 11, a shutter 12, a microlens array 13, and an image cell array 14. The shutter 12 has a plurality of opening patterns, and in each pattern, light passing each opening part of the shutter 12 enters each different domain of the image cell array 14 respectively. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a technique for measuring a distance to an object based on image data of the object acquired by an image sensor.

  In recent years, in order to construct a system that automates production lines using industrial robots and various FA (Factory Automation) devices, or to make intelligent robots intelligent, there has been a visual sensor for acquiring distance information of objects. It has become important. In particular, in a system using a robot arm, an image sensor camera is often used as a visual sensor for measuring the position, posture, shape, and the like of a workpiece.

  Also, in the manufacture of mounting boards for semiconductor devices and circuit components, there is an increasing need to measure height information of minute devices arranged in two dimensions such as solder bumps and gold bumps in order to control quality. Yes.

  In order to meet such needs, a conventional method for detecting height and posture information from an image of a two-dimensional image sensor camera, and a stereo three-dimensional information measurement system using a plurality of image sensor cameras. Etc. are used.

  As a method for measuring the height information of a relatively small object, a height measurement method using the principle of a confocal microscope has been proposed. For example, Japanese Patent Laid-Open No. 2003-75119 (Patent Document 1) describes a height information measuring device using the principle of a confocal microscope.

  This height information measuring device will be described with reference to FIGS. FIG. 15 is a diagram showing a configuration of a height information measuring device using the principle of a confocal microscope. FIG. 16 is a diagram for explaining a detection algorithm of the height information of the sample by the height information measuring device using the principle of the confocal microscope. As shown in FIG. 15, the confocal optical system 101 including the light source 103, the objective lens 104, and the two-dimensional image camera 102 moves the stage 105 on which the sample 106 is placed in the height direction while moving the stage 105 in the height direction. It has the structure which each images the confocal image of a horizontal surface in several different height positions (FIG. 16 (a) (b)). This height information measuring device compares the luminance information for each pixel of a plurality of confocal images to obtain the maximum luminance, and performs particle analysis using confocal pixel data including the pixel having the maximum luminance to extract a specific region. To do. Then, representative values of luminance and height in the extracted region are calculated to obtain sample height information (FIG. 16C).

In addition, in a camera system using an image sensor, a plenoptic camera technology that synthesizes acquired image data by digital processing and can move the focus to the front or back as desired later is “Single Lens Stereo with a”. Plenoptic Camera ", IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 14, NO. 2, FEBRUARY 1992 (Non-patent Document 1) and “Light Field Photographic with a Hand-Held Plenoptic Camera”, Stanford Tech Report CTSR 2005-02 (Non-patent Document 2). The plenoptic cameras described in these documents have an imaging lens similar to a normal camera lens, but with a normal camera, the microlens array is precisely placed in the image plane and more than the microlens array. The difference is that an image sensor array (imaging cell array) having a plurality of imaging pixels is placed immediately behind the microlens array. Although the number of pixels in the final image is determined by the number of lenses in the microlens array, the direction and intensity of light incident on the microlens can be recorded by a large number of sensor pixels assigned to a single microlens. Using that data, an image focused on a predetermined distance can be reconstructed.
JP 2003-75119 A “Single Lens Stereo with a Plenoptic Camera”, IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 14, NO. 2, FEBRUARY 1992 “Light Field Photography with a Hand-held Plenoptic Camera”, Stanford Tech Report CTSR 2005-02

  There is a problem that the depth distance and height information cannot be directly detected by the method of detecting height and posture information in a pseudo manner from the image of the two-dimensional image sensor camera. Further, distance measurement using a stereo three-dimensional information measurement system using a plurality of image sensor cameras has a problem that a plurality of cameras and complicated image processing are required.

  In addition, when acquiring a plurality of confocal images using the principle of the confocal microscope described in Patent Document 1, an operation of scanning in the surface direction of the stage and moving in the height direction is necessary. There is a problem that the moving mechanism system including the focus mechanism becomes large. In addition, there is a problem that it takes a long time to change the height of the sample and acquire two-dimensional image data of the sample a plurality of times due to movement of the stage, stationary, auto-focusing of the objective lens, and the like.

  In the plenoptic camera technology described in Non-Patent Documents 1 and 2, for example, when an operator inputs a focus position (distance) to acquired image data, an image focused on that position is converted into data. However, it is not possible to calculate the distance information of the object by using it.

  The present invention has been made paying attention to such a problem, and an object of the present invention is to provide a small information measuring apparatus capable of measuring distance information of an object in a short time.

  The present invention according to one aspect is a distance measuring device for measuring a distance to an object, and includes a camera having an image sensor, and each of the image sensors uses object light from the object as an image signal. The camera includes a plurality of imaging cells to be converted, and the camera includes an imaging lens that collects the object light, a condensing array that includes a plurality of condensers on which the object light that has passed through the imaging lens is incident, and light that is incident on the imaging element. And a plurality of light-shielding portions that shield light transmitted through the plurality of concentrators and a plurality of openings that respectively transmit light transmitted through the plurality of concentrators. The light passing through each condensing part passes through the shutter in any one of the arrangement patterns, and the object light transmitted through each opening in each arrangement pattern Different , Each incident on an area larger than the size of the condenser, and the image signal obtained by the camera in each arrangement pattern is applied to the condenser of the object light determined by the position of the imaging cell that detected the object light in each area. Reconstructing means for rearranging based on the incident direction, combining the rearranged image signals, and generating a reconstructed image to be obtained when the image sensor is located at each of a plurality of virtual positions; It further comprises distance calculation means for specifying a virtual position where the amount of incident light is maximum, converting the specified virtual position based on the parameters of the optical system of the camera, and calculating the distance.

Preferably, the outer peripheries of the respective regions are in contact with each other.
Preferably, in each arrangement pattern, the plurality of light shielding portions and the plurality of openings are arranged in a checkered pattern.

  Preferably, in each arrangement pattern, the plurality of light shielding portions and the plurality of openings corresponding to the adjacent 2 × 2 concentrators emit light from a predetermined one of the 2 × 2 concentrators. It arrange | positions so that it may permeate | transmit.

Preferably, the shutter is a liquid crystal shutter.
Preferably, the shutter is a light shielding mask having a predetermined opening and movable in accordance with a plurality of patterns.

  The present invention according to another aspect is a distance measuring method for measuring a distance to an object using a camera having an image sensor, each of which captures object light from the object as an image signal. The camera includes an imaging lens that collects the object light, a condensing array that includes a plurality of condensers on which the object light that has passed through the imaging lens enters, and an image sensor. A shutter for controlling light, and the shutter is provided with a plurality of light shielding portions for shielding light transmitted through the plurality of collectors and a plurality of openings for passing the light transmitted through the plurality of collectors, respectively. The light that has a plurality of arrangement patterns and passes through each condensing unit passes through the shutter in any one arrangement pattern, and the object light that passes through each opening in each arrangement pattern is on the image sensor. Different from each other , Each of which is incident on an area larger than the size of the condenser, and obtaining an image signal by the camera in each arrangement pattern, and imaging the image signal obtained by the camera in each arrangement pattern by detecting object light in each area Reordering based on the incident direction of the object light determined by the position of the cell to the condenser, combining the rearranged image signals, and re-acquisition to be obtained when the image sensor is located at each of a plurality of virtual positions. A step of generating a configuration image; and a step of specifying a virtual position where the amount of light incident on each region is maximum, converting the specified virtual position based on a parameter of an optical system of the camera, and calculating a distance.

  According to the present invention, a plurality of pieces of image data in different focusing states can be generated by image processing from imaging data of object light that has passed through a plurality of patterns of shutters, and distance information to an object can be obtained from the image data. Accordingly, a camera lens focusing mechanism and a moving mechanism such as a stage are not required, and a small distance information measuring apparatus can be realized. In addition, the time required for moving the camera and the time required for matching each image associated with the movement are not required, and distance measurement can be performed in a short time.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In addition, this invention is not limited to the following embodiment.

[First Embodiment]
(1. Configuration)
The configuration of the distance measuring apparatus according to the first embodiment will be described with reference to FIG. FIG. 1 is a schematic configuration diagram of a distance measuring apparatus according to an embodiment of the present invention.

  The distance measuring device includes an image sensor camera 1, an image capture board 2, a memory 3 that stores image data, an image processing unit 4 that calculates distance information from the image data stored in the memory 3 to an object, A display monitor 5 for displaying the distance information calculated by the image processing unit 4.

  The image sensor camera 1 acquires image data of an object. The image sensor camera 1 includes an imaging lens 11 (hereinafter referred to as a main lens), a shutter 12, a microlens array 13 including a plurality of microlenses, and an imaging cell array 14.

  Object light from the object is incident on the main lens 11. The main lens 11 condenses object light from the incident object.

  The shutter 12 is installed in front of the microlens array 13 (on the main lens 11 side), and controls the amount of light transmitted through the microlens. Here, the shutter 12 is installed in front of the microlens array 13, but may be installed behind the microlens array 13 (on the imaging cell array 14 side). That is, it is only necessary to be disposed in the vicinity of the microlens array 13 so that the amount of light transmitted through each microlens can be controlled. As will be described in detail later, the shutter 12 controls the amount of light with a plurality of opening patterns. In the present embodiment, a liquid crystal shutter is used as the shutter 12. However, the shutter 12 is not limited to a liquid crystal shutter. For example, a light-shielding mask having a predetermined opening pattern and movable in accordance with a plurality of opening patterns can be used as the shutter 12.

  Object light that has passed through the main lens 11 and the shutter 12 enters the microlens array 13. Each of the plurality of microlenses included in the microlens array 13 condenses object light. Note that the microlens array 13 is an example of a light collection array including a plurality of light collectors. For example, instead of the microlens array 13, a pinhole array composed of a plurality of pinholes may be used.

  The imaging cell array 14 acquires image data of object light that has passed through the shutter 12 and the microlens array 13. The imaging cell array 14 has a plurality of imaging cells. The imaging cell of the imaging cell array 14 and the image data acquired by the imaging cell array 14 will be described in detail later. As the imaging cell, for example, a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) sensor can be used.

  The image capture board 2 captures the image data acquired by the imaging cell array 14 and converts it into a form that allows subsequent processing. The memory 3 stores the image data transferred from the image capture board 2.

  The image processing unit 4 reads image data from the memory 3 and performs a process of calculating a distance to the object based on the read image data. The image processing unit 4 rearranges the pixel data included in the image data to generate a reconstructed unit 4a that generates a reconstructed image to be obtained when the imaging cell array 14 is located at each of a plurality of different virtual positions. It comprises a distance information calculation unit 4b that calculates the distance to the object based on the configuration image. Details of the processing performed by the image processing unit 4 will be described later.

(2. Features of image sensor camera)
In the present invention, there is a feature in the configuration of the optical system of the image sensor camera 1, and the feature will be described with reference to FIG. FIG. 2 is a diagram illustrating an optical system of the image sensor camera 1. For simplicity, FIG. 2 shows a case where the object 15 is a point light source.

  The microlens array 13 of the image sensor camera 1 is arranged at a position where the object 15 is substantially imaged by the main lens 12 when the object 15 is at the position A. The position A at this time is called a focal point position, the side closer to the main lens 11 than the focal point position is called a near point position, and the side farther from the main lens 11 than the focal point position is called a far point position. The imaging cell array 14 is disposed almost at the focal position of the microlens array 13. The light of the object enters the imaging cell array 14 through the microlens. In addition, it is assumed that the optical system of the image sensor camera 1 is adjusted so that light that has passed through each microlens enters the imaging cell array 14 without overlapping each other.

  Next, an image captured by the image sensor camera 1 will be described with reference to FIGS.

  The imaging of the object 15 will be described with reference to FIG. FIG. 3 is a diagram showing a state of light collection when the object 15 is at the near point position B or the far point position C. When the object 15 is at the position B, the light from the object 15 is condensed (imaged) on the surface D by the main lens 11. Further, when it is at the position C, it is condensed (imaged) on the surface F. The surface E is a light condensing surface with respect to the in-focus position A and substantially corresponds to the position of the microlens array 13. The planes F, E, and D are called virtual focal planes for C, A, and B, respectively.

  Next, what kind of image is captured by the imaging cell array 14 when the object 15 is at the positions A, B, and C in FIG. 3 will be described with reference to FIGS. 4 and 5. FIG. 4 is a diagram showing the shape of the incident light 15 on the microlens array 13, and FIG. 5 is a diagram showing the shape of the incident light 16 on the imaging cell array 14.

  First, the shape of the incident light 15 incident on the microlens array 13 when the object 15 is at positions A, B, and C in FIG. 3 will be described with reference to FIG. As shown in FIG. 4, in this embodiment, the microlens array 13 is composed of circular microlenses arranged in a two-dimensional plane. Each microlens is given a number M (i, j) for identifying them.

  In FIG. 4, the opening pattern of the shutter 12 is also shown. In the present embodiment, the shutter 12 includes the first pattern shown in FIGS. 4A, 4B, and 4C, and FIGS. 4D, 4E, and 4C. It has the 2nd pattern shown in (f). In FIG. 4, the white portion represents the opening, and the shaded portion represents the light shielding portion. As shown in FIG. 4, in the first pattern and the second pattern, the openings and the light shielding portions are arranged in a checkered pattern. In other words, the opening and the light shielding part of each pattern are arranged so as to be adjacent to each other. In the first pattern and the second pattern, the light shielding part and the opening part are inverted.

  4 (a) (d), 4 (b) (e), 4 (c), and (f), respectively, when the object 15 is at the far point position C and at the in-focus position A, respectively. FIG. 5 is a diagram showing the shape of incident light 15 on the microlens array 13 when it is at the near point position B.

  The light from the object 15 at the in-focus position A is condensed almost on the microlens M (2, 2) as shown in FIGS.

  When the object 15 is located at the far point position C, the light from the object 15 is once condensed through the main lens 11, then further defocused and spreads on the microlens 12. Therefore, as shown in FIGS. 4A and 4D, the incident light 15 on the microlens array 13 is transmitted from the peripheral lenses M (2,3) and M (3,2) of the microlens M (3,3). , M (3,4), M (4,3).

  When the object 15 is at the near point position B, light in a defocused state before focusing is incident on the microlens 12. Therefore, as shown in FIGS. 4C and 4F, the incident light 15 on the microlens array 13 is transmitted to the peripheral lenses M (2,3) and M (3,2) of the microlens M (3,3). , M (3,4), M (4,3).

  The incident light 16 on the imaging cell array 14 will be described with reference to FIG. As shown in FIG. 5, the imaging cell array 14 includes a plurality of imaging cells. In FIG. 5, the imaging cell array 14 is shown divided into areas with numbers T (i, j) corresponding to the microlens numbers. In addition, although the figure has shown the case where each area is comprised by the imaging cell of 10x10, the number of the cells of the imaging cell array 14 is not restricted to this.

  When the object 15 is at the in-focus position A, the light condensed on the microlenses M (3, 3) as shown in FIGS. Therefore, when the opening position of the shutter 12 is the first pattern, the number of light incident on the imaging cell array 14 is a number centered on the area T (3, 3) of the imaging cell array 14 as shown in FIG. Spread over the area. When the opening position of the shutter 12 is the first pattern, the light is blocked by the shutter 12, so that no light enters the imaging cell array 14 as shown in FIG.

  When the object 15 is at the far point position C and the opening position of the shutter 12 is the first pattern, as shown in FIG. 5A, light is incident near the center of the section T (3, 3). When the opening position is the second pattern, the peripheral lenses M (2,3), M (3,2), M (3,4), M (on the microlens array 13 as shown in FIG. 4 and 3) is incident on the imaging cell array 14. As shown in FIG. 5D, the light incident on these peripheral lenses is divided into areas T (1,3), T (2,3), T (3,1), T ( 3,2), T (3,4), T (3,5), T (4,3), and part of T (4,5).

  When the object 15 is located at the near point position B, the shape of the light incident on the imaging cell array 14 is as shown in FIG. 5C or FIG. When the opening position is the first pattern, light is incident on the area T (3, 3) as shown in FIG. When the opening position is the second pattern, light is emitted to a part of the sections T (3, 3), T (3, 2), T (3,4), T (2, 3) T (4, 3). Incident.

(3. Image reconstruction)
Next, image reconstruction on each virtual plane, which is a basic operation for calculating distance information in this apparatus, will be described.

  When reconstructing an image, it is important that information on the incident direction of the object light on the microlens can be obtained from the image data acquired by the image sensor camera 1. The fact that information on the incident direction can be obtained from the image data will be described with reference to FIG. 5 again. For example, when the opening position is the second pattern, the light from the object 15 at the far point position C enters the cells around the area T (3, 3) as shown in FIG. . Further, as shown in FIG. 5F, the light from the object 15 at the near point position B is incident on the cell centered on the section T (3, 3). Thus, the distance from the area T (3, 3) of the cell on which the light is incident differs depending on whether the object 15 is at the far point position C or the near point position B. By utilizing this fact, it is possible to obtain information on the incident direction of the object light on each microlens depending on which cell the light is incident on.

  Based on the information on the incident direction, the position of the virtual surface (virtual position), and the parameters of the optical system (such as the distance between the main lens 11 and the microlens array 13), the pixel data acquired by the imaging cell array 14 is rearranged. Furthermore, by combining the image data obtained by the rearrangement, it is possible to reconstruct the image data to be acquired when the imaging cell array 14 is at the virtual position.

  A specific example of reconstruction will be described with reference to FIG. FIG. 6 is a diagram illustrating a reconstructed image obtained from data of a captured image of the object 15.

  When the object is at the in-focus position A, the reconstructed image of FIG. 6B is obtained by combining the image data of FIGS. 5B and 5E.

  FIG. 6A shows the light distribution on the virtual plane F reconstructed from the image data shown in FIGS. 5A and 5D, and the light beam state at the in-focus position. Is almost the same. The output signal (image signal) obtained by converting the light incident on the surrounding fixed cell is rearranged according to the incident direction that can be understood based on the cell position, and the output signal near the center of the section T (3, 3). Can be added to create a virtual focal plane image.

  Similarly, the reconstructed image of FIG. 6C can be generated for the object (point image) at the near point position B. However, since the spread (divided state) on the imaging cell array 14 is different from the far point state as shown in FIG. 5C, the rearrangement calculation is the case where the reconstructed image of FIG. 6A is obtained. Different.

  FIG. 6 shows a reconstructed image to be obtained on the virtual plane at the focal position. In general, reconstructed images are calculated for a plurality of virtual positions. Among the plurality of virtual positions, there is a position where a reconstructed image in focus as shown in FIG. 6B can be obtained. From that position, the distance to the object can be obtained.

  Specifically, a virtual position where the maximum amount of light has entered a region where light transmitted through each opening can enter is determined as a focused position. In this embodiment, the total or average value of the amount of light incident on each region (hereinafter collectively referred to as “luminance information”) is calculated, and the virtual position when the luminance information is maximized is the focused position. Suppose that

  Although an example in which the object is a point image is shown here, an image can be reconstructed by a similar operation even in the case of an object having a size. However, in the case of an image having a size, since light enters all the openings, it is necessary to perform the same operation on the light transmitted through each opening.

(4. Distance resolution)
By using image data acquired with a plurality of aperture patterns as in the present invention, the distance resolution can be improved as compared to the case where no shutter is used. This will be described with reference to FIGS. FIG. 7 is a diagram showing light incident on the optical system and the imaging cell when no shutter is used. FIG. 8 is a diagram showing light incident on the optical system and the imaging cell according to the first embodiment.

  First, distance resolution when no shutter is used will be described with reference to FIG. The distance resolution depends on the number of pixels arranged on the X-axis or Y-axis of the incident light transmitted through each microlens. When the number of pixels decreases, the distance resolution decreases.

  Therefore, the distance resolution depends on the F number of the main lens and the micro lens. Here, the F number of the lens is (lens focal length) / (lens diameter). In FIG. 7A, the F number (Fm) of the main lens is fm / Wm, and the F number ( Fa) is expressed as fa / Wa.

  When Fm is increased, the size of light on the imaging cell array 14 is reduced, so that the distance resolution is lowered. Therefore, it seems that Fm should be reduced in order to improve the distance resolution.

  However, under the condition of Fm <Fa, the size of the light transmitted through the microlens is larger than the microlens array pitch, so that the outer edges of the light that has passed through different microlenses overlap. This overlap makes it impossible to obtain distance information.

  Therefore, the maximum distance resolution can be obtained when the shutter is not used when the F number Fm of the main lens and the F number Fa of the micro lens satisfy Fm≈Fa. FIG. 7B shows a light distribution pattern on the imaging cell array 14 at this time. In FIG. 7B, the intensity distribution of the light transmitted through each microlens is indicated by a broken line. A reconstructed image is obtained by rearranging the image signals within a square area surrounding the dashed circle.

  Next, the case where the shutter 12 is used as shown in FIG. In this case, Fa can be increased compared to the case where no shutter is used. In the present embodiment, it is assumed that Fa = √2 × Fm.

  Even when Fa = √2 × Fm, since light from some microlenses is blocked by the shutter 12, the distribution pattern of light transmitted through the shutter 12 and the microlens array 13 on the imaging cell array 14 is As shown by the broken line in FIG. That is, light that passes through different microlenses enters different regions. In the case of Fa = √2 × Fm, the outer peripheries of the respective regions are in contact with each other, and the number of cells corresponding to the light from each opening is maximized.

  As described above, even if the size of the light distribution pattern on the imaging cell array 14 is increased to √2 times (≈1.4 times) the microlens array pitch, the outer edges of the light distribution patterns do not overlap. The number of cells to obtain can be increased.

  In this embodiment, since the F number of the main lens is determined so that Fa = √2 × Fm, the size of the light distribution pattern on the imaging cell array 14 is enlarged to √2 times the microlens array pitch. The Therefore, a square region surrounded by a thick line in FIG. 8B can be used as a region corresponding to one microlens. The number of cells on the X-axis or Y-axis in this region is approximately 1.4 times that of the case without a shutter, and a distance resolution of 1.4 times can be obtained.

  Assuming that distance information is acquired from an image corresponding to one pattern of the shutter 12, the resolution in the X direction and the Y direction is halved. Therefore, in the present embodiment, the depth (distance) resolution is improved by a factor of 1.4 while maintaining the horizontal resolution by acquiring the captured image in two steps by changing the aperture position. ing.

(5. Process flow)
The flow of processing in distance measurement using the distance measuring apparatus according to the first embodiment will be described with reference to FIG. FIG. 9 is a flowchart of distance measurement according to the first embodiment.

  In step S101, the camera 1 acquires image data when the opening pattern of the shutter 12 is the first pattern.

  In step S102, the camera 1 acquires image data when the opening of the shutter 12 is the second pattern.

  In step S103, the reconstruction unit 4a rearranges the image data obtained in steps S101 and S102. Further, the rearranged image data is synthesized to create a reconstructed image on a plurality of virtual planes.

  In step S104, the distance information calculation unit 4b calculates the luminance information of the reconstructed image. That is, the sum or average value of the light intensities detected in the region corresponding to the light that has passed through each opening is calculated.

  In step S105, the distance information calculation unit 4b obtains a virtual surface that gives the largest luminance information for each region. Here, such a virtual surface is referred to as a representative surface.

  In step S106, the distance information calculation unit 4b converts the position of the representative surface into a distance to the object using the parameters of the optical system.

(6. Modification of the first embodiment)
In the first embodiment, the shutter 12 is a liquid crystal shutter. However, what can implement the two patterns described can be used instead of the liquid crystal shutter. For example, a metal mask having an opening shown in FIG. 10 that can be moved in the X or Y direction by a distance corresponding to the diameter of the microlens can be used as the shutter 12.

[Second Embodiment]
Next, a distance measuring apparatus according to the second embodiment will be described. The distance measuring device according to the second embodiment has substantially the same configuration as the distance measuring device according to the first embodiment. However, the opening pattern of the shutter 12 is different from that of the first embodiment. Therefore, the image reconstruction method is also different.

  The opening pattern of the shutter 12 will be described with reference to FIG. FIG. 11 is a diagram showing the shape of the incident light 16 on the microlens array 13. The white portion in FIG. 11 indicates the opening of the shutter 12, and the shaded portion indicates the light shielding portion of the shutter 12.

  First, the shutter 12 is controlled to provide an opening in one of the four sections as shown in FIG. 11A in the first step. In the second step, as shown in FIG. 11B, the shutter 12 is controlled so as to provide an opening in a section different from the first step among the four sections. Furthermore, the shutter 12 is controlled in the third and fourth steps so that the opening position changes as shown in FIGS. 11C and 11D, respectively.

  In the present embodiment, it is assumed that the F number of the microlens is set to twice the F number of the main lens. Therefore, the size (indicated by a bold circle in FIG. 11) of the light distribution pattern from the point image at the in-focus position A after passing through the shutter 12 and the microlens array 13 is indicated by the bold circle in FIG. Double the F number and the main lens F number.

  The incident light 16 to the imaging cell array 14 when the object 15 in FIG. 3 is located at the far point position C will be described with reference to FIG. FIG. 12 is a diagram showing the shape of the incident light 16 on the imaging cell array 14 when the object 15 is located at the far point position C. FIG.

  The captured image at this time changes as shown in FIG. 12 corresponding to the switching of the pattern of the shutter 12. The captured images in the aperture states of FIGS. 11A to 11D are as shown in FIGS. 12A to 12D, respectively.

  The light from the point C is once condensed through the main lens 11, spreads in a defocused state, and enters the shutter 12 and the microlens array 13. At this time, as shown in FIG. 11, not only the micro lens of M (3, 3) but also M (2, 2), M (3, 2), M (4, 2), M (2, 3). , M (4,3), M (2,4), M (3,4), M (4,4) micro-lenses (hereinafter referred to as peripheral lenses) also spread light.

  The shutter 12 of the first pattern blocks light that passes through the peripheral lens. Therefore, in the first step, only the light transmitted through the microlenses M (3, 3) is guided to the imaging cell array 14. At this time, light as shown in FIG. 12A is incident on the imaging cell array 14.

  In the second step, since the opening pattern is as shown in FIG. 11B, the microlenses M (3,3) are shielded from light and the shutters on the peripheral lenses M (3,2) and M (3,4) are provided. Is released. Accordingly, light having the shape shown in FIG. 12B is incident on the imaging cell array 14.

  In the third step, the opening position of the shutter 12 is as shown in FIG. 11C, the microlenses M (3,3) are shielded from light, and the peripheral lenses M (2,2), M (4,2), M The shutters on (2, 4) and M (4, 4) are opened. Light having a shape as shown in FIG. 12C is incident on the imaging cell array 14.

  In the fourth step, the opening position of the liquid crystal shutter 12 changes as shown in FIG. At this time, the lens M (3, 3) is shielded from light, and the shutters on the peripheral lenses M (2, 3), M (4, 3) are opened. Light having a shape as shown in FIG. 12D enters the imaging cell array 14.

  A reconstructed image as shown in FIG. 12E is obtained for the four captured images obtained in the four steps by the same operation as described in the first embodiment. The reconstructed image shown in FIG. 12 (e) is a case where the virtual plane is at the focused position. Distance information can be obtained from reconstructed images corresponding to a plurality of virtual planes.

  In the present embodiment, as shown in FIG. 13, the spot diameter formed in the imaging cell array 14 by the light from the point image at the in-focus position A makes the F number of the microlens and the F number of the main lens the same. Twice the time. A pixel group included in a square region surrounded by a thick line in FIG. 13B corresponds to the spot diameter. The number of pixels in each direction of the pixel group is twice that when the F number of the microlens and the F number of the main lens are made the same. Therefore, the distance resolution in the Z direction is twice that at that time.

  However, since the resolution in the X and Y directions is ¼ at this time, the depth resolution is set to 2 while maintaining the horizontal resolution by changing the aperture position and acquiring the captured image in four steps. It has improved twice.

  Note that this change in opening position can also be realized by moving the metal mask having the opening shown in FIG. 11 in the X and Y directions by a distance corresponding to the diameter of the microlens, as in the first embodiment. .

  With reference to FIG. 14, the flow of the distance measurement process according to the second embodiment will be described. FIG. 14 is a flowchart of distance measurement according to the second embodiment.

  In step S201, the camera 1 acquires image data when the opening pattern of the shutter 12 is the first pattern.

  Next, in step S202, the camera 1 acquires image data when the opening of the shutter 12 is the second pattern.

  Next, in step S203, the camera 1 acquires image data when the opening of the shutter 12 is the third pattern.

  Next, in step S204, image data when the opening of the shutter 12 is the fourth pattern is acquired by the camera 1.

  In step S205, the reconstruction unit 4a rearranges the image data obtained in steps S201 to S204. Further, the rearranged image data is synthesized to create a reconstructed image on a plurality of virtual planes.

  In step S206, the distance information calculation unit 4b calculates the luminance information of the reconstructed image. That is, the sum or average value of the light intensities detected in the region corresponding to the light that has passed through each opening is calculated.

  In step S207, the distance information calculation unit 4b obtains a virtual surface that gives the largest luminance information for each region. Here, such a virtual surface is referred to as a representative surface.

  In step S208, the distance information calculation unit 4b converts the position of the representative surface into the distance to the object using the parameters of the optical system.

[Summary]
As described above, a plurality of images in different focusing states (virtual surfaces) are generated by image processing from a plurality of imaging data acquired by changing the aperture pattern of the shutter by the distance information calculation algorithm, and the object is obtained from the data Can be obtained. It is also possible to obtain the tilt and posture from the positional relationship before and after in the depth direction and the distance information of different edges of the object. Conventional plenoptic cameras such as those described in Non-Patent Documents 1 and 2 can reconstruct an image focused on a set distance, but cannot obtain distance information to an object. . On the other hand, according to the present invention, distance information can be calculated by using luminance information in a region including a plurality of cells corresponding to one microlens instead of a cell unit of the image sensor.

  By using a shutter having a pattern in which some microlenses are shielded from light, the number of cells under one microlens can be increased and the depth resolution can be improved. Furthermore, by using the image data acquired for the aperture patterns of the plurality of shutters, the depth resolution can be improved while maintaining the horizontal resolution.

  Further, as in Patent Document 1, after autofocusing is performed by changing the height of an object on a stage or the like, a plurality of images are acquired, and height information is obtained based on the luminance information. In contrast, the method of the present invention eliminates the need for a camera lens focusing mechanism and a moving mechanism such as a stage, and is therefore small and low-cost.

  Furthermore, since this optical system is configured to incorporate a microlens array in the immediate vicinity of the image sensor, it is almost the same size as a conventional image sensor camera, so it is widely used in visual sensors such as robots for FA applications. Because it fits in the same outer diameter size as the image sensor camera, compatibility (replacement) with conventional cameras is high.

  The embodiments disclosed herein are illustrative and non-restrictive in every respect. The scope of the present invention is shown not by the above-described embodiment but by the scope of claims, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

It is a schematic block diagram of the distance measuring device which concerns on embodiment of this invention. 1 is a diagram illustrating an optical system of an image sensor camera 1. FIG. It is a figure which shows the mode of condensing in case the target object 15 exists in the near point position B or the far point position C. It is a figure which shows the shape of the incident light 15 on the micro lens array. It is a figure which shows the shape of the incident light 16 on the imaging cell array. It is a figure which shows the reconstruction image acquired from the data of the picked-up image of the target object. It is a figure which shows the incident light to the optical system and imaging cell when not using a shutter. It is a figure which shows the incident light to the optical system and imaging cell which concern on 1st Embodiment. It is a flowchart of distance measurement concerning a 1st embodiment. It is a figure which shows the metal mask which can be used as a shutter. It is a figure which shows the shape of the incident light 16 on the micro lens array. It is a figure which shows the shape of the incident light 16 on the imaging cell array 14 when the target object 15 is located in the far point position C. It is a figure which shows the incident light to the optical system and imaging cell which concern on 2nd Embodiment. It is a flowchart of the distance measurement which concerns on 2nd Embodiment. It is a figure which shows the structure of the height information measuring apparatus using the principle of a confocal microscope. It is a figure explaining the detection algorithm of the sample height information by the height information measuring device using the principle of a confocal microscope.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Image sensor camera, 2 Image capture board, 3 Memory, 4 Image processing part, 4a Reconstruction part, 4b Luminance information calculation part, 4c Distance information calculation part, 5 Display monitor, 11 Imaging lens, 12 Shutter, 13 Micro lens array , 14 imaging cell array, 15 object, 16 incident light.

Claims (7)

A distance measuring device for measuring a distance to an object,
A camera having an image sensor;
The imaging device is composed of a plurality of imaging cells each converting object light from the object into an image signal,
The camera
An imaging lens for condensing the object light;
A condensing array comprising a plurality of concentrators on which the object light that has passed through the imaging lens is incident;
Including a shutter for controlling light incident on the image sensor,
The shutter has a plurality of arrangement patterns in which a plurality of light-shielding portions that shield light that passes through the plurality of collectors and a plurality of openings that pass light that passes through the plurality of collectors are respectively arranged. And
The light that passes through each of the light collecting parts passes through the shutter in any one of the arrangement patterns,
In each of the arrangement patterns, the object lights transmitted through the openings are different from each other on the image sensor, and each is incident on a region larger than the size of the condenser,
The image signals obtained by the camera in each of the arrangement patterns are rearranged based on the incident direction of the object light to the condenser determined by the position of the imaging cell that detected the object light in each of the regions. Reconstructing means for synthesizing the rearranged image signals and generating a reconstructed image to be obtained when the image sensor is located at each of a plurality of virtual positions;
A distance calculation unit that specifies the virtual position where the amount of light incident on each of the regions is maximum, converts the specified virtual position based on a parameter of an optical system of the camera, and calculates the distance; Distance measuring device.   The distance measuring device according to claim 1, wherein outer peripheries of the respective regions are in contact with each other.   The distance measuring device according to claim 1 or 2, wherein in each of the arrangement patterns, the plurality of light shielding portions and the plurality of openings are arranged in a checkered pattern.   In each of the arrangement patterns, the plurality of light shielding portions and the plurality of openings corresponding to the adjacent 2 × 2 light collectors are a predetermined one of the 2 × 2 light collectors. The distance measuring device according to claim 1, wherein the distance measuring device is arranged so as to transmit the light of.   The distance measuring device according to claim 1, wherein the shutter is a liquid crystal shutter.   5. The distance measuring device according to claim 1, wherein the shutter is a light shielding mask having a predetermined opening and movable in accordance with the plurality of patterns. 6. A distance measurement method for measuring a distance to an object using a camera having an image sensor,
The imaging device is composed of a plurality of imaging cells each converting object light from the object into an image signal,
The camera
An imaging lens for condensing the object light;
A condensing array comprising a plurality of concentrators on which the object light that has passed through the imaging lens is incident;
Including a shutter for controlling light incident on the image sensor,
The shutter has a plurality of arrangement patterns in which a plurality of light-shielding portions that shield light that passes through the plurality of collectors and a plurality of openings that pass light that passes through the plurality of collectors are respectively arranged. And
The light that passes through each of the light collecting parts passes through the shutter in any one of the arrangement patterns,
In each of the arrangement patterns, the object lights transmitted through the openings are different from each other on the image sensor, and each is incident on a region larger than the size of the condenser,
Obtaining the image signal with the camera in each of the arrangement patterns;
The image signals obtained by the camera in each of the arrangement patterns are rearranged based on the incident direction of the object light to the condenser determined by the position of the imaging cell that detected the object light in each of the regions. Combining the rearranged image signals and generating a reconstructed image to be obtained when the image sensor is located at each of a plurality of virtual positions;
A step of specifying the virtual position where the amount of light incident on each of the regions is maximum, converting the specified virtual position based on a parameter of an optical system of the camera, and calculating the distance. .

Images