JP2021018081A - Imaging apparatus, measuring device, and measuring method - Google Patents

Abstract

To prevent a whiteout or black crushing in a photographed image of an object without increasing the number of times of photography and without increasing the size of a system.SOLUTION: An imaging apparatus is used which has a filter unit that includes a plurality of filter piece parts different in extinction ratio of light, and a plurality of light receiving elements that each receive the light passing through the filter piece parts and each output a light receiving signal corresponding to information on the received light. The imaging apparatus irradiates an object with light of a predetermined projection pattern, and measures depth information on the object by selectively using the light receiving signals through the filter piece parts different in extinction ratio in the imaging apparatus. Thus, light receiving signals in a number of filters different in extinction ratio (light receiving signals different in exposure amount) can be obtained in single photography. Consequently, a whiteout or black crushing in a photographed image of the object can be prevented, while preventing an increase in the number of times of photography and preventing an increase in size of a system.SELECTED DRAWING: Figure 21

Description

The present invention relates to an imaging device, a measuring device, and a measuring method.

In recent years, a three-dimensional shape measuring device for measuring the surface shape of an object by a contact type or a non-contact type has been known. In the case of a contact-type three-dimensional shape measuring device, since the coordinates are measured while the sensor is in contact with the object to be measured, it takes some time to measure, but it is possible to measure with high accuracy. On the other hand, the non-contact type three-dimensional shape measuring device is slightly inferior to the contact type in terms of accuracy, but can measure in a short time. For this reason, non-contact three-dimensional shape measuring devices are currently used more often than contact-type three-dimensional shape measuring devices.

As the "non-contact type", many measurement methods such as an optical cutting method, a stereo method, and a TOF (Time of Flight) are known. Among them, as a measurement method using triangulation, a passive stereo method (stereo camera) using two camera devices and a measurement object in which slit light or pattern light is projected using a projector device are measured by the camera device. The active stereo method for imaging is known. As the light source of the projector device, for example, a semiconductor laser such as a Vertical Cavity Surface Emitting Laser (VCSEL) is used.

In the case of the active stereo method, since the light is projected onto the measurement object by itself, it is possible to perform three-dimensional measurement of the measurement object having a flat three-dimensional shape without features, and to increase the projection brightness. Therefore, it is possible to measure dark objects for which it is difficult to obtain reflected light. Therefore, a three-dimensional shape measuring device that performs three-dimensional measurement by the active stereo method is often used for industrial FA (Factory Automation) such as a robot.

On the other hand, in Patent Document 1 (Japanese Unexamined Patent Publication No. 2013-157360), when the three-dimensional shape of the measurement object is recognized by using the phase shift method, the camera lens of the imaging unit has a different dimming rate (ND). A mounting substrate production apparatus that prevents the occurrence of overexposure or underexposure by using a Neutral Density filter is disclosed.

However, the actual measurement target has various shapes, materials, surface conditions, and the like, and even if it is one object, there is an object whose reflectance greatly differs depending on the measurement location. In this case, the width between the minimum value and the maximum value of the luminance value of the captured image data is larger than that in the case of measuring an object having a uniform reflectance such as a test plate. Therefore, in the phase shift method in which the amount of projected light is changed, overexposure or underexposure occurs in the captured image.

A method of taking a plurality of shots by changing the exposure time of the camera is known in order to prevent overexposure or underexposure of the shot image, but this method has a problem that the number of shots increases. Further, a method of branching the optical path of incident light and taking an image with two image pickup elements is known, but in this method, there is a problem that the system becomes large and the cost becomes high.

The same applies to the mounting board production apparatus disclosed in Patent Document 1, and since a moving mechanism for a measurement object or a camera is required, there is a problem that the system becomes large and the cost increases.

The present invention has been made in view of the above-mentioned problems, and is capable of preventing overexposure or underexposure of a photographed image of an object without increasing the number of times of photography and increasing the size of the system. An object of the present invention is to provide an apparatus, a measuring apparatus, and a measuring method.

In order to solve the above-mentioned problems and achieve the object, the present invention receives a filter unit having a plurality of filter pieces having different light dimming rates and each light transmitted through each filter piece. It has a plurality of light receiving elements that output light receiving signals corresponding to the information of the received light.

According to the present invention, it is possible to prevent overexposure or underexposure of a photographed image of an object without increasing the number of times of photographing and without increasing the size of the system.

FIG. 1 is a configuration diagram of a main part of a three-dimensional shape measuring device according to the first embodiment. FIG. 2 is a diagram showing a state of light projection onto a measurement object by an optical device provided in the three-dimensional shape measuring device according to the first embodiment. FIG. 3 is a diagram showing an example of the configuration of the VCSEL chip. FIG. 4 is a diagram showing another configuration of the VCSEL chip. FIG. 5 is a diagram showing an example of an optical system of an optical device. FIG. 6 is a diagram showing another example of the optical system of the optical device. FIG. 7 is a diagram showing another example of the optical system of the optical device. FIG. 8 is a diagram showing another example of the optical system of the optical device. FIG. 9 is a diagram showing another example of the optical system of the optical device. FIG. 10 is a diagram showing another example of the optical system of the optical device. FIG. 11 is a diagram showing a configuration of a MEMS mirror which is an example of an optical deflection element. FIG. 12 is a diagram showing a configuration of a polygon mirror which is an example of an optical deflection element. FIG. 13 is a diagram showing the configuration of the camera. FIG. 14 is a block diagram of the three-dimensional shape measuring device. FIG. 15 is a diagram for explaining measurement using the phase shift method. FIG. 16 is a diagram for explaining measurement using the optical cutting method. FIG. 17 is a diagram showing a state in which three-dimensional measurement of a measurement object is performed in the three-dimensional measuring device of the first embodiment. FIG. 18 is a diagram showing an example of a sinusoidal fringe pattern irradiated from the optical device 10 to the object to be measured. FIG. 19 is a diagram showing an example of a captured image in which overexposure occurs. FIG. 20 is an equivalent circuit diagram of the pixels of the CMOS image sensor. FIG. 21 is a perspective view for explaining an ND filter provided on an image sensor of a camera device in the three-dimensional measuring device of the first embodiment. FIG. 22 is a flowchart showing the flow of the three-dimensional measurement operation in the three-dimensional measuring device according to the first embodiment. FIG. 23 is a perspective view of the robot arm according to the second embodiment. FIG. 24 is a perspective view of a smartphone according to the third embodiment. FIG. 25 is a perspective view of the inside of the automobile according to the fourth embodiment. FIG. 26 is a diagram showing an example in which a three-dimensional measuring device is provided on an autonomous moving body in the fourth embodiment. FIG. 27 is a perspective view of a main part of the three-dimensional printer device according to the fifth embodiment.

Hereinafter, embodiments of an imaging device, a measuring device, and a measuring method will be described.

[First Embodiment]
First, the first embodiment is an application example to a three-dimensional shape measuring device. FIG. 1 is a configuration diagram of a main part of the three-dimensional shape measuring device according to the first embodiment. As shown in FIG. 1, the three-dimensional shape measuring device 1 of the first embodiment has a measurement information acquisition unit 20 and a control unit 30.

(Configuration of measurement information acquisition unit)
The measurement information acquisition unit 20 has an optical device 10 which is a projection unit and a camera 21 which is an imaging unit. The optical device 10 includes a VCSEL (Vertical Cavity Surface Emitting Laser) chip 11, a line generator 12 (optical system), and a light deflection element (mirror) 13.

The measurement information acquisition unit 20 operates under the control of the control unit 31 of the control unit 30. For example, a plurality of light emitting elements a of the VCSEL chip 11 are optically oscillated, and the light emitted through the line generator 12 is deflected by the optical deflection element 13 to scan the measurement target. The control unit 31 projects pattern light onto the entire measurement object by controlling the emission timing and intensity of each light emitting element a of the VCSEL chip 11 during optical scanning. For example, by controlling the light emitting element a on and off, a desired projection pattern such as a black and white Gray code pattern can be projected onto the measurement object.

The camera 21 has a fixed installation position and shooting angle so as to capture the projection area of the projection pattern. Specifically, the camera 21 is installed at an installation position and so that the projection center 300 of the pattern light (projected image) projected by the optical device 10 (an example of the irradiation unit) is the center of the imaging region 40 at the measurement target position. The shooting angle is fixed. The configuration of the optical device 10 described in the present embodiment is an example of an irradiation unit, and may be a configuration in which light of a plurality of different wavelengths of a predetermined projection pattern is irradiated to a measurement object as one or more groups. ..

The camera 21 has a lens 210 and an image sensor 211. As the image sensor 211, for example, an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) is used. The light incident on the camera 21 is imaged on the image sensor 211 via the lens 210. The image pickup device 211 forms an image pickup signal, which is an electric signal, by photoelectrically converting the imaged light. This image pickup signal is converted into an image signal and supplied to the arithmetic processing unit 32 of the control unit 30.

(Control unit configuration)
The control unit 30 has a control unit 31 and an arithmetic processing unit 32. The control unit 30 controls to project the pattern light from the optical device 10, controls to capture the pattern light by the camera 21, and the like. The control unit 30 performs arithmetic processing such as three-dimensional measurement of the object to be measured based on the image signal supplied from the camera 21. The control unit 31 may control to switch the pattern light projected by the optical device 10 to another pattern light. Further, the control unit 31 may control the arithmetic processing unit 32 to output the calibration information used for calculating the three-dimensional coordinates.

The arithmetic processing unit 32 (an example of the measurement unit) of the control unit 30 calculates the three-dimensional coordinates based on the input image signal, and acquires the three-dimensional shape of the object to be measured. Further, the arithmetic processing unit 32 outputs the three-dimensional shape information indicating the calculated three-dimensional shape of the measurement target object to, for example, a personal computer device. Although FIG. 1 shows a configuration in which one set of measurement information acquisition units 20 is provided for the control unit 30, a configuration in which a plurality of sets of measurement information acquisition units 20 are provided for the control unit 30 may be provided.

(Operation of optical device)
FIG. 2 is a diagram showing a state of light projection on a measurement object by the optical device 10. In FIG. 2, the optical device 10 emits the line light 14 toward the measurement object 15. The line light 14 is light in which a plurality of lights from each light emitting element a of the VCSEL chip 11 are overlapped with each other, is deflected by the mirror surface of the light deflection element (mirror) 13, and is a measurement target as shown by a broken line in FIG. The object 15 is irradiated. Specifically, the light deflection element 13 deflects the light emitted to the mirror surface by driving the mirror surface in the direction M around the longitudinal axis of the line light shown in FIG. Each line light is controlled so as to have a predetermined pattern light. As a result, the measurement object 15 is irradiated with the two-dimensional pattern light, and the projected image 60 is projected on the measurement object 15. The projected image 60 is projected onto an area including, for example, the measurement object 15.

(Configuration of VCSEL)
FIG. 3 is a diagram showing an example of the configuration of the VCSEL chip 11. The VCSEL chip 11 shown in FIG. 3 is a surface-emitting semiconductor laser in which light emitting elements a can be easily integrated on the same substrate, and has a plurality of light emitting elements a arranged one-dimensionally.

In VCSEL, the average brightness of the speckle pattern (“speckle pattern” is also referred to as “speckle pattern”) formed by each light emitting source is Si, the standard deviation is σi, and the speckle contrast is Csi. When laser irradiation is performed from each light emitting source with the same power, it can be considered that S1 = S2 = S3 = ... = S0, σ1 = σ2 = σ3 = ... = σ0. When n speckle pattern images are combined, the brightness value of the combined image (superimposed image) is S1 + S2 + ... + Sn = S0 × n.

Since the additivity of the variance holds for the variation, σ2 = σ12 + σ22 + ... + σn2, and σ = √ (n × σ02) = σ0√n.

Therefore, the speckle contrast Csn of n composite images is expressed as Csn = σ√n / (S × n) = (√n / n) × (σ0 / S0) = 1 / √n × Cs0.

That is, the speckle contrast can be improved to 1 / √n by synthesizing the images of n speckle patterns, and as a result, the speckle noise can be reduced.

The pitch of each light emitting element a shown in FIG. 3 may be arbitrary as long as the interval D1 between the virtual light sources m1, m2, ... Can be expected to have a speckle noise reduction effect of 1 / √n.

The arrangement of the light emitting elements a shown in FIG. 3 is an example, and the light emitting elements a may be arranged two-dimensionally. For example, the arrangement may be a honeycomb structure in which more elements can be arranged, and the arrangement is not limited to this, and other arrangements may be used. Further, although the shape of the openings of the plurality of light emitting elements a is shown by a quadrangle, for example, it may be a hexagon or the like, and the shape is not limited to this and may be another shape. Further, the wavelength of the laser light of each light emitting element a may be appropriately set. For example, it may be visible or invisible. Each light emitting element a may be configured so that light emission can be controlled independently.

FIG. 4 is a diagram showing another configuration of the VCSEL chip 11. The VCSEL chip 11 shown in FIG. 4 has at least one or more light emitting element group a1 called a layer that causes a plurality of light emitting elements to emit light together. FIG. 4 shows a form in which the light emitting element group a1 is arranged one-dimensionally, but may have a configuration in which the light emitting element group a1 is arranged two-dimensionally.

In the layer 222 shown in FIG. 4, five light emitting elements a2 are arranged in a cross shape. Within the same layer 222, each light emitting element a2 emits light at the same timing.

The pitch A of each layer 222 and the pitch (pitch B and pitch C) of each light emitting element a2 shown in FIG. 4 are appropriately different depending on the specifications of the measuring device 1, but the virtual light sources m1, m2, ... Any setting is sufficient as long as the interval D1 is set so that the speckle noise reduction effect of 1 / √n can be expected.

Here, five light emitting elements a2 of the layer 222 are arranged in a cross shape, but the present invention is not limited to this. The number of light emitting elements a2 may be increased or decreased, or more light emitting elements a2 may be arranged in a layout such as a honeycomb structure.

Further, although the opening of the light emitting element a2 is also shown as a quadrangle, it may have another shape such as a hexagon. The light emission may be controlled independently in each layer 222.

(Lens configuration of line generator)
FIG. 5 is a diagram showing an example of the optical system of the optical device 10. Of the upper and lower views of FIG. 5, the upper figure is a view of the optical system of the optical device 10 as viewed from the horizontal direction (H). Further, among the upper and lower views of FIG. 5, the lower figure is a view of the optical system of the optical device 10 as viewed from the vertical direction (V).

FIG. 5 shows an example of the lens configuration of the line generator 12 using four cylindrical lenses 121 to 124. The cylindrical lenses 121 to 124 convert the light from each light emitting element a of the VCSEL chip 11 into line light.

Specifically, in the horizontal direction (H), the light emitted from the VCSEL chip 11 is converted into a parallel light flux or a substantially parallel light flux by the cylindrical lens 121, and the line light width in the lateral direction is formed by the cylindrical lens 123. Further, in the vertical direction (V), the light emitted from the VCSEL chip 11 is converted into a parallel light flux or a substantially parallel light flux by the cylindrical lens 122, and the line light length in the longitudinal direction is formed by the cylindrical lens 124. At this time, the focus is formed at the position where the light is focused on the light deflection element (mirror) 13. Each line light is formed on the light deflection element 13 with a setting that can be expected to have a speckle noise reduction effect of 1 / √n.

The material of the cylindrical lenses 121 to 124 is, for example, glass or plastic. The cylindrical lenses 121 to 124 may be formed of other members. Further, the cylindrical lenses 121 to 124 may be subjected to antireflection processing such as AR (Anti Reflection) coating.

Further, the cylindrical lens may be inserted in either direction, but considering the number of refractions, it is preferable to insert the cylindrical lens so that the convex surfaces face each other as shown in FIG.

The light deflection element 13 is driven around the longitudinal axis of the line light, and scans the measurement object 15 with the line light incident on the light deflection element 13. By modulating the output of the line light by the control unit 31 during scanning, a projected image of a predetermined pattern is projected onto the measurement object 15.

FIG. 6 is a diagram showing another example of the optical system of the optical device 10. FIG. 6 shows an example in which a spherical lens 126 and two cylindrical lenses (cylindrical lens 123, cylindrical lens 124) are used as an example of the lens configuration of the line generator 12. In the example of FIG. 5, the light emitted from the VCSEL chip 11 is formed into a parallel light flux or a substantially parallel light flux by using separate cylindrical lenses 121 and 122 in the horizontal direction (H) and the vertical direction (V). On the other hand, in the example of FIG. 6, the single n-spherical lens 126 converts the light emitted from the VCSEL chip 11 into a parallel luminous flux. This makes it possible to reduce the number of lenses required.

FIG. 7 is a diagram showing another example of the optical system of the optical device 10. FIG. 7 shows a line generator 12 using a cylindrical lens 121 as an example of the lens configuration. In the horizontal direction (H), the light emitted from the VCSEL chip 11 is formed by the cylindrical lens 121 to have a line light width in the lateral direction. In the vertical direction (V), only the light emitted from the VCSEL chip 11 forms the line light length in the longitudinal direction. In this configuration, only one lens is used, so that the number of lenses required can be reduced to the maximum.

FIG. 8 is a diagram showing another example of the optical system of the optical device 10. FIG. 8 shows an example of the lens configuration of the line generator 12 using two cylindrical lenses (cylindrical lens 121 and cylindrical lens 123).

In the horizontal direction (H), the light emitted from the VCSEL chip 11 is converted into a parallel light flux or a substantially parallel light flux by the cylindrical lens 121, and the line light width in the lateral direction is formed by the cylindrical lens 123. In the vertical direction (V), only the light emitted from the VCSEL chip 11 forms the line light length in the longitudinal direction.

FIG. 9 is a diagram showing another example of the optical system of the optical device 10. FIG. 9 shows a lens configuration shown in FIG. 5 with a diaphragm 125 added. When the light cannot be sufficiently focused on the light deflection element 13 in the lens configuration shown in FIG. 5, the diaphragm 125 is inserted. The aperture 125 is not limited to this. At least one diaphragm 125 may be inserted at any position. Further, although FIG. 9 shows the aperture 125 corresponding to the horizontal direction (H), the aperture 125 may be inserted in the vertical direction (V) as well. Further, the diaphragm 125 may be inserted for the purpose of removing stray light.

FIG. 10 is a diagram showing another example of the optical system of the optical device 10. FIG. 10 shows a configuration in which a microlens array 127 and a microcylindrical lens array 128 are inserted in front of the VCSEL chip 11 (in the optical axis direction) in order to control the divergence angle of each light emitting element a of the VCSEL chip 11. Shown. The configuration of the subsequent stage of the microcylindrical lens array 128 is not shown. The divergence angle of each light emitting element a can be controlled by the microlens array and / or the microcylindrical lens array. That is, a microlens array may be used to control the divergence angle of each light emitting element a, or a microcylindrical lens array may be used. Moreover, you may combine both of these. Here, as an example, an example in which both are combined is shown.

Each lens of the microlens array 127 has a spherical shape, and converts the light emitted from each light emitting element a of the VCSEL chip 11 into a parallel light flux or a substantially parallel light flux in the horizontal direction (H) and the vertical direction (V). Then, the luminous flux emitted from the microlens array 127 forms the line light length in the longitudinal direction shown in the vertical direction (V) by the microcylindrical lens array 128. With this configuration, the divergence angle of the VCSEL chip 11 is controlled.

Although there is only one row of light emitting elements a in the horizontal direction in FIG. 10, the light emitting elements a may be arranged in the horizontal direction and the light emitting elements on the VCSEL may be arranged in a matrix. Along with this, the microlens array and the microcylindrical lens array may also be formed in a matrix.

(Light deflection element)
Next, the light deflection element 13 is a movable mirror that scans the laser beam in the uniaxial or biaxial directions. As the movable mirror, for example, a MEMS mirror, a polygon mirror, a galvano mirror, or the like can be used. If the laser beam can be scanned in the uniaxial or biaxial directions, other methods may be used. The three-dimensional shape measuring device uses a movable mirror that uniaxially scans the line light 14 formed by the line generator 12 on the measurement object 15 in the scanning range. A two-dimensional planar projection pattern is formed by lightly scanning line light with a movable mirror.

FIG. 11 is a diagram showing a configuration of a MEMS mirror (also referred to as a MEMS mirror scanner) which is an example of the optical deflection element 13. The MEMS mirror scanner shown in FIG. 11 has a movable portion 132 and two sets of meandering beam portions 133 on the support substrate 131.

The movable portion 132 includes a reflection mirror 1320. One end of each of the two sets of meandering beam portions 133 is connected to the movable portion 132, and the other end is supported by the support substrate 131. Each of the two sets of meandering beam portions 133 is composed of a plurality of meander-shaped beam portions. Further, each of the two sets of serpentine beam portions 133 includes a first piezoelectric member 1331 that is deformed by applying a first voltage and a second piezoelectric member 1332 that is deformed by applying a second voltage. Have every other one.

The first piezoelectric member 1331 and the second piezoelectric member 1332 are independently provided for each adjacent beam portion. The two sets of meandering beam portions 133 are deformed by applying a voltage to the first piezoelectric member 1331 and the second piezoelectric member 1332, respectively, and rotate the reflection mirror 1320 of the movable portion 132 around the rotation axis.

Specifically, voltages having opposite phases are applied to the first piezoelectric member 1331 and the second piezoelectric member 1332 to generate warpage in each beam portion. As a result, the adjacent beam portions are bent in different directions, which are accumulated, and the reflection mirror 1320 reciprocates around the rotation axis together with the movable portion 132 connected to the two sets of meandering beam portions 133. Further, by applying a sine wave having a drive frequency matched to the mirror resonance mode centered on the rotation axis to the first piezoelectric member 1331 and the second piezoelectric member 1332 in opposite phase, the voltage is very low. A large rotation angle can be obtained.

The drive waveform is not limited to a sine wave. For example, it may be a sawtooth wave. Further, the drive may be performed not only in the resonance mode but also in the non-resonance mode.

FIG. 12 is a diagram showing a configuration of a polygon mirror which is an example of the light deflection element 13. The polygon mirror shown in FIG. 12 includes a plurality of plane mirrors 1320A on a rotating body 13A that rotates at a constant velocity in the M direction around a rotation axis. The line light input from the line generator 12 to the plane mirror 1320A uniaxially scans the measurement target by changing the angle of the plane mirror 1320A. As shown by the arrows in FIG. 12, the polygon mirror enables measurement over a wide range in the horizontal direction (direction perpendicular to the Y axis).

Further, in the configuration shown in FIG. 12, the tilt angles with respect to the rotation axis are different from each other on each mirror surface 1320A of the polygon mirror. When different tilt angles are given to each mirror surface 1320A in this way, the vertical emission angle of the line light is controlled, so that the vertical output angle changes each time the mirror surface 1320A changes due to the rotation of the rotating body 13A. become. Therefore, by giving each mirror surface 1320A a different tilt angle, it is possible to widen the scanning area in the vertical direction according to the number of firing surfaces provided in the polygon mirror.

(camera)
FIG. 13 is a diagram showing the configuration of the camera 21. The camera 21 has a lens 210 and an image sensor 211. As the image sensor 211, for example, a CCD image sensor, a CMOS image sensor, or the like can be used. The light incident on the camera 21 is imaged on the image sensor 211 via the lens 210 and photoelectrically converted. The electrical signal photoelectrically converted by the image sensor 211 is converted into an image signal, and the image signal is output from the camera 21 to the arithmetic processing unit 32 (see FIG. 1) of the control unit 30.

(Structure of control unit)
Next, FIG. 14 is a block diagram of the three-dimensional shape measuring device 1. In FIG. 14, the arithmetic processing unit 32 analyzes the image signal output from the camera 21. The arithmetic processing unit 32 performs the restoration processing of the three-dimensional information by the arithmetic processing using the analysis result of the image signal and the calibration information, and thereby executes the target three-dimensional measurement. The arithmetic processing unit 32 supplies the restored three-dimensional information to the control unit 31.

The control unit 31 includes a projection control unit 310, a pattern storage unit 311, a light source drive / detection unit 312, an optical scanning drive / detection unit 313, and an image pickup control unit 314.

The optical scanning drive / detection unit 313 drives the optical deflection element 13 under the control of the projection control unit 310. The projection control unit 310 controls the light scanning drive / detection unit 313 so that the line light applied to the deflection center of the light deflection element 13 scans the measurement target. The image pickup control unit 314 controls the image pickup timing, the exposure amount, and the like of the camera 21 according to the control of the projection control unit 310.

The light source drive / detection unit 312 controls lighting and extinguishing of each light emitting element of the VCSEL chip 11 according to the control of the projection control unit 310. The light source drive / detection unit 312 has a feedback control circuit. A part or all of the feedback control circuit may be provided on the VCSEL chip 11.

The pattern storage unit 311 is a non-volatile memory, and stores pattern information for forming a projected image (projection pattern). The projection control unit 310 controls the light source drive / detection unit 312 based on the pattern information read from the pattern storage unit 311.

Further, the projection control unit 310 may instruct the pattern storage unit 311 to read the pattern information based on the restored three-dimensional information supplied from the arithmetic processing unit 32, or the read pattern information may be used. The calculation method may be instructed to the calculation processing unit 32 accordingly.

The arithmetic processing unit 32, the projection control unit 310, and the image pickup control unit 314 have functions realized by software when the control unit 31 such as a CPU (Central Processing Unit) executes a measurement program stored in the storage unit. Is. Specifically, the control unit 31 reads and executes a measurement program stored in a storage unit such as a ROM (Read Only Memory) of the three-dimensional shape measuring device 1, thereby performing the arithmetic processing unit 32 and the projection control unit 310. And the image pickup control unit 314 is realized. A part of the arithmetic processing unit 32, the projection control unit 310, and the image pickup control unit 314 may be formed by hardware. Further, functions other than the arithmetic processing unit 32, the projection control unit 310, and the image pickup control unit 314 may be realized by the measurement program.

(Projection pattern)
Next, a projection pattern for scanning the measurement object will be described. As a three-dimensional measurement method for irradiating a measurement object with light and acquiring the shape and orientation of the measurement object as three-dimensional information, for example, a measurement method using a phase shift method and a measurement using an optical cutting method. The method is known. Each of these measurement methods is disclosed in, for example, the following non-patent documents.

(1) Improving the accuracy of active stereo by the phase shift method using the response function of the projector / camera system "Image Recognition and Understanding Symposium (MIRU2009)" July 2009

(2) "Appearance inspection technique using a three-dimensional image by an optical cutting method" RICOH TECHNICAL REPORT, No. 39, 2013, published January 28, 2014

First, the measurement using the phase shift method of (1) will be schematically described. In the phase shift method, a plurality of projection patterns 60 (10), 60 (11), 60 (12), and 60 (13), which are phase shift patterns having different phases, which are exemplified in FIG. 15A, are used. The three-dimensional shape and orientation are restored by phase analysis. At this time, a spatial coding method using a plurality of projection patterns 60 (20), 60 (21), 60 (22), and 60 (23), which are different gray code patterns, which are exemplified in FIG. 15 (b). By using the above in combination and performing phase connection based on the results of these spatial coding methods and phase shift methods, it is possible to restore the three-dimensional shape and orientation with high accuracy.

As described above, in the measurement using the phase shift method of (1), imaging is performed for each of the plurality of projection patterns 60 (10) to 60 (13) and 60 (20) to 60 (23).

Next, the measurement using the optical cutting method of (2) will be schematically described. In the light cutting method, a line light source irradiates a measurement target with a bright line, and the measurement target irradiated with the bright line is imaged to obtain a bright line image. For example, as shown in FIG. 16, a line light (bright line) 14 is formed from an optical deflection element. Based on this emission line image, a three-dimensional shape for one line to be measured is generated. As shown in the projection pattern 60 (3) of FIG. 16, the irradiation position of the line light 14 is changed in the direction of the arrow by using the light deflection element, and a plurality of emission line images are obtained for the measurement target. As a result, the entire three-dimensional shape of the measurement target can be generated. The light cutting method using such a light cutting pattern is suitable for measuring a glossy measurement target.

(Specific example of 3D measurement)
The above-mentioned phase shift method is a method of measuring a distance by triangulation as in stereo measurement, and is based on the measurement principle of an optical cutting method that restores a three-dimensional shape from the degree of distortion of line light according to the object shape (multiple lines). ) Is an extended measurement method. In the optical cutting method, since the information of one line is acquired by one measurement, it is necessary to measure the number of pixels in the horizontal direction in order to obtain the data of the entire object, and the measurement time is longer than that of the phase shift method. ..

As shown in FIG. 17, the three-dimensional measuring device 1 of the first embodiment has an optical device 10 such as a camera 21 and a projector device, and a phase image of a sinusoidal fringe pattern is measured from the optical device 10. It is projected onto an object and the reflected light is photographed by the camera 21. As illustrated in FIGS. 18 (a) to 18 (d), the measurement object is measured by projecting and imaging four types of images in which the phase of the sinusoidal fringe pattern is shifted by π / 2. FIG. 18A is an image of a sinusoidal fringe pattern having a phase difference of “0”, and FIG. 18B is a phase shift of π / 2 with respect to the sinusoidal fringe pattern having a phase difference of “0”. It is an image of a sine wave fringe pattern. Further, FIG. 18 (c) is an image of a sinusoidal fringe pattern whose phase is shifted by π with respect to a sinusoidal fringe pattern having a phase difference of “0”, and FIG. 18 (d) is an image of a sinusoidal fringe pattern having a phase difference of “0”. This is an image of a sine wave fringe pattern whose phase is shifted by 3π / 2 with respect to the sine wave fringe pattern of.

From the measured images of these four sinusoidal fringe patterns, the phase information of each pixel is calculated to correspond the optical center coordinates of the camera 21 with the optical center coordinates of the optical device 10, and the known camera 21 and the optical device 10 The three-dimensional shape on the surface of the object is calculated from the geometric relationship. Therefore, when the brightness value of each pixel of the camera 21 reaches the saturation upper limit, overexposure occurs as shown in FIG. 19, or when the reflected light is weak and buried in noise, blackout occurs, the phase information is displayed. It becomes difficult to calculate. The image of FIG. 19 is an image taken by the camera 21 by irradiating the metal gear with the above-mentioned sinusoidal stripe pattern. The vertical stripes in FIG. 19 are sinusoidal stripe patterns. Also. The white areas on the gear are the areas where overexposure occurs.

Next, the image sensor in the camera 21 will be described. As an example, as the image pickup device, a silicon IC having sensitivity in the visible light to NIR region such as a CCD image sensor or a CMOS image sensor is used. Several um pixels are two-dimensionally arranged, and the luminance value information is acquired by photoelectrically converting the incident light of each pixel and outputting it as an electric signal.

FIG. 20 is an equivalent circuit diagram of the pixels of the CMOS image sensor. In FIG. 20, the light incident on each pixel within the exposure time at the time of shooting is converted into electric charges by the photoelectric conversion element D1. The photoelectric conversion element D1 is a photodiode formed by a PN junction, and the generated charge is accumulated in the junction capacitance (parasitic capacitance) C1. When the exposure time is over, the transfer transistor Q2 is turned on and the electric charge accumulated in C1 is supplied to the voltage conversion unit Q3 and converted into an electric signal. This electric signal is output to the circuit in the subsequent stage when the output unit Q4 is turned on, and is converted into luminance value data.

Since the light is converted into an electric signal by such a flow, if the incident light is too bright, the charge is saturated at the junction capacitance C1 and the brightness value becomes a white level. Further, the image sensor has an offset noise component such as a dark current. Therefore, when the incident light is too dark (the amount of the incident light is too small), the electric signal is buried in noise and the brightness value becomes a black level. In the phase shift method, it is difficult to calculate the phase information in either state, so when performing three-dimensional measurement of an object with non-uniform reflectance, it is necessary to change the exposure time and perform multiple measurements. Become.

(Configuration of image sensor)
FIG. 21 shows a partially enlarged perspective view of the image sensor 211 provided in the camera 21 of the three-dimensional shape measuring device 1 of the first embodiment. In the case of a general image sensor, a color filter or a microlens is often provided on the pixel. On the other hand, in the case of the three-dimensional shape measuring device 1 of the first embodiment, as an on-chip filter of the image sensor 211, ND (Neutral Density) having a different dimming rate (transmittance) on each pixel 58. A filter is provided.

That is, each square shown in FIG. 21 indicates each pixel 58 of the image sensor 211 (an example of the light receiving element). As an example, the first ND filter 57OD1, the second ND filter 57OD2, the third ND filter 57OD3, and the fourth ND filter 57OD4 have different dimming rates (first dimming rate to first dimming rate to the first). It has a dimming rate of 4). With these four types of ND filters 57OD1 to ND filters 57OD4 as one set, the ND filter group 59 of each set is placed on the image sensor 211. It is arranged periodically. Each ND filter OD1 to ND filter OD4 is formed on the image pickup device 211 as a reflection type or absorption type filter by, for example, vapor deposition of a metal thin film.

In other words, on the image sensor 211, a filter unit having different dimming rate filter pieces (ND filter 57OD1 to ND filter 57OD4) is provided corresponding to each pixel 58. Each of the filter pieces (ND filter 57OD1 to ND filter 57OD4) is arranged so that the adjacent filter pieces (ND filter 57OD1 to ND filter 57OD4) have different dimming rates.

As an example, each filter piece (ND filter 57OD1 to ND filter 57OD4) has a rectangular shape. Then, four filter pieces (ND filter 57OD1 to ND filter 57OD4) adjacent to the adjacent filter piece on each of the two sides are set as one set of filter groups 59, and a plurality of sets of filter groups 59 are placed on the image sensor 211. It is arranged periodically in.

Further, as an example, the light dimming rate of the four ND filters 57OD1 to ND filter 57OD4 is "57OD1> 57OD2> 57OD3> 57OD4", and the light dimming rate of the first ND filter 57OD1. Is the highest, and the light dimming rate of each ND filter 57OD1 to ND filter 57OD4 is set so that the light dimming rate of the fourth ND filter 57OD4 is the lowest.

By taking a picture with the camera 21 provided with such an image sensor 211, overexposure or underexposure occurs from at least one ND filter among the ND filters 57OD1 to ND filter 57OD4 in one shooting. Information on no brightness value can be obtained. Therefore, even if the object to be measured has a non-uniform reflectance, it can be measured satisfactorily.

(Measurement operation of the first embodiment)
FIG. 22 is a flowchart showing the flow of the measurement operation of the measurement object in the three-dimensional shape measuring device 1 of the first embodiment. First, in steps S1 to S4, the sine wave fringe pattern of phase θ, the sine wave fringe pattern of phase θ + 1 / 2π, θ + π, and θ + 3 / 2π (see FIG. 18 as an example) are sequentially formed in the same manner as the conventional phase shift method. , Projected from an optical device 10 such as a projector, and photographed by the camera 21 (see FIG. 17).

Next, in step S5, the brightness value of the image pickup light is confirmed through the ND filter 57OD1 for all the addresses (each pixel 58) of the image pickup element 211. If there is no address (pixel 58) in which overexposure or underexposure occurs at this point, the process proceeds to step S6, and three-dimensional restoration processing is performed using the phase information of the brightness value of the imaging light via the ND filter 57OD1. Is performed, and the measurement is completed.

The presence or absence of overexposure and underexposure may be determined as no occurrence of overexposure and underexposure when all pixels 58 do not have overexposure and underexposure. Of these, when the number of pixels 58 in which overexposure and underexposure occur is less than a predetermined number, it may be determined that overexposure and underexposure do not occur.

In the imaging light passing through the ND filter 57OD1, if whiteout or blackout occurs at some addresses (pixels 58), the brightness value of the ND filter 57OD2 at the target address (target pixel) in step S7. To confirm. If overexposure or underexposure does not occur at this point, in step S8, three-dimensionally using the phase information of the luminance value of the imaging light through the ND filter 57OD1 and the luminance value of the imaging light via the ND filter 57OD2. Perform a restore. Hereinafter, three-dimensional restoration is performed using the data of OD3 and OD4 in the same manner.

Similarly, in the imaging light via the ND filter 57OD2, if whiteout or blackout occurs at some addresses (pixels 58), in step S9, the ND filter 57OD3 of the target address (target pixel) Check the brightness value of. If overexposure or underexposure does not occur at this point, in step S10, three-dimensional restoration is performed using the phase information of the brightness value of the imaging light via the ND filters 57OD1 to OD3.

On the other hand, in the imaging light via the ND filter 57OD3, if whiteout or blackout occurs at some addresses (pixels 58), the ND filters 57OD1 to 57OD4 are used in step S11. Three-dimensional restoration is performed using the phase information of the brightness value of the imaging light.

In this way, for example, by using four types of ND filters 57OD1 to 57OD4 having different dimming rates, a dynamic range of two digits or more can be obtained. Therefore, most of the measurement objects can be accurately measured in three dimensions.

(Effect of the first embodiment)
As is clear from the above description, in the three-dimensional measuring device 1 of the first embodiment, a predetermined projection pattern is projected from the optical device 10 onto the measurement object, and the reflected light is imaged by the camera 21. Acquire distance information of the object to be measured. At this time, ND filters 57OD1 to 57OD4 having different dimming rates are provided on the pixels 58 of the image sensor 211 of the camera 21. As a result, it is possible to acquire image data (image data having different exposure amounts) for filters having different dimming rates in one shooting. Therefore, it is possible to prevent an increase in the number of times of shooting and prevent the system from becoming large in size, and prevent overexposure or underexposure of the image data of the measurement object.

[Second Embodiment]
Next, a second embodiment in which the three-dimensional measuring device 1 described in the first embodiment described above is provided on the robot arm (articulated arm) will be described. FIG. 23 is a perspective view of the robot arm of the second embodiment. In FIG. 23, the robot arm 70 is provided with a hand unit 71 for picking an object, and the three-dimensional measuring device 1 described in the above-described first embodiment is provided in the immediate vicinity of the hand unit 71. .. The robot arm 70 includes a plurality of bendable movable portions, and changes the position and orientation of the hand portion 71 according to control.

The three-dimensional measuring device 1 is provided so that the projection direction of light coincides with the direction in which the hand unit 71 faces, and measures the picking target 15 of the hand unit 71 as a measurement object.

By providing the robot arm 70 with the three-dimensional measuring device 1 as in the third embodiment, the object to be picked can be three-dimensionally measured from a short distance, and the picking can be performed by a camera provided at a distance. It is possible to improve the measurement accuracy and the recognition accuracy as compared with the case where the object is imaged and three-dimensionally measured.

For example, in the FA (Factory Automation) field in various assembly lines of factories, robots such as a robot arm 70 are used for inspection and recognition of parts. By providing the robot with the three-dimensional measuring device 1, it is possible to accurately inspect and recognize parts.

[Third Embodiment]
Next, a third embodiment in which the three-dimensional measuring device 1 described in the above-described first embodiment is provided in an electronic device such as a smartphone or a PC will be described. FIG. 24 is a perspective view of the smartphone of the third embodiment provided with the three-dimensional measuring device 1. The smartphone 80 is provided with a three-dimensional measuring device 1 and a user authentication function. As a user authentication function, for example, dedicated hardware is provided. In addition, the CPU of the computer configuration may execute a program such as ROM to realize this function. The three-dimensional measuring device 1 measures the shape of the face, ears, head, and the like of the user 81. Although it is an example, the user 81 is authenticated based on the measurement result.

As described above, the smartphone of the third embodiment can measure the shape of the face, ears, head, etc. of the user 81 with high accuracy by the three-dimensional measuring device 1. Therefore, the recognition accuracy of the user 81 can be improved.

In this example, the three-dimensional measuring device 1 is provided in the smartphone 80, but it may be provided in another electronic device such as a personal computer device or a printer device. In addition to personal authentication, it may be used for scanning the face shape of the user 81.

[Fourth Embodiment]
Next, a fourth embodiment in which the three-dimensional measuring device 1 described in the above-described first embodiment is provided on a moving body such as an automobile will be described. FIG. 25 is a perspective view of a main part of the moving body of the fourth embodiment provided with the three-dimensional measuring device 1. In FIG. 25, the interior 85 of the automobile is provided with the three-dimensional measuring device 1 and the driving support function. The driving support function is realized by, for example, dedicated hardware. Alternatively, the driving support function is realized by software when the CPU of the computer configuration executes the driving support program stored in the ROM or the like.

The three-dimensional measuring device 1 measures the face or posture of the driver 86. Based on this measurement result, the driving support function provides appropriate support according to the situation of the driver 86, such as automatic braking control or traveling lane control.

In such a fourth embodiment, by providing the three-dimensional measuring device 1 in the automobile, the face, posture, etc. of the driver 86 can be measured with high accuracy, so that the state recognition accuracy of the driver 86 in the vehicle 85 is high. Can be improved.

In this example, the three-dimensional measuring device 1 is provided in an automobile, but the three-dimensional measuring device 1 can be used in a bus vehicle, a truck vehicle, a railroad vehicle, an airplane, a ship, or the like, a driver's seat, or a passenger seat. It may be provided in. Further, the three-dimensional measuring device 1 may be used not only for recognizing the state of the driver 86 such as the face and posture, but also for recognizing the state of a passenger other than the driver 86 or the inside of the vehicle 85. Further, the three-dimensional measuring device 1 may be used for the security of the moving body, such as performing personal authentication of the driver 86 and determining whether or not the driver has the proper authority to control the moving body. ..

For example, as shown in FIG. 26, the three-dimensional measuring device 1 may be provided on the autonomous moving body. The moving body 87 shown in FIG. 26 measures the circumference of the moving body 87 with the three-dimensional measuring device 1. Based on this measurement result, the moving body 87 determines its own moving route and calculates the layout of the room 89 such as the position of the desk 88.

By providing the three-dimensional measuring device 1 on the moving body 87 in this way, it is possible to measure the periphery of the moving body 87 with high accuracy, and it is possible to provide accurate driving support for the moving body 87. The three-dimensional measuring device 1 may be used not only indoors but also outdoors, and may be used for measuring a building or the like.

[Fifth Embodiment]
Next, a fifth embodiment in which the three-dimensional measuring device 1 described in the first embodiment described above is provided in a three-dimensional printer device which is an example of a modeling device will be described. FIG. 27 is a perspective view of a main part of the three-dimensional printer device according to the fifth embodiment. In FIG. 27, the above-mentioned three-dimensional measuring device 1 is provided in the head portion 91 of the three-dimensional printer device 90. The head portion 91 has a nozzle 93 that discharges a modeling liquid for forming the formation 92. The three-dimensional measuring device 1 measures the shape of the formation 92 formed by the three-dimensional printer device 90 during formation. Then, based on this measurement result, the formation of the formed product 92 of the three-dimensional printer device 90 is controlled.

In such a fifth embodiment, by providing the three-dimensional measuring device 1 in the three-dimensional printer device 90, the shape can be measured during the formation of the formed product 92, so that the formed product 92 can be measured with high accuracy. It can be formed. In this example, the three-dimensional measuring device 1 is provided in the head portion 91 of the three-dimensional printer device 90, but the three-dimensional measuring device 1 may be provided at another position in the three-dimensional printer device 90.

Finally, each of the above embodiments is presented as an example and is not intended to limit the scope of the invention. Each of the novel embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. In addition, each embodiment and modifications of each embodiment are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and the equivalent scope thereof.

1 3D measuring device 10 Optical device 12 Line generator 13 Light deflection element (mirror)
20 Measurement information acquisition unit 21 Camera 21
30 Control unit 31 Control unit 32 Arithmetic processing unit 57OD1 First dimming rate filter piece 57OD2 Second dimming filter piece 57OD3 Third dimming filter piece 57OD4 Fourth dimming rate Filter piece 58 pixels 59 Filter group 210 Lens 211 Image sensor

Japanese Unexamined Patent Publication No. 2013-157360

Improving the accuracy of active stereo by the phase shift method using the response function of the projector / camera system "Image Recognition and Understanding Symposium (MIRU2009)" July 2009 "Appearance inspection technique using three-dimensional image by optical cutting method" RICOH TECHNICAL REPORT, No. 39, 2013, published January 28, 2014

Claims (13)

A filter unit with multiple filter pieces with different light dimming rates,
An image pickup apparatus having a plurality of light receiving elements that receive each light transmitted through each of the filter pieces and output a light receiving signal corresponding to the information of the received light. The imaging device according to claim 1, wherein each of the filter pieces of the filter unit is arranged so that the adjacent filter pieces have different dimming rates. Each of the filter pieces of the filter unit has a rectangular shape, and four filter pieces adjacent to the adjacent filter piece on each of the two sides form a set of filter groups, and a plurality of sets of the filter groups are used. 2. The imaging apparatus according to claim 2, wherein the light receiving element is periodically arranged and formed. The imaging device according to any one of claims 1 to 3, wherein the light information is the amount of light. An irradiation unit that irradiates an object with light of a predetermined projection pattern,
The imaging device according to any one of claims 1 to 4,
A measuring device having a measuring unit for measuring depth information of the object by selectively using light receiving signals that have passed through filter pieces having different dimming rates in the imaging device. In the filter piece portion, the first filter piece portion and the second filter piece are set so that the dimming rate of the first filter piece portion is the highest and the dimming rate is set to be lower in order. A unit, a third filter piece, and a fourth filter piece are provided.
In the measuring unit, the level of the received signal of the first filter piece is less than the first threshold value which is the reference value of overexposure of the captured image, and the level of the blackout of the captured image is the reference value. The measuring device according to claim 5, wherein when the level is equal to or higher than the threshold value of 2, the depth information of the object is measured by using the received signal of the first filter piece. In the measuring unit, the level of the received signal of the first filter piece is at a level equal to or higher than the first threshold value which is the reference value of overexposure of the captured image, or is the reference value of blackout of the captured image. The level of the received signal of the second filter piece is less than the second threshold value and the level of the received signal of the second filter piece is less than the first threshold value which is the reference value of overexposure of the captured image and the captured image. Depth information of the object using the received signal of the first filter piece and the received signal of the second filter piece when the level is equal to or higher than the second threshold value which is the reference value of blackout. The measuring device according to claim 6, wherein the measuring device is characterized in that. In the measuring unit, the level of the received signal of the first filter piece portion and the second filter piece portion is at a level equal to or higher than the first threshold value which is the reference value of overexposure of the captured image, or the captured image. The level of the received signal of the third filter piece is less than the first threshold value, which is the reference value of overexposure of the captured image, and the level is less than the second threshold value, which is the reference value of blackout. When the level is equal to or higher than the second threshold value, which is the reference value for blackening of the captured image, the received signal of the first filter piece, the received signal of the second filter piece, and the light received signal of the second filter piece. The measuring device according to claim 7, wherein the depth information of the object is measured by using the received signal of the third filter piece. In the measuring unit, the level of the received signal of the first filter piece, the second filter piece, and the third filter piece is equal to or higher than the first threshold value, which is the reference value for overexposure of the captured image. If the level is lower than the second threshold value, which is the reference value for blackening of the captured image, the first filter piece, the second filter piece, and the third filter piece. The measuring device according to claim 8, wherein the depth information of the object is measured by using the received signal of the unit and the fourth filter piece. In the measuring unit, the level of the received signal of all the light receiving elements passed through each of the filter pieces is a level lower than the first threshold value which is the reference value of the overexposure of the captured image, and the captured image When the level is less than the second threshold value which is the reference value of blackout, or among the received signals of the light receiving element via each of the filter pieces, the first value which is the reference value of overexposure of the captured image. When the number of received signals that are below the threshold value of and below the second threshold value, which is the reference value for blackout of the captured image, is less than a predetermined number, it is determined that there is no overexposure or no blackout. The measuring device according to any one of claims 5 to 9, wherein the measuring device is characterized by the above. The measuring device according to any one of claims 5 to 10, wherein the irradiation unit is a surface emitting semiconductor laser having a plurality of oscillation wavelengths. The measuring device according to any one of claims 5 to 11, wherein the irradiation unit includes a projection pattern forming unit that forms the projection pattern having a plurality of different wavelengths. The imaging apparatus according to any one of claims 1 to 4 is used.
The object is irradiated with light of a predetermined projection pattern,
A measurement method for measuring depth information of an object by selectively using light-receiving signals that pass through filter pieces having different dimming rates in the imaging device.