JP2020085618A - Measuring device and method for measuring optical characteristics of object interior - Google Patents

Abstract

To provide a measuring device with which it is possible to measure the optical characteristics of an object interior without changing the arrangement and focal distance of each part of the device.SOLUTION: The measuring device comprises a projector 20, an imaging device 30, a pattern image generation unit 40, and a data processing unit 50. The pattern image generation unit 40 generates the image data of a plurality of pattern images respectively corresponding to a plurality of special frequencies differing in the number of sine functions, the pattern images having a cyclical change of optical intensity represented by sine functions in one direction in a first plane crossing an object 60, and inputs the generated pattern image to the projector, and causes the plurality of pattern images differing in spatial frequency to be respectively projected to the object by the projector while sequentially changing the spatial frequency in time. The data processing unit 50 extracts the signal amplitude of light reflected from a specific three-dimensional position on the surface and inside of the object on the basis of the digital image signal of a plurality of composite images outputted from the imaging device, corresponding to the plurality of spatial frequencies and sequentially changing in time and calculates the optical characteristics of the object interior.SELECTED DRAWING: Figure 1

Description

The present invention relates to a measuring apparatus and method for measuring optical characteristics in a three-dimensional space inside an object, and more particularly to a measuring apparatus and method using a projector/camera system.

Although the measurement inside an object is a difficult task, it is important in various applications such as bioimaging, medical imaging, and industrial inspection. Various imaging techniques such as fluorescent X-ray analysis technique, terahertz imaging, and infrared reflectography that enable clear observation of the inside of an object have been developed. However, these techniques, which utilize various characteristics of electromagnetic waves, require expensive equipment that is difficult to obtain. In contrast, projector-camera systems are relatively low cost and readily available.

As a measuring apparatus and method for measuring optical characteristics in a three-dimensional space inside an object using a projector/camera system, a synthetic aperture technology and a confocal imaging technology, which are one of radar technology, are applied to block the object. A first conventional technique (see Non-Patent Document 1 below) that enables imaging of a different location, by physically changing the focal length of the projector and projecting a spatially high frequency periodic pattern Second conventional technique (see Non-Patent Document 2 below) that restores the global illumination image in the shape and target scene, and when a periodic pattern of different spatial frequencies is projected multiple times using a parallel projector-camera system. There is a third conventional technique (see Non-Patent Document 3 below) in which an image inside an object is restored from changes in subsurface scattering components.

Marc Levoy, Billy Chen, Vaibhav Vaish, Mark Horowitz, Ian McDowall, and Mark Bolas. 2004. Synthetic aperture confocal imaging. In ACM Transactions on Graphics (ToG), Vol. 23. ACM, 825-834. Supreeth Achar and Srinivasa G Narasimhan. 2014. Multi focus structured light for recovering scene shape and global illumination. In European Conference on Computer Vision. Springer, 205-219. Kenichiro Tanaka, Yashiro Mukaigawa, Hiroyuki Kibo, Yasukiki Matsushita, and Yasushi Yagi. 2017. Recovering Inner Slices of Layered Translucent Objects by Multi-Frequency Illumination. IEEE transactions on pattern analysis and machine intelligence 39, 4 (2017), 746-757.

The first conventional technique has an advantage of being able to reduce the influence of the shield between the target object and the system, but in order to apply the synthetic aperture technique, the system is physically moved or a similar system is used. However, there is a problem that the system configuration becomes complicated or large in size.

The second conventional technique can be interpreted as performing spatiotemporal frequency modulation by physically changing the focal length of the projector and is similar to the content of the invention of the present application to be described later, but the focus of the projector is physically In order to change the distance, it has to be done manually or by using a special mechanical structure. Therefore, there is a problem that the measurement process becomes complicated when the focal length is changed manually, and the system configuration becomes complicated when a special mechanical structure is used.

The third conventional technique is similar to the content of the present invention described later in that it projects a periodic pattern of different spatial frequencies, but requires a parallel optical system, and the target object is composed of a small number of discrete layers. Since the optimization calculation is applied by assuming that it becomes, there is a problem that it cannot cope with a more complicated internal structure.

The present invention has been made in view of the above problems, and an object thereof is to complicate without moving the projector and the camera at the time of measurement and physically changing the focal lengths of the projector and the camera. An object of the present invention is to provide a measuring device and method capable of measuring optical characteristics in a three-dimensional space inside an object, which can be applied to various internal structures.

A measuring device according to the present invention is a measuring device for measuring optical characteristics inside an object,
A projector that projects a pattern image toward the object using light that can pass through the object;
Of the light of the pattern image projected from the projector, the light reflected at each of the three-dimensional positions on the surface and the inside of the object is incident on the imaging surface where the light can be sensed and is imaged to be combined. An imaging device for converting the synthesized image thus converted into a digital image signal and outputting the digital image signal;
Image data of the pattern image having a periodic amplitude change of a light intensity represented by a sine function in at least one direction in a predetermined first plane across the object, the plurality of spaces having different sine functions. By generating the image data of the plurality of pattern images respectively corresponding to frequencies and inputting the image data to the projector, the plurality of pattern images having different spatial frequencies from each other with respect to the projector, A pattern image generation unit for projecting toward the object separately while sequentially changing in time,
Input the plurality of spatial frequencies in the image data generated by the pattern image generation unit and the digital image signals of the plurality of composite images respectively corresponding to the plurality of spatial frequencies output from the imaging device. As a result, a predetermined correlation calculation process is performed on these inputs to extract the signal amplitude of the light reflected from a specific three-dimensional position on the surface and inside of the object, and the signal amplitude of each extracted three-dimensional position. A data processing unit that calculates the optical characteristics at each of the three-dimensional positions on the surface and inside of the object based on
The first feature is that the optical systems of the projector and the imaging device have different focal lengths.

A measuring method according to the present invention is a measuring method for measuring an optical characteristic inside an object,
A projector that projects a pattern image toward the object by using light that can pass through the object, and a light reflected by the surface of the object and three-dimensional positions inside the light of the pattern image projected from the projector. By using an image pickup device that converts the light into a digital image signal and outputs the combined image by combining the light into an image pickup surface capable of sensing the light and forming an image.
Image data of the pattern image having a periodic amplitude change of a light intensity represented by a sine function in at least one direction in a predetermined first plane across the object, the plurality of spaces having different sine functions. By generating the image data of the plurality of pattern images respectively corresponding to frequencies and inputting the image data to the projector, the plurality of pattern images having different spatial frequencies from each other with respect to the projector, A pattern image generating step of projecting toward the object separately while sequentially changing in time;
Input the plurality of spatial frequencies in the image data generated in the pattern image generation step, and the digital image signals of the plurality of composite images respectively corresponding to the plurality of spatial frequencies output from the imaging device. As a result, a predetermined correlation calculation process is performed on these inputs to extract the signal amplitude of the light reflected from a specific three-dimensional position on the surface and inside of the object, and the signal amplitude of each extracted three-dimensional position. A data processing step of calculating the optical characteristic at each of the three-dimensional positions on the surface and inside of the object based on
The first feature is that the optical systems of the projector and the imaging device have different focal lengths.

According to the measurement device and method of the first feature, as will be described later, the measurement target object is not moved without moving the projector and the camera at the time of measurement, and is not physically changed the focal lengths of the projector and the camera. Even if the internal structure of is complex, the optical characteristics in the three-dimensional space inside the object can be measured.

Further, in the measuring apparatus and method of the first feature, the first plane is orthogonal to the optical axis of the optical system of the projector, or the first plane is parallel to the imaging plane. Is a preferred embodiment.

Further, in the measuring device and method of the first feature, it is more preferable that the optical axes of the optical systems of the projector and the imaging device are parallel to each other and are orthogonal to the first plane. It is a mode.

Further, in the measuring device and method of the first feature, it is a preferable embodiment that the projector is a perspective projection projector and the imaging device is a parallel projection camera.

According to these preferred embodiments, as will be described later, a process of generating image data of a pattern image to be input to the projector and a signal amplitude of light reflected from a specific three-dimensional position on the surface and inside of the object are calculated. It is possible to simplify and improve the accuracy of the process of extracting and calculating the optical characteristics at each position on the surface and inside of the object.

Further, in the measuring device according to the present invention, in addition to the first feature, the data processing unit outputs a plurality of light intensities individually corresponding to the spatial frequencies of the digital image signals that sequentially change. , A first data string obtained by multiplying a sine function value of an angular frequency corresponding to each spatial frequency separately, and a plurality of light intensities, a cosine function value of the angular frequency corresponding to each spatial frequency A second feature is that the signal amplitude is extracted based on the respective calculation results obtained by individually performing the discrete Fourier transform on the second data sequence obtained by multiplying separately.

Further, in addition to the first feature, the measuring method according to the present invention, in the data processing step, provides a plurality of light intensities individually corresponding to the spatial frequencies of the digital image signals that sequentially change. , A first data string obtained by multiplying a sine function value of an angular frequency corresponding to each spatial frequency separately, and a plurality of light intensities, a cosine function value of the angular frequency corresponding to each spatial frequency A second feature is that the signal amplitude is extracted based on the respective calculation results obtained by individually performing the discrete Fourier transform on the second data sequence obtained by multiplying separately.

Further, in the measuring apparatus and method of the first or second feature, it is preferable that the periodic amplitude change of the light intensity of the pattern image is a change in two orthogonal directions in the first plane. This is one embodiment.

According to the preferred embodiment, as will be described later, as compared with the case where the periodic amplitude change of the light intensity of the pattern image is in only one direction, the error in the calculated optical characteristic is reduced and the optical characteristic is calculated. The accuracy of the processing can be improved.

Further, in the measuring device and method of the first or second feature, it is preferable that the optical characteristic is a reflectance at each of three-dimensional positions on the surface and inside of the object.

According to the measuring device and method according to the present invention, it is possible to cope with a complicated internal structure without moving the projector and the camera at the time of measurement, and without physically changing the focal lengths of the projector and the camera. A measuring device and method for measuring optical characteristics in a three-dimensional space inside an object can be provided.

The figure which shows typically the example of a schematic structure of the projector camera system which is a measuring device which concerns on this invention. FIG. 2 is a diagram schematically showing a schematic configuration example in which a projector, an imaging device, and a measurement target of the projector/camera system shown in FIG. 1 are suitably arranged. The figure which shows the structure of the projector camera system used for the evaluation experiment. The figure which shows typically the schematic structure of the object of the measurement object used for the evaluation experiment. The figure which shows the calculation result of the internal reflectance of four layers restored by the evaluation experiment.

Hereinafter, embodiments of a measuring apparatus and a method (hereinafter, appropriately referred to as “present apparatus” and “present method”) according to the present invention will be described with reference to the drawings.

[Schematic configuration of measuring device]
As schematically shown in FIG. 1, the present device 10 is a measuring device that measures optical characteristics in a three-dimensional space inside an object 60 to be measured, and includes a projector 20, an imaging device 30, and a pattern image generating unit 40. , And a data processing unit 50, and configured as a projector/camera system.

The existing projector/camera system is generally used to measure the three-dimensional shape of the measurement target. However, in the present invention, the projector/camera system is used not only on the surface of the measurement target but also inside the object. It is applied to the measurement of optical characteristics in dimensional space.

Based on the image data D1 input from the pattern image generation unit 40, the projector 20 changes the periodic amplitude of the light intensity represented by a sine function in at least one direction within a predetermined first plane that crosses the object 60. It is configured to generate a pattern image having the pattern image and optically project the pattern image toward the object 60 using light that can pass through the object 60. In the present specification, “optically” means to use optical equipment (appropriately referred to as “optical system”) including an optical element such as an optical lens.

In this embodiment, it is assumed that a general perspective projection projector is used as the projector 20. Therefore, when the position of the first plane recedes in the direction perpendicular to the first plane, that is, when it moves away from the projector 20, the pattern image projected on the first plane is enlarged in a similar manner.

When the three-dimensional object 60 is transparent or semi-transparent to visible light such as white light, and the reflectance and the transmittance of the visible light inside the object are three-dimensionally spatially changed. In the above, the projector 20 projects the pattern image toward the object 60 using the visible light.

In the projector 20, the arrangement and the focal length of the optical system of the projector 20 are preset so that the pattern image is projected on the measurement target region of the object 60, and the set arrangement and the focal length are maintained throughout the measurement period. It The projection system and the projection mechanism of the projector 20 are not limited to a particular system and the like as long as they are suitable for the wavelength of the light to be projected, and a commercially available perspective projection projector can be used. it can. The electrical and optical specifications (for example, frame rate, focal length, resolution, etc.) of the projector 20 may be appropriately selected depending on the measurement content, and are not limited to specific specifications. The specific structure of the projector 20 is not the gist of the present invention, and thus the detailed description is omitted.

The imaging device 30 includes an image sensor (CCD sensor, MOS sensor, etc.) in which photoelectric conversion elements capable of sensing the light projected from the projector 20 are arranged in an array, and the surface and the inside of the object 60 are three-dimensional. An optical system for forming an image on the image pickup surface, which is the light reflected by the position and incident on the image pickup surface that is the surface of the image sensor, and amplifies the photoelectrically converted electric signal output from the image sensor, etc. Then, a signal processing circuit section for outputting as a digital image signal D2 of a predetermined output format is provided. When the light projected from the projector 20 is visible light such as white light, for example, a configuration including a commercially available RGB camera is assumed as the imaging device 30. The image sensor to be used may be a monochrome sensor other than the color sensor depending on the measurement content.

The image pickup apparatus 30 is preset with the arrangement of the image pickup apparatus 30 and the focal length of the optical system so that the light incident from the measurement target region of the object 60 can be taken, and the set arrangement and the focal length are maintained throughout the measurement period. .. The electrical and optical specifications (for example, frame rate, focal length, resolution, etc.) of the image pickup apparatus 30 may be appropriately selected according to the measurement content, and are not limited to specific specifications. .. Since the specific structure of the image pickup device 30 is not the gist of the present invention, detailed description thereof will be omitted.

The pattern image generation unit 40 generates image data D1 of a pattern image having a periodic amplitude change of light intensity represented by a sine function in at least one direction within the first plane. Here, the spatial frequency on the first plane is set so that the spatial frequency representing a periodic amplitude change in at least one direction on one pattern image projected on the first plane sequentially changes in time. Image data D1 corresponding to a plurality of different pattern images are created. As for the created image data D1, it is preferable to sequentially transmit the image data D1 corresponding to each spatial frequency to the projector 20 for each spatial frequency. All the image data D1 corresponding to each spatial frequency may be collectively transmitted to the projector 20, but in this case, it is necessary for the projector 20 side to temporarily store the received image data D1. Since the memory capacity is significantly increased, it is preferable to sequentially transmit the image data D1 corresponding to each spatial frequency separately in order to reduce the burden on the resource amount and the processing amount on the projector 20 side.

In this embodiment, image data D1 in which a periodic amplitude change along at least one direction (x direction) on the pattern image is represented by a sine function in which x is a variable is created. That is, the intensity L(x, y, z) of the light with which the three-dimensional position (x, y, z) on the first plane is irradiated is represented by the following formula 1. As shown in Expression 1, the intensity L(x, y, z) is uniform in the y direction. In the xyz orthogonal coordinate system that defines the three-dimensional position (x, y, z), the first plane is a plane parallel to the xy plane, and the z axis is an axis perpendicular to the first plane. Further, the first plane can take an arbitrary z value within the range across (including contact with) the object 60.

In Expression 1, D(z) indicates an offset value, A(z) indicates an amplitude, ω(z) indicates an angular frequency, and φ indicates a phase.

As described above, in the present embodiment, since the projector 20 is a perspective projection projector, when the first plane recedes in the +z direction perpendicular to the first plane, the pattern image projected on the first plane is The spatial frequency f(z) (=ω(z)/2π) on the first plane is reduced in a similar manner as a result. That is, one pattern image projected from the projector 20 has a spatial frequency f(z) on the first plane depending on the position (z value) where the first plane onto which the pattern image is projected crosses the object 60. Increase or decrease. On the other hand, as described above, even if the position (z value) where the first plane crosses the object 60 is the same, the spatial frequency f(z) changes sequentially in time. That is, the spatial frequency f(z) of the pattern image observed on the first plane is spatially (z direction) and temporally determined by the image data D1 generated by the pattern image generation unit 40 and input to the projector 20. Is modulated.

In the following description, the spatial frequency f(z) and the angular frequency ω(z) that sequentially change with time use the index i (i=1 to n) that defines the order of the sequentially changing time. Therefore, they are appropriately expressed as f(z)[i] and ω(z)[i]. n is the total number of spatial frequencies f(z) that sequentially change with time and is preset according to the content of measurement, but in the present embodiment, as an example, it is set within the range of about 100 to 1000. .. Further, a period during which the pattern image having the spatial frequency f(z)[i] is projected is referred to as a time point [i]. Therefore, at the time point [i], only the pattern image with the spatial frequency f(z)[i] is the pattern image with the spatial frequency f(z)[i-1] one time before and the spatial frequency after the one time point. It is projected without overlapping with any of the pattern images of f(z)[i+1]. That is, within each period [i], the pattern image of the spatial frequency f(z)[i] is projected as a still image.

On the other hand, the image formed on the imaging surface of the imaging device 30 at a specific time point [i] is up to the three-dimensional positions (x, y, z) on the surface and inside of the object 60, in front of (-). Part of the light that has reached through the object 60 existing in the z direction) is reflected according to the reflectance at the three-dimensional position (x, y, z), and the other part that is not reflected is reflected. , Light that is transmitted rearward (+z direction) and reflected at the three-dimensional position (x, y, z+Δz) at the rear in accordance with the reflectance at the three-dimensional position is obtained by combining them all. It becomes a composite image. The time length of each time point [i] is set to be longer than the exposure time of the combined image on the imaging surface, and is determined depending on the specifications of the projector 20 and the imaging device 30 used. Therefore, on the imaging device 30 side, during each period [i], a composite image corresponding to only the pattern image of the spatial frequency f(z)[i] projected as a still image is formed on the imaging surface. , The digital image signal D2 corresponding to the composite image at each time point [i] is discretely output in synchronization with each time point [i]. Here, at each time point [i], the pattern image is continuously projected as a still image for a certain period, so that the synchronization between the projector 20 and the imaging device 30 can be easily established.

As described above, in the composite image represented by the digital image signal D2 output from the imaging device 30 at a specific time point [i], the position on the z-axis at the specific time point [i] is on the first plane. The information of the amplitude of the light intensity of the pattern image and the reflectance and the transmittance of the object 60 at the two-dimensional position (x, y) is included in the z direction in a superimposed manner.

The pattern image generation unit 40 is, for example, on a general-purpose computer such as a personal computer, a processing (corresponding to the pattern image generation step of the present method) computer program for generating the image data D1 of the pattern image at each time [i]. Can be configured as a computer software means for executing.

The data processing unit 50 includes a plurality of spatial frequencies f(z)[i] (i=1 to n) in the image data D1 generated by the pattern image generating unit 40, and a plurality of spaces output from the imaging device 30. The digital image signals D2 of a plurality of composite images corresponding to the frequencies f(z)[i] are input, and a correlation calculation process described later is performed on these inputs to identify the surface and the inside of the object 60. The signal amplitude of the light reflected from the three-dimensional position (x, y, z) is extracted, and based on the extracted signal amplitude of each three-dimensional position (x, y, z), the object 60 of the object 60 is extracted as described later. The optical characteristics at each three-dimensional position (x, y, z) on the surface and inside are calculated. In this embodiment, the reflectance R(x, y, z) is calculated as the optical characteristic.

The data processing unit 50, for example, on a general-purpose computer such as a personal computer, a process of extracting a signal amplitude of light reflected from a specific three-dimensional position (x, y, z) on the surface and inside of the object 60, and A process of calculating the optical characteristics at each of the three-dimensional positions (x, y, z) on the surface and inside of the object 60 based on the extracted signal amplitudes of the three-dimensional positions (x, y, z) (both processes are It can be configured as computer software means for executing a computer program for performing the data processing step of the method).

The pattern image generation unit 40 and the data processing unit 50 can each be configured as computer software means. However, the computer as hardware that executes each computer program can be one or two general-purpose computers. The hardware is not limited to this, and may be configured as a single dedicated hardware, and the dedicated hardware of the pattern image generation unit 40 and the data processing unit 50 may be individually provided.

Next, the composite image observed on the imaging surface of the imaging device 30 and the object 60 at an arbitrary three-dimensional position (x, y, z) on an arbitrary first plane from the digital image signal D2 of the composite image. The procedure for calculating the reflectance (x, y, z) will be described.

2, in the projector/camera system, the projector 20 is a perspective projection projector, the imaging device 30 is a parallel projection camera, and each of the projector 20 and the imaging device 30 is illustrated to facilitate description. It is assumed that the projector 20 and the imaging device 30 are preferably arranged such that the optical axis of the optical system is orthogonal to the first plane. That is, the projection surface of the projector 20, the imaging surface of the imaging device 30, and the first plane are parallel to each other.

In the system configuration shown in FIG. 2, the beam splitter (half mirror) 21 is interposed between the projector 20 and the object 60 so that the projector 20 does not block the space between the object 60 and the imaging device 30. The optical axis of the optical system is bent. Thereby, the optical axes of the optical systems of the projector 20 and the imaging device 30 can be made coaxial, and the origin of each xy coordinate of the projection surface of the projector 20, the imaging surface of the imaging device 30, and the first plane (x=0, y= 0) can be matched. Here, the projection center of the projector 20 is defined as z=0. In an actual system configuration (for example, an experimental system configuration shown in FIG. 3 to be described later), the beam splitter is not always necessary, and the optical axes of the optical systems of the projector 20 and the imaging device 30 are not coaxial but parallel to each other. If set, the relative relationship between the coordinate system on the projection surface of the projector 20 and the coordinate system on the imaging surface of the imaging device 30 is only parallel movement, and thus the same calculation as the calculation content described below can be performed. is there. In the following description, when the distance on the optical axis (z axis) from z=0 is referred to as “depth”, the z value may be referred to as depth z.

Further, since the image pickup device 30 is configured by a parallel projection camera, the pattern image on the first plane has the same size, that is, it is formed on the image pickup surface without being enlarged or reduced.

When the projector 20 is a perspective projection projector and the imaging device 30 is a parallel projection camera, the focal lengths of the optical systems are not equal to each other. In this device and this method, it is a necessary condition that the focal lengths of the optical systems of the projector 20 and the imaging device 30 are not equal. Therefore, the configuration of the projector/camera system shown in FIG. 2 is an embodiment that satisfies the condition. In an actual embodiment, as a configuration of the projector/camera system, the projector 20 may be a perspective projection projector and the imaging device 30 may be a perspective projection camera. Further, the projector 20 is a parallel projection projector and the imaging device 30 may be a perspective projection camera.

[Composite image observed on the imaging surface of the imaging device]
First, a composite image observed on the imaging surface of the imaging device 30 will be described. As described above, the pattern image projected on the first plane is represented by the sine function having x as a variable as shown in the above mathematical expression 1, and the angular frequency ω(z) decreases as z increases. Here, if the z value of the first plane at the forefront of the object 60 is z ₁ and the angular frequency ω(z) of the pattern image on the first plane (z=z ₁ ) is ω ₁ , then arbitrary z The angular frequency ω(z) on the first plane at is expressed by the following equation 2. That is, by using a perspective projection projector as the projector 20, a natural spatial frequency modulation in the z direction can be obtained for the pattern image.

The light intensity I(x, y) of the composite image observed at the pixel (x, y) on the imaging surface is expressed by the following mathematical expression 3 as an integrated value of all the light intensities at different depths z. .. Here, it is assumed that nothing exists between the imaging device 30 and the frontmost surface (z=z ₁ ) of the object 60.

Each three-dimensional position (x, y, z) is illuminated with the pattern image of the light intensity L(x, y, z) shown by the above formula 1, and the reflectance R at each three-dimensional position (x, y, z). When the light reflected by (x, y, z) passes through the object 60 before reaching the pixel (x, y) on the imaging surface, the light is attenuated. The intensity I _p (x, y, z) of the light that reaches the pixel (x, y) from the dimension position (x, y, z) is represented by the following Expression 4.

The E(x, y, z) of the equation 4 is the attenuation rate of the light intensity until the light reflected at the three-dimensional position (x, y, z) reaches the pixel (x, y), and the three-dimensional position. When the light transmittance at (x, y, z) is T(x, y, z), it can be formulated as the following Expression 5.

Here, as described above, since it is assumed that nothing exists between the imaging device 30 and the forefront (z=z ₁ ) of the object 60, the beam splitter 21 described above is a virtual one. Then, it is assumed that the light incident from the object 60 toward the imaging surface is not reflected by the beam splitter 21 and is 100% transmitted. However, if the beam splitter 21 is physically present and the light transmittance of the beam splitter 21 is not 100%, the right side of the above equation 5 may be multiplied by a certain transmittance.

Ideally, the offset value D(z) and the amplitude A(z) of the light intensity L(x, y, z) of the pattern image irradiated on the first plane with an arbitrary depth z are set to the depth z. On the other hand, it is homogeneous, but is actually affected by the object 60. For example, the irradiation to the depth z ₂ (z ₂ >z ₁ ) may be attenuated from each point of the depth z ₁ , resulting in a non-uniform offset value D(x, y, z ) And the amplitude A(x, y, z).

The offset value D(z ₁ ) and the amplitude A(z ₁ ) at the depth z ₁ are uniquely determined by the image data D1 generated by the pattern image generation unit 40. The offset value D(x,y,z) and the amplitude A(x,y,z) at the depth z (z>z ₁ ) are the offset value D(z ₁ ) and the amplitude A(z at the depth z ₁ . ₁ ) is multiplied by the attenuation rate G(x, y, z) of light from the depth z ₁ to the three-dimensional position (x, y, z), and is expressed by the following formulas 6 and 7. Further, the attenuation rate G(x, y, z) can be formulated as the following Expression 8. However, the variable x ^* and the variable y ^* in the right side of the equation 8 are given by the equation 9.

From the above, the light intensity I(x, y) of the combined image observed at the pixel (x, y) on the image pickup surface is modified from Equation 3 and expressed by Equation 10 below. However, the offset value D ^* (x, y, z) and the amplitude A ^* (x, y, z) on the right side of Expression 10 are given by Expressions 11 and 12 below.

[Calculation procedure of reflectance (x, y, z) (data processing step)]
Next, at an arbitrary three-dimensional position (x, y, z) on an arbitrary first plane from the digital image signal D2 indicating the light intensity I(x, y) of the composite image on the image pickup surface given by Expression 10. A procedure for calculating the reflectance (x, y, z) of the object 60 will be described.

The light intensity I(x, y) is spatially modulated by the spatial frequency f(z) (=ω(z)/2π) depending on the depth z, as shown in Expression 10. Therefore, a signal having a specific spatial frequency, that is, a signal having a specific depth z can be extracted by the spatial correlation processing. However, the spatial correlation process is affected by the spatial frequency of the reflectance map. The inventor of the present application adopts a direct conversion technique (homodyne detection) in order to solve this problem. The direct conversion makes it possible to detect a signal having a specific frequency in a complex signal.

Here, in the present device and the present method, it is a necessary condition that the focal lengths of the optical systems of the projector 20 and the imaging device 30 are not equal. However, if the condition is not satisfied, the first crossing of the object 60 is performed. On one plane, spatial modulation by the angular frequency ω(z) represented by the above mathematical expression 2 causes the exact reverse spatial demodulation in the process of forming a composite image on the imaging surface, so that the light intensity Since I(x, y) is not spatially modulated, the reflectance (x, y, z) cannot be calculated by direct conversion as described later.

By the way, as described above, the pattern image generation unit 40 sequentially changes the spatial frequency f(z) (=ω(z)/2π) temporally, and at the same time, a plurality of pattern images having different spatial frequencies f(z) are generated. And the projector 20 sequentially projects a plurality of pattern images having different spatial frequencies f(z) toward the object 60 based on the received image data D1. Therefore, when the light intensity I(x, y) of the combined image observed on the imaging surface at time [i] is I(x, y) [i], the imaging device 30 causes the projector 20 to generate a plurality of pattern images. , Which is obtained by photoelectrically converting the light intensity I(x, y) [i] of the composite image projected at each time [i] and observed on the imaging surface in synchronization with the time [i] at which the images are sequentially projected. The image signal D2 is output.

Therefore, the light intensity I(x, y) [i] of the combined image at the time [i] obtained by the data processing unit 50 from the digital image signal D2 is represented by the following Expression 13.

By multiplying both sides of the above Expression 13 by a sine wave signal having an angular frequency of ω(w)[i], the following Expression 14 is obtained.

All terms in the integral on the right-hand side of Equation 14 above depend on the index i that defines the order of time. Here, only in the case of z=w, that is, in the case of ω(z)[i]−ω(w)[i]=0, the third term in the integral on the right-hand side of Equation 14 is the index It does not depend on i, and becomes a direct current (DC) component of the continuous signal I(x,y)[i]sin(ω(w)[i]x) of the above-mentioned equation 14. The DC component of the continuous signal can be easily extracted by performing a well-known discrete Fourier transform (DFT). When the operator of the discrete Fourier transform is defined as <F{s[i]}> _DC , the third term in the integral on the right-hand side of Expression 14 above is extracted as shown in Expression 15 below.

Similarly, when both sides of Expression 13 are multiplied by a cosine wave signal having an angular frequency of ω(w)[i] and the discrete Fourier transform (DFT) is performed, Expression 16 below is obtained.

Since cos ² φ+sin ² φ=1, the following Equation 17 is derived from Equation 15 and Equation 16.

The reflectance R(x, y, z) of the object 60 at an arbitrary three-dimensional position (x, y, z) which is the calculation target of this section is given by the following Equation 18 by modifying the above Equation 12. ..

As described above, since it is assumed that nothing exists in front of the depth z ₁ (z<z ₁ ), the denominator E(x, y, z) of the right-hand side of Equation 18 is at the depth z ₁ . , E(x, y, z ₁ )=1. Furthermore, the amplitude A(x,y,z ₁ ) at depth z ₁ is known as A(z ₁ ). From the above, the reflectance R(x, y, z ₁ ) of the object 60 at the depth z ₁ is represented by the following Expression 19.

Next, it is assumed that the transmittance T(x, y, z) of the object 60 at an arbitrary three-dimensional position (x, y, z) is given by the following Expression 20.

Here, the depth of the back slightly from the depth _{z 1 z 2 (z 2 =} z 1 + Δz) attenuation in _{E (x, y, z 2} ) and amplitude _{A (x, y, z 2} ) , respectively , And can be calculated by Equations 5 and 7. Therefore, the reflectance R(x, y, z ₂ ) of the object 60 at the depth z ₂ can be calculated based on Equation 18. Repeating the above procedure, the reflectivity R of the object 60 at the depth _{z 1} from the rear of any depth _{z (z> z 1) (} x, y, z) can be calculated.

[Evaluation experiment]
Next, the results of evaluating the present apparatus and the present method using the experimental projector-camera system shown in FIG. 3 will be described. In the above system, a laser projector 202 (model number: LB-UH6CB, resolution: 1280×720) manufactured by SK Telecom Co., Ltd. is used as the projector 20, and a telecentric lens 31 (product by Edmond Optics Co., Ltd. An RGB camera 32 (trade name: Grasshopper 3, resolution: 1920×1440) manufactured by FLIR Co. equipped with a code: #55-348) was used. In addition, in order to prevent specular reflection from the object 60 that obstructs the observation of the projection pattern from entering the imaging surface of the imaging device 30, a polarizer is provided on each front surface (on the object 60 side) of the laser projector 202 and the telecentric lens 31. 23 and 33 are installed. In the experimental system setting shown in FIG. 3, the beam splitter 21 assumed in the explanatory system configuration of FIG. 2 is not used. Therefore, the optical axes of the optical systems of the projector 20 and the imaging device 30 are parallel to each other, but not coaxial. Further, in the system setting, the depth z ₁ is 252.83 mm, and the offset value and the amplitude at the depth z ₁ are set to D(z ₁ )=A(z ₁ )=127. Further, the spatial frequency of the sine wave pattern at the depth z ₁ was sequentially changed in 0.78125 mHz steps within the range of 0.78125≦f(z)≦500 mHz (Hz=1/m). The total number n of the indexes i is 600.

The laser projector 202 used is not completely focus free and we have determined that a focus range of at least 250 mm is required.

Further, the used telecentric lens 31 has practical limitations, has a shallow depth of field, and has a limited working distance. In the set imaging device 30, the depth of field is 60 mm and the working distance at the near end is 130 mm.

As the object 60 to be measured, as shown in FIG. 4, four overhead projector (OHP) sheets 61 to 64 are stacked at 10 mm intervals in the front-rear direction (z direction). Characters A, B, C, and D are printed on the sheets 61 to 64 in order from the front.

5A to 5D show the calculation results of the internal reflectances when the depth in the +z direction from the front surface is 0 mm, 10 mm, 20 mm, and 30 mm. In FIG. 5(A) (depth: 0 mm), the character A can be confirmed more clearly than the other characters B to D. In FIG. 5B (depth: 10 mm) and FIG. 5(C) (depth: 20 mm), the letters B and C can be clearly confirmed as compared with other letters. In FIG. 5D (depth: 30 mm), both the characters C and D can be confirmed, but the characters A and B can hardly be confirmed.

The experimental results shown in FIGS. 5A to 5D sufficiently show that the present apparatus and method can restore the reflectance inside various objects.

[Another embodiment]
Next, a modified example (another embodiment) of the above embodiment will be described.

<1> In the above embodiment, the optical axes of the optical systems of the projector 20 and the imaging device 30 are coaxial, and the optical axes are orthogonal to the first plane. Therefore, the first plane is the projection surface of the projector 20 and the imaging plane. It is parallel to both imaging planes of device 30. Therefore, the pattern image projected on the first plane and having the periodic amplitude change of the light intensity represented by the sine function in the x direction is represented by the sine function in the same x direction on the projection surface of the projector 20. The component corresponding to the pattern image in the combined image formed on the image pickup surface of the image pickup apparatus 30 is represented by a sine function in the same x direction. It has a periodic amplitude change of the light intensity.

Therefore, between the orthogonal coordinate system on the projector 20 side in which the xy plane is parallel to the projection plane and the optical axis of the optical system of the projector 20 is the z axis, and the orthogonal coordinate system that defines the three-dimensional position on the first plane. There is no need to change the coordinates, and the pattern image generation unit 40 uses the same coordinate system to generate the image data D1 of the pattern image having the periodic amplitude change of the light intensity represented by the sine function in the direction on the first plane. Can be created.

Further, between the orthogonal coordinate system on the side of the imaging device 30 in which the xy plane is parallel to the imaging surface and the optical axis of the optical system of the imaging device 30 is the z axis, and the orthogonal coordinate system that defines the three-dimensional position on the first plane. Since it is not necessary to change the coordinates at, the digital image signal D2 indicating the light intensity I(x, y) of the composite image on the image pickup surface given by the equation 10 is used to determine an arbitrary three-dimensional position on the arbitrary first plane ( In the process of calculating the reflectance (x, y, z) of the object 60 at (x, y, z), coordinate conversion is performed between the orthogonal coordinate system on the first plane and the orthogonal coordinate system on the side of the imaging device 30. Instead, processing can be performed in the same coordinate system.

However, the arrangement between the projector 20 and the imaging device 30 is not limited to the relative positional relationship illustrated in FIGS. 2 and 3, and therefore, the optical systems of the projector 20 and the imaging device 30 are generally arranged. There may be system settings where the optical axes of the are not parallel to each other. In this case, one of the orthogonal coordinate system on the side of the projector 20 and the orthogonal coordinate system on the side of the imaging device 30 is made to coincide with the orthogonal coordinate system on the first plane, and coordinate conversion is performed between the other and the orthogonal coordinate system on the first plane. Then, the pattern image generation unit 40 and the data processing unit 50 can perform the same processing as that described in the above embodiment.

Here, in a setting in which the optical axes of the optical systems of the projector 20 and the image pickup apparatus 30 are not parallel to each other, the orthogonal coordinate system on the projector 20 side matches the orthogonal coordinate system on the first plane. It is considered that this is more preferable than the second embodiment in which the orthogonal coordinate system on the side of the imaging device 30 matches the orthogonal coordinate system on the first plane. The reason is that, in general, the projector element has a lower spatial resolution and a lower dynamic range than the image pickup element used in the image pickup apparatus 30, and thus the first embodiment is more preferable than the second embodiment. It is considered that the error caused by coordinate conversion can be suppressed.

<2> In the above-described embodiment, the direct conversion technique (homodyne detection) is adopted as the spatial correlation processing with respect to the light intensity I(x, y) of the combined image to obtain a signal of a specific depth z. Extracted. However, the same result can be obtained even if the heterodyne detection is adopted as the spatial correlation processing.

In the case of heterodyne detection, in Equation 14, the angular frequency of the sine wave signal multiplied on both sides of Equation 13 is not ω(w)[i]=ω(z)[i], but ω(w)[i]=ω( z) [i]+Δω. As a result, the angular frequency of the third term in the integral on the right side of Expression 14 is Δω. Here, if a small value is set as Δω, the third term can be extracted by performing low-pass filter processing. Further, since the third term includes frequency components from DC to Δω, homodyne detection can be further performed to extract a signal of angular frequency Δω. However, in the present apparatus and the present method, when heterodyne detection is adopted, the number of processes for extracting the DC component shown in Expressions 15 and 16 is increased as compared with the case where the direct conversion technique (homodyne detection) is adopted. Therefore, it is sufficient to use the direct conversion technique (homodyne detection).

<3> In the above embodiment, the pattern image has the light intensity L(x, y, z) on the first plane along one direction (x direction) represented by a sine function having x as a variable. It is assumed that the amplitude changes periodically, but the amplitude is cyclic along two directions (for example, the x direction and the y direction, that is, two directions that are orthogonal) represented by a sine function having x and y as variables. It may be changed.

As a result of evaluation of a case where the object 60 having a more complicated internal structure is measured as a measurement target by simulation separately from the above-described evaluation experiment, the pattern image periodically changes in amplitude along one direction (x direction). In this case, it has been confirmed that a portion having a large error in the calculation result of the reflectance inside the object is likely to occur at a relatively deep portion inside the object along the direction (y direction) orthogonal to the one direction. There is. Therefore, it is preferable to change the amplitude of the pattern image periodically along two directions represented by a sine function having x and y as variables, because the above-mentioned error is suppressed.

In the following, in the case where the light intensity L(x, y, z) of the pattern image changes periodically along two directions, which of the mathematical expressions has been described in the procedure for calculating the reflectance (x, y, z) is Only the major formulas will be explained.

Expression 13 is changed to Expression 21 below, and Expression 2 is decomposed into Expressions 22 and 23 below.

Next, by multiplying both sides of Expression 21 by a sine wave signal having an angular frequency ω _x (w)[i] in the x direction, the following Expression 24 is obtained instead of Expression 14 above.

Here, only when z=w, the third term in the integral on the right side of the equation 24 does not depend on the index i, and the continuous signal I(x, y)[i]sin(24) of the equation 24 is obtained. It becomes a DC component of ω _x (w)[i]x). The DC component of the continuous signal can be easily extracted as the following Expression 25 instead of Expression 15 by performing the well-known discrete Fourier transform (DFT).

Similarly, when the discrete Fourier transform (DFT) described above is performed by multiplying both sides of Expression 21 by a cosine wave signal having an angular frequency of ω _x (w)[i], instead of Expression 16 below, 26 is obtained.

Since cos ² φ _x +sin ² φ _x =1, the following Equation 27 is derived from Equation 25 and Equation 26 instead of Equation 17 above.

Next, by multiplying both sides of Expression 21 by a sine wave signal having an angular frequency of ω _y (w)[i] in the y direction, the following Expression 28 is obtained instead of Expression 14 above.

Here, only when z=w, the fifth term in the integral on the right-hand side of Equation 28 does not depend on the index i, and the continuous signal I(x,y)[i]sin( It becomes the DC component of ω _y (w)[i]y). The DC component of the continuous signal can be easily extracted by the well-known Discrete Fourier Transform (DFT) as the following Equation 29 instead of the above Equation 15.

Similarly, when both sides of the equation 21 are multiplied by a cosine wave signal having an angular frequency of ω _y (w)[i] and the discrete Fourier transform (DFT) is performed, Is obtained.

Since cos ² φ _y +sin ² φ _y =1, the following Equation 31 is derived from Equation 29 and Equation 30 instead of Equation 17 above.

From the above, the amplitude A ^* (x, y, z) can be obtained by two direct conversion processes in the x-direction and the y-direction, respectively, as two calculation results of Expressions 27 and 31. Therefore, the reflectance R(x, y, z) of the object 60 at an arbitrary three-dimensional position (x, y, z), which is the calculation target of this section given by the above-mentioned Equation 18, also has two calculation results. Be done. Therefore, as the simplest method, it is conceivable to use the average value of the two calculation results of the equations 27 and 31 as the numerator amplitude A ^* (x, y, z) on the right side of the equation 18. Further, in any three-dimensional position (x, y, z), an amplitude ^{A * (x, y, z} ) the difference between the calculation results of the two types of ^{ΔA * (x, y, z} ) is the predetermined threshold value ^A _{* th} for more than ^{(ΔA * (x, y,} z) ≧ a * th), * the amplitude ^{a * (x, y, z} ) of the two types of calculation results of the periphery of the amplitude ^a (x, y, z) A method such as selecting the calculation result having the smaller difference from is also conceivable.

When the reflectance R(x, y, z) of the object 60 at an arbitrary three-dimensional position (x, y, z) which is the calculation target of this section is calculated in the above manner, the first plane of the pattern image is calculated. Even when the upper light intensity L(x, y, z) changes periodically in amplitude along two directions represented by a sine function having x and y as variables, the pattern described in the above embodiment Similar to the case where the light intensity L(x, y, z) on the first plane of the image periodically changes in amplitude along one direction (x direction) represented by a sine function having x as a variable. , The reflectance R(x, y, z ₁ ) of the object 60 at the depth z ₁ is calculated, and thereafter, the same procedure is repeated to determine an arbitrary depth z() after the depth z _1. The reflectance R(x, y, z) of the object 60 when z>z ₁ ) can be calculated.

<4> In the above embodiment, as an optical characteristic inside the object, a specific embodiment in which the reflectance (x, y, z) inside the object 60 at an arbitrary three-dimensional position (x, y, z) is calculated. As described above, instead of or in addition to the reflectance, the transmittance, the scattering coefficient, the absorption coefficient, etc. inside the object 60 at each three-dimensional position (x, y, z) inside the object are calculated as the reflectance (x , Y, z), the light intensity I(x, y) of the combined image on the imaging surface may be calculated from the digital image signal D2.

INDUSTRIAL APPLICABILITY The measuring device and method of the present invention can be used as a measuring device and method for measuring optical characteristics in a three-dimensional space inside an object. As an example, it can be applied to a visualization device inside an object such as a human body or food. More specifically, with the measuring device and method of the present invention, medical devices such as a fundus examination device and a subcutaneous imaging device can be realized at low cost, and further, a simple device for home use can be expected.

10: Measuring device 21: Beam splitter 22: Laser projector 23: Polarizer 20: Projector 30: Imaging device 31: Telecentric lens 32: RGB camera 33: Polarizer 40: Pattern image generating unit 50: Data processing unit 60: Object ( Measurement object)
61-64: OHP sheet

Claims (10)

A measuring device for measuring optical characteristics inside an object,
A projector that projects a pattern image toward the object using light that can pass through the object;
Of the light of the pattern image projected from the projector, the light reflected at each of the three-dimensional positions on the surface and the inside of the object is incident on the imaging surface where the light can be sensed and is imaged to be combined. An imaging device for converting the synthesized image thus converted into a digital image signal and outputting the digital image signal;
Image data of the pattern image having a periodic amplitude change of a light intensity represented by a sine function in at least one direction in a predetermined first plane across the object, the plurality of spaces having different sine functions. By generating the image data of the plurality of pattern images respectively corresponding to frequencies and inputting the image data to the projector, the plurality of pattern images having different spatial frequencies from each other with respect to the projector, A pattern image generation unit for projecting toward the object separately while sequentially changing in time,
Input the plurality of spatial frequencies in the image data generated by the pattern image generation unit and the digital image signals of the plurality of composite images respectively corresponding to the plurality of spatial frequencies output from the imaging device. As a result, a predetermined correlation calculation process is performed on these inputs to extract the signal amplitude of the light reflected from a specific three-dimensional position on the surface and inside of the object, and the signal amplitude of each extracted three-dimensional position. A data processing unit that calculates the optical characteristics at each of the three-dimensional positions on the surface and inside of the object based on
A measuring apparatus, wherein focal lengths of optical systems of the projector and the image pickup apparatus are different from each other. The measuring device according to claim 1, wherein the optical axes of the optical systems of the projector and the imaging device are parallel to each other and orthogonal to the first plane. The measuring device according to claim 1, wherein the projector is a perspective projection projector, and the imaging device is a parallel projection camera. The data processing unit multiplies a plurality of light intensities corresponding to the spatial frequencies that sequentially change of the plurality of digital image signals, respectively, by a sine function value of an angular frequency corresponding to each spatial frequency. A discrete Fourier transform is performed on each of the obtained first data sequence and the second data sequence obtained by multiplying the plurality of light intensities by the cosine function value of the angular frequency corresponding to each spatial frequency. The measuring device according to claim 1, wherein the signal amplitude is extracted based on each calculation result obtained separately. The measuring apparatus according to claim 1, wherein the periodic amplitude change of the light intensity of the pattern image is a change in two orthogonal directions in the first plane. A measurement method for measuring optical characteristics inside an object,
A projector that projects a pattern image toward the object using light that can pass through the object;
Of the light of the pattern image projected from the projector, the light reflected at each of the three-dimensional positions on the surface and the inside of the object is incident on the imaging surface where the light can be sensed and is imaged to be combined. By using the image pickup device that converts the synthesized image into a digital image signal and outputs it,
Image data of the pattern image having a periodic amplitude change of a light intensity represented by a sine function in at least one direction in a predetermined first plane across the object, the plurality of spaces having different sine functions. By generating the image data of the plurality of pattern images respectively corresponding to frequencies and inputting the image data to the projector, the plurality of pattern images having different spatial frequencies from each other with respect to the projector, A pattern image generating step of projecting toward the object separately while sequentially changing in time;
Input the plurality of spatial frequencies in the image data generated in the pattern image generation step, and the digital image signals of the plurality of composite images respectively corresponding to the plurality of spatial frequencies output from the imaging device. As a result, a predetermined correlation calculation process is performed on these inputs to extract the signal amplitude of the light reflected from a specific three-dimensional position on the surface and inside of the object, and the signal amplitude of each extracted three-dimensional position. A data processing step of calculating the optical characteristic at each of the three-dimensional positions on the surface and inside of the object based on
A measuring method characterized in that focal lengths of respective optical systems of the projector and the imaging device are different from each other. 7. The measuring method according to claim 6, wherein the optical axes of the optical systems of the projector and the imaging device are parallel to each other and orthogonal to the first plane. 8. The measuring method according to claim 6, wherein the projector is a perspective projection projector, and the imaging device is a parallel projection camera. In the data processing step, a plurality of light intensities corresponding to the spatial frequencies of the plurality of digital image signals that sequentially change are respectively multiplied by a sine function value of an angular frequency corresponding to the spatial frequencies. A discrete Fourier transform is performed on each of the obtained first data sequence and the second data sequence obtained by multiplying the plurality of light intensities by the cosine function value of the angular frequency corresponding to each spatial frequency. 9. The measuring method according to claim 6, wherein the signal amplitude is extracted based on each calculation result obtained separately. The measuring method according to claim 6, wherein the periodic amplitude change of the light intensity of the pattern image is a change in two orthogonal directions in the first plane.

Images