JP7390918B2 - Imaging device for 3D images and imaging display device for 3D images - Google Patents

Description

The present invention relates to a three-dimensional image capturing device and a three-dimensional image capturing and display device that capture three-dimensional images for stereoscopic television and the like using so-called integral photography technology.

Conventionally, integral photography has been proposed as a method for capturing three-dimensional images. The optical system of a conventional integral photography imaging device is configured as shown in FIG. 22, for example. That is, the configuration is such that an object 301 is irradiated with illumination light 302a from an illumination light source 302, and reflected light 301a from the object 301 is focused using a lens 303, and a lens array 305 is provided on the light incident side of the image sensor 304. It is provided.
A plurality of unit elements of the image sensor 304 are arranged on the imaging plane of each lens element of the lens array 305, and a two-dimensional image (element image) when the object is viewed from each lens element position is captured. (See Non-Patent Document 1). Since the elemental images 306 formed by each lens element become two-dimensional images of one object viewed from different angles, the three-dimensional image reconstruction means 308 of the arithmetic processing unit 309 reconstructs the elemental images 306 into three-dimensional images. A dimensional image can be obtained (see non-patent documents 1 and 2).

The resolution of the reconstructed three-dimensional image is determined by the diameter of each lens element of the lens array 305, and the resolution improves as each lens element becomes smaller. 305 is required. Further, the angular resolution depends on the number of unit elements forming one elemental image 306, and as the number of unit elements forming one elemental image 306 increases, the angular resolution improves. It is desirable that the number of unit elements of the image sensor 304 correspondingly arranged is large. However, since there is a limit to reducing the size of the unit element of the image sensor 304, it is difficult to improve both resolutions at the same time.
In order to solve these problems, a method has been proposed to improve the resolution of three-dimensional images using single pixel imaging or ghost imaging technology (Non-Patent Document 3).

Single pixel imaging and ghost imaging are methods of acquiring a two-dimensional image using one photodiode, and first measure the reflected light intensity of an image that is a composite of an image of an object and an image of a spatial two-dimensional pattern.
The selected two-dimensional pattern is randomly changed multiple times, and the intensity of reflected light at that time is measured using a single photodiode.

１つのフォトダイオードで２次元画像を取得することができ、その空間解像度は２次元パターンの空間解像度に依存する。このシングルピクセルイメージングあるいはゴーストイメージングをインテグラルフォトグラフィに適用した高密度３次元画像撮像装置の概略を以下に説明する。 In the 2D pattern array P(x,y), measure the nth 2D pattern array when changing the 2D pattern N times when P _n (x,y), the nth 2D pattern When the reflected light intensity is S _n , the image I(x,y) is expressed by the following equation (1).

A two-dimensional image can be acquired with one photodiode, and its spatial resolution depends on the spatial resolution of the two-dimensional pattern. An outline of a high-density three-dimensional image capturing device that applies this single pixel imaging or ghost imaging to integral photography will be described below.

In this method, the intensity of reflected light from an object is modulated in a two-dimensional pattern and is acquired by each pixel (hereinafter also referred to as a unit element) of a two-dimensional image sensor such as a CMOS sensor. A lens array as shown in FIG. 22 is not required on the front surface of the two-dimensional image sensor, and a two-dimensional image is acquired with one pixel of the two-dimensional image sensor using single pixel imaging or ghost imaging. Since the appearance of the object differs depending on the physical measurement position, the two-dimensional image obtained from each pixel of the image sensor becomes a parallax image, which is an image when the object is viewed from the pixel position.

By rearranging the disparity images acquired at each pixel, elemental images that can be obtained in integral photography are obtained, and based on these elemental images, a method similar to conventional integral photography is applied. Since a three-dimensional image can be constructed using the three-dimensional image, it is possible to obtain a high-density three-dimensional image.

Research and development of integral 3D television: Masato Miura: NHK Giken R&D No.158 2016.8 Recent progress in three-dimensional information ocessing based on integral imaging: Jae-Hyeung Park, Keehoon Hong, and Byoungho Lee: 1 December 2009 / Vol. 48, No. 34 / APPLIED OPTICS R. Usami, T. Nobukawa, M. Miura, N. Ishii, E. Watanabe, and T. Muroi, Opt. Lett. 45, 25 (2020).

By the way, in a three-dimensional image capturing device that applies single pixel imaging or ghost imaging to integral photography, when a two-dimensional pattern is displayed on a spatial light modulator and acquired, the intensity distribution within the plane of the two-dimensional image sensor is The difference in the average brightness distribution between the reconstructed parallax images caused unnecessary intensity variations and discontinuities in the three-dimensional image.

The present invention was made in view of the above circumstances, and when a two-dimensional pattern is displayed on a spatial light modulator and imaged, the intensity distribution in the plane of the two-dimensional image sensor and the accompanying SNR distribution are suppressed to a small value, and reconstruction is performed. An imaging device for 3D images and an imaging display device for 3D images that can reduce differences in average luminance distribution between parallax images and suppress unnecessary intensity variations and discontinuities in 3D images. The purpose is to provide

The first three-dimensional image imaging device of the present invention includes:
A spatial light modulator that displays a multivalued two-dimensional pattern is provided between an illumination light source that emits illumination light and an object that is a subject, and the illumination light patterned by the spatial light modulator forms an image on the object. an optical system consisting of a lens arranged so that Prepare,
multi-value two-dimensional pattern variable instruction means for temporally changing the multi-value two-dimensional pattern displayed on the spatial light modulator;
A multi-value two-dimensional pattern displayed on the spatial light modulator according to instructions from the multi-value two-dimensional pattern variable instruction means, and each of the unit elements of the image pickup device when the multi-value two-dimensional pattern is displayed. The relationship of the light intensity information acquired in is captured for each display of the multi-valued two-dimensional pattern, and a two-dimensional element image is obtained for each unit element based on the values captured for each of the multi-valued two-dimensional patterns. Elemental image reconstruction means;
A correction process is performed such that the intensity distribution of the two-dimensional elemental image for each unit element obtained by the two-dimensional elemental image reconstruction means is smoothed, and the intensity distribution of the two-dimensional elemental image is smoothed. a smoothing correction processing means for outputting a dimensional element image;
3-dimensional image reconstruction means for reconstructing a 3-dimensional image of the object based on the 2-dimensional element image of each unit element subjected to correction processing by the smoothing correction processing means;
It is characterized by having the following.

Further, the second three-dimensional image imaging device of the present invention includes:
a spatial light modulator that displays a multivalued two-dimensional pattern, which is provided on the light exit side of an object that is a subject that is irradiated with illumination light from an illumination light source; and reflected light or transmitted light of the illumination light from the object. an image sensor in which unit elements are two-dimensionally arranged to obtain image information of the object patterned by the spatial light modulator; and reflected light from the object carrying image information of the object; an optical system configured with a lens arranged to image transmitted light on the spatial light modulator,
multi-value two-dimensional pattern variable instruction means for temporally changing the multi-value two-dimensional pattern displayed on the spatial light modulator;
A multi-value two-dimensional pattern displayed on the spatial light modulator according to instructions from the multi-value two-dimensional pattern variable instruction means, and each of the unit elements of the image pickup device when the multi-value two-dimensional pattern is displayed. The relationship of the light intensity information acquired in is captured for each display of the multi-valued two-dimensional pattern, and a two-dimensional element image is obtained for each unit element based on the values captured for each of the multi-valued two-dimensional patterns. Elemental image reconstruction means;
A correction process is performed such that the intensity distribution of the two-dimensional elemental image for each unit element obtained by the two-dimensional elemental image reconstruction means is smoothed, and the intensity distribution of the two-dimensional elemental image is smoothed. a smoothing correction processing means for outputting a dimensional element image;
3-dimensional image reconstruction means for reconstructing a 3-dimensional image of the object based on the 2-dimensional element image of each unit element subjected to correction processing by the smoothing correction processing means;
It is characterized by having the following.

In addition, in the imaging device for the first and second three-dimensional images, the "two-dimensional elemental image reconstruction means", the "three-dimensional image reconstruction means", and the "smoothing correction processing means" are implemented using a predetermined program. It is created by software or by combining software and hardware.
In addition, the "illumination light source" may be configured exclusively for this device, may be natural light such as sunlight, or may be a light source used for other purposes. It may be something like this.

It is preferable that the smoothing correction processing performed by the smoothing correction processing means is one of moving average filter processing, low-pass filter processing, two-dimensional convolution filter processing, and processing using the least squares method.
Here, when the smoothing correction process is a low-pass filter process, it is preferable that the intensity distribution is a low-pass filter process that cuts frequencies with a spatial frequency of 0.25 or more; When the process uses the least squares method, it is preferable that the process uses the least squares method of a cubic function model.

但し、P_n(x,y)は、２次元パターンの配列P(x,y)において、パターンをN回変えたときのn番目のパターンの配列、S_nは、n番目のパターンのときに計測した反射光光強度である。 The multi-value two-dimensional pattern is a binary two-dimensional pattern, and the manner in which the binary two-dimensional pattern changes in response to an instruction from the multi-value two-dimensional pattern variable instruction means can be made random.
In this case, the two-dimensional elemental image reconstruction means uses the calculation results for each of the multivalued two-dimensional patterns and calculates the two-dimensional elemental image I (x ,y) is preferable.

However, P _n (x,y) is the nth pattern array when the pattern is changed N times in the two-dimensional pattern array P(x,y), and S _n is the nth pattern array when the pattern is changed N times. This is the measured reflected light intensity.

Further, the imaging display device for three-dimensional images of the present invention includes any one of the above-mentioned imaging devices for three-dimensional images, and display means for displaying the three-dimensional image reconstructed by the three-dimensional image reconstruction means. It is characterized by the fact that it is equipped with

In the three-dimensional image capturing device of the present invention, a three-dimensional image capturing method using integral photography using single pixel imaging or ghost imaging is introduced, and an image of an object and an image of a spatial two-dimensional pattern are captured. The reflected light intensity of the combined image is captured for each unit element of the image sensor, and a two-dimensional element image is obtained for each unit element. As a result, one two-dimensional elemental image can be obtained by one unit element, and a three-dimensional image can be constructed based on this elemental image using a method similar to conventional integral photography. , it becomes possible to obtain high-density three-dimensional images.

In addition, in conventional methods, there is an intensity distribution within the plane of the two-dimensional image sensor due to the optical system such as a lens inserted in the optical path, and an associated SNR distribution, so the average brightness distribution between the reconstructed parallax images This difference causes unnecessary intensity variations and discontinuities in the three-dimensional image. However, in the apparatus of the present invention, the smoothing correction processing means performs correction processing such that the intensity distribution of the two-dimensional element images between adjacent unit elements in the image sensor is smoothed, and the intensity distribution Since the smoothed two-dimensional element image is output to the three-dimensional image reconstruction means, noise in the constructed three-dimensional image can be reduced, and each unit element of the image sensor It becomes possible to correct unevenness in brightness values between parallax images reconstructed in .

Further, the imaging display device for three-dimensional images of the present invention includes any one of the above imaging devices for three-dimensional images, and display means for displaying the three-dimensional image reconstructed by the three-dimensional image reconstruction means. Since the reconstructed three-dimensional image is displayed on this display means, it is possible to capture and display a three-dimensional image with excellent spatial resolution and angular resolution while making it easy to manufacture the image sensor. can.

1 is a schematic diagram for explaining the principle of an imaging display device for three-dimensional images according to Embodiment 1 of the present invention. FIG. 2 is a schematic diagram for explaining the principle of an imaging display device for three-dimensional images according to Embodiment 2 of the present invention. 2 is a flowchart for explaining an outline of the flow of processing performed in the imaging device for three-dimensional images according to the present embodiment. FIG. 2 is a conceptual diagram for explaining pixel positions of an image sensor in the three-dimensional image imaging device according to the present embodiment. Schematic diagram showing the intensity distribution on one line (one row) when the reflected light intensity of an object modulated by a two-dimensional pattern is acquired by a two-dimensional image sensor ((a) is the state before correction processing, (b) represents the state after correction processing). Diagrams showing objects (two dice placed 25 mm apart) used in each example ((a) is a diagram showing an image seen from the actual imaging direction, (b) is a diagram showing an image transverse to the imaging direction. (Figure showing the image seen from the front). 2 is a graph showing intensity changes due to application of low-pass filtering in Example 1 ((a) shows the state before applying low-pass filtering, and (b) shows the state after applying low-pass filtering). 3 shows the intensity value distribution on the image sensor surface for a selected two-dimensional pattern before and after processing in Example 1 (moving average filter processing). The reconstructed images before and after processing in Example 1 (moving average filter processing) are compared. The figure compares numerical values for reconstructed images before and after processing in Example 1 (moving average filter processing). 12 is a comparison of brightness value unevenness in reconstructed images of adjacent pixels before and after processing in Example 1 (moving average filter processing). 3 is a comparison of luminance value dispersion of reconstructed images of adjacent pixels before and after processing in Example 1 (moving average filter processing). It shows the intensity value distribution on the image sensor surface for a selected two-dimensional pattern before and after processing in Example 2 (low-pass filter processing). The reconstructed images before and after processing in Example 2 (low-pass filter processing) are compared. It is a comparison of numerical values for reconstructed images before and after processing in Example 2 (low-pass filter processing). This is a comparison of brightness value unevenness in reconstructed images of adjacent pixels before and after processing in Example 2 (low-pass filter processing). This is a comparison of luminance value dispersion of reconstructed images of adjacent pixels before and after processing in Example 2 (low-pass filter processing). It shows the intensity value distribution on the image sensor surface for a selected two-dimensional pattern before and after processing in Example 3 (least squares method processing). The reconstructed images before and after processing in Example 3 (least squares method processing) are compared. This is a comparison of numerical values for reconstructed images before and after processing in Example 3 (least squares method processing). This is a comparison of luminance value unevenness in reconstructed images of adjacent pixels before and after processing in Example 3 (least squares method processing). This is a comparison of the luminance value dispersion of reconstructed images of adjacent pixels before and after processing in Example 3 (least squares method processing). 1 is a schematic diagram for explaining an imaging device for three-dimensional images according to the prior art.

DESCRIPTION OF THE PREFERRED EMBODIMENTS An imaging device and an imaging display device for three-dimensional images according to embodiments of the present invention will be described below with reference to the drawings.
First, a three-dimensional image imaging device and a three-dimensional image imaging display device according to Embodiment 1 and Embodiment 2 will be described using FIGS. 1A and 1B. Note that the imaging display device for three-dimensional images is constructed by adding display means 10 and 110 to the imaging device for three-dimensional images.
In either device, a correction process is performed to smooth the intensity distribution of the two-dimensional elemental image for each unit element of the image sensor 4, 104 obtained by the two-dimensional elemental image reconstruction means 7, 107. , smoothing correction processing means 9 and 109 for outputting two-dimensional elemental images with smoothed intensity distributions, and this embodiment is characterized in this respect. A common overall schematic configuration will be described, and then each embodiment device will be described.

(Common to both embodiment devices)
As shown in FIGS. 1A and 1B, the imaging apparatus for three-dimensional images according to Embodiments 1 and 2 is arranged on an incident path to an object 1 that is a subject or on an exit path from an object 101 that is a subject. , a binary two-dimensional pattern 12b, etc. in the image information of the object 1, 101 (in the second embodiment, the two-dimensional pattern is considered to match the display pattern 112a, so the two-dimensional pattern in the first and second embodiments is In the explanation, it will be referred to as "12b, etc." (The same applies hereinafter), and from the image information of the object 1, 101 on which information such as the binary two-dimensional pattern 12b is superimposed, , and an image sensor 4, 104 that acquires light intensity information in each unit element.

Furthermore, each time the information such as the binary two-dimensional pattern 12b is randomly switched to another display pattern 12a, 112a, the light intensity information acquired in each unit element of the image sensors 4, 104 is used to display a predetermined image. Two-dimensional elemental image reconstruction means 7 and 107 are provided for reconstructing a two-dimensional elemental image for each unit element based on an arithmetic expression.

In addition, correction processing is performed to smooth the intensity distribution of the two-dimensional elemental images for each unit element obtained by the two-dimensional elemental image reconstruction means 7 and 107, and the intensity distribution is smoothed. It includes smoothing correction processing means 9 and 109 that output dimensional element images. The smoothing correction processing means 9 and 109 are also a key point in the configuration of this embodiment.

Furthermore, using the two-dimensional element images that have been subjected to smoothing correction processing by the smoothing correction processing means 9 and 109, a three-dimensional It includes three-dimensional image reconstruction means 8, 108 for reconstructing an image.
These two-dimensional elemental image reconstruction means 7 and 107, smoothing correction processing means 9 and 109, and three-dimensional image reconstruction means 8 and 108 constitute a control arithmetic processing device 5 and 105.
The control processing devices 5, 105 also have a function of instructing the two-dimensional pattern information providing means 12, 112 to switch the display patterns 12a, 112a displayed on the spatial light modulators 11, 111.

In addition to the configuration of the above-described three-dimensional image imaging device, the three-dimensional image capturing and displaying device also captures three-dimensional images of the objects 1 and 101 reconstructed by the three-dimensional image reconstruction means 8 and 108. It is equipped with display means 10 and 110 for displaying images.

Next, the flow of processing in this embodiment (common to embodiments 1 and 2) will be explained using FIG. That is, first, a first two-dimensional pattern is generated (S1), then this two-dimensional pattern is displayed on the spatial light modulators 11 and 111 (S2), and each unit element of the image sensors 4 and 104 generates a two-dimensional pattern. Elemental images are acquired (S3), and a series of these processes (S1 to S3) is repeated each time a two-dimensional pattern is generated (S1 to S4). When the series of processes described above are completed for the scheduled number of two-dimensional patterns, the smoothing correction processing means 9, 109 performs smoothing correction processing on the intensity distribution for each two-dimensional element image (S5). The three-dimensional image reconstruction means 8 and 108 perform a process of reconstructing a three-dimensional image from the two-dimensional elemental image of each unit element that has been subjected to the correction process (S6).
As a processing means for superimposing binary two-dimensional pattern information on the image information of the objects 1 and 101, for example, a spatial light modulator 11 such as a liquid crystal display element (transmissive type or reflective type) or a DMD (digital micromirror device) is used. , 111 are preferable, and the spatial light modulators 11 and 111 are configured to sequentially display a mode in which the black and white binary display patterns 12a and 112a are randomly changed.

Specifically, for example, a mode in which the binary display patterns 12a and 112a are randomly changed in a predetermined memory (for example, the two-dimensional patterns shown in FIGS. 7, 12, and 17) is The spatial light modulators 11 and 111 display one of the display patterns 12a and 112a among the two-dimensional patterns stored and sequentially selected by a random number generator (two-dimensional pattern information providing means). may be configured.

Of course, the display patterns 12a, 112a are not stored in the memory as two-dimensional patterns as described above, but the display patterns 12a, 112a are displayed on the spatial light modulators 11, 111 by the two-dimensional pattern information providing means 12, 112. The displayed patterns 12a and 112a may be formed in real time by random number lottery immediately before the display.

In addition, in the two-dimensional elemental image reconstruction means 7, 107, the calculation formula for reconstructing the two-dimensional elemental images 6, 106 using the acquired light intensity information for each unit element of the image sensor 4, 104 is , for example, is configured as follows.

このようにして、撮像素子４、１０４の単位素子のそれぞれにより取得された２次元要素画像は、撮像素子４、１０４の単位素子の位置に応じて、異なる角度から見た物体の形状情報を表すこととなり、取得された２次元要素画像は、互いにずれた物体位置の形状情報を有するものとなる。 That is, in the two-dimensional pattern array P(x,y) described above, the n-th pattern array when the display patterns 12a, 112a are changed N times is P _n (x,y), the image sensor 4, If the reflected light intensity measured at the nth pattern at pixel (i,j) of 104 is S _i,j,n , then the two-dimensional element image I at pixel (i,j) of image sensor 4, 104 is _i,j (x,y) is expressed by the following formula (3).

In this way, the two-dimensional element images acquired by each of the unit elements of the image sensors 4 and 104 represent shape information of the object viewed from different angles depending on the position of the unit element of the image sensors 4 and 104. Therefore, the acquired two-dimensional element images have shape information of mutually shifted object positions.

Furthermore, in the three-dimensional image reconstruction means 8 and 108, the three-dimensional image is As mentioned above, the predetermined calculation for reconstructing the dimensional image is performed using the well-known method in integral photography described in the above-mentioned Non-Patent Document 2, for example. Note that a detailed explanation of the smoothing correction processing means 9 and 109, which is the key point of the first and second embodiments, will be described later.
Further, as the image pickup devices 4 and 104, various types of image pickup devices such as CMOS and CCD can be used.

<Embodiment 1>
An imaging display device according to Embodiment 1 of the present invention will be described using FIG. 1A. Note that the contents described in the items common to the devices of both embodiments described above are often omitted to avoid duplicate description.
The imaging display device according to the first embodiment includes an illumination light source 2 that irradiates an object 1 with illumination light 2a, and an illumination light source 2 that is disposed in the optical path of the illumination light 2a and between the illumination light source 2 and the object 1. a spatial light modulator 11 that carries a black and white binary two-dimensional matrix-like display pattern 12a; and a system that images the pattern information carried by the illumination light 2a emitted from the spatial light modulator 11 onto the object 1. It includes an image lens 3 and an image sensor 4 that receives reflected light from the object 1 in which image information of the object 1 is superimposed on the pattern information.

Note that it is common to both embodiments that the imaging display device according to Embodiment 1 additionally includes two-dimensional element image reconstruction means 7, smoothing correction processing means 9, and three-dimensional image reconstruction means 8. As stated in the item.
Note that the two-dimensional elemental image reconstruction means 7, the smoothing correction processing means 9, and the three-dimensional image reconstruction means 8 constitute the control arithmetic processing device 5. The control processing device 5 also has a function of instructing the two-dimensional pattern information providing means 12 to sequentially switch the display pattern 12a displayed on the spatial light modulator 11.
That is, the imaging display device according to the first embodiment is characterized in that, as shown in FIG. 1A, a spatial light modulator 11 and a lens 3 are provided between an illumination light source 2 and an object 1 to be imaged.

The position of the imaging lens 3 on the optical axis is adjusted so that the display pattern 12a modulated by the spatial light modulator 11 is imaged on the object 1.
At this time, each unit element of the image sensor 4 receives the light intensity of the reflected light 1a from the object 1 on which the display pattern 12a is superimposed.
The image sensor 4 is connected to a control processing unit 5, and the relationship between the display pattern 12a displayed on the spatial light modulator 11 and the light intensity obtained by each unit element of the image sensor 4 when the pattern 12a is displayed is determined. is stored in the two-dimensional element image reconstruction means 7. The display pattern 12a displayed on the spatial light modulator 11 is changed multiple times, and a two-dimensional elemental image is reconstructed for each unit element based on the light intensity acquired by each unit element of the image sensor 4 each time. Since the display pattern 12a to be changed is random, the two-dimensional elemental image of each unit element can be easily reconstructed by using the above-mentioned equation (3).

As described above, a three-dimensional image can be reconstructed by a well-known method similar to conventional integral photography based on two-dimensional elemental images obtained corresponding to each unit element.
Note that a large capacity memory is stored in the control processing unit 5 or the display means 10 to store the obtained three-dimensional image, and the three-dimensional image of the object 1 can be displayed on the display means 10 as necessary. (The same applies to Embodiment 2).

<Embodiment 2>
An imaging display device according to Embodiment 2 of the present invention will be described using FIG. 1B. Note that the contents described in the items common to the devices of both embodiments described above are often omitted to avoid duplicate description.
Note that since the imaging display device is similar to Embodiment 1, the members corresponding to those in FIG. 1A are given the same reference numerals as those in FIG. 1B plus 100.

As shown in FIG. 1B, an illumination light source 102 that irradiates an object 101 with illumination light 102a, an image sensor 104 that receives reflected light 101a of the illumination light 102a from the object 101, and an image sensor 104 arranged in the optical path of the reflected light 101a. , a spatial light modulator 111 that causes the reflected light 101a to carry a black and white binary two-dimensional matrix display pattern 112a, and an imaging lens that forms an image of the reflected light 101a reflected from the object 101 onto the spatial light modulator 111. 103.

The fact that the imaging display device according to the second embodiment also includes a two-dimensional elemental image reconstruction means 107, a smoothing correction processing means 109, and a three-dimensional image reconstruction means 108 is a common feature of both embodiments. As described in
Note that these two-dimensional elemental image reconstruction means 107, smoothing correction processing means 109, and three-dimensional image reconstruction means 108 constitute a control arithmetic processing device 105. The control processing unit 105 also has a function of instructing the two-dimensional pattern information providing means 112 to switch the display pattern 112a displayed on the spatial light modulator 111.

That is, this imaging display device is characterized in that a lens 103 and a spatial light modulator 111 are provided between an object 101 and an image sensor 104, as shown in FIG. 1B.
In the imaging display device according to the second embodiment, the position of the imaging lens 103 on the optical axis is such that the reflected light 101a from the object 101 carrying the image information of the object 101 forms an image on the spatial light modulator 111. The information of the display pattern 112a displayed on the spatial light modulator 111 is carried in this reflected light 101a.

The image sensor 104 is connected to a control arithmetic processing unit 105, and the display pattern 112a displayed on the spatial light modulator 111 and the light intensity obtained by each unit element of the image sensor 104 when the display pattern 112a is displayed are processed. The relationship is stored in the two-dimensional element image reconstruction means 107. The display pattern 102a displayed on the spatial light modulator 111 is changed multiple times, and a two-dimensional elemental image is reconstructed for each unit element based on the light intensity acquired by each unit element of the image sensor 104 each time. Since the display pattern 112a to be changed is random, the two-dimensional elemental image of each unit element can be easily reconstructed by using the above-mentioned equation (3).

Note that, as described above, a three-dimensional image can be reconstructed by a well-known method similar to conventional integral photography based on two-dimensional element images obtained corresponding to each unit element.

<About the smoothing correction processing means>
By the way, as described above, the three-dimensional image capturing and display apparatuses according to the first and second embodiments apply single pixel imaging or ghost imaging to integral photography. However, in the imaging display device for three-dimensional images according to the first and second embodiments, an intensity distribution and an accompanying SNR distribution inevitably exist within the plane of the two-dimensional image sensor, and the average luminance between reconstructed parallax images The difference in distribution causes unnecessary intensity variations and discontinuities in the three-dimensional image.

Therefore, in the imaging display device for three-dimensional images according to the first and second embodiments, smoothing correction processing means 9 and 109 are provided, and each unit element obtained by the two-dimensional element image reconstruction means 7 and 107 is Correction processing is performed to smooth the intensity distribution of the two-dimensional elemental image and the SNR distribution accompanying it. As a result, a two-dimensional elemental image in which the intensity distribution and the accompanying SNR distribution have been smoothed can be output to the three-dimensional image reconstruction means 8 and 108.

The smoothing correction processing performed by the smoothing correction processing means 9 and 109 includes image shading ( It is a process for smoothing changes in intensity), and includes various image processes that can reduce unnecessary shading fluctuations such as noise included in an image.
Below, representative processing among the above-mentioned processes will be explained in detail, that is, in Example 1, an example using the above-mentioned moving average filter processing, and in Example 2, an example using low-pass filter processing. In the third embodiment, examples using the least squares method will be described.

As the smoothing correction process in Example 1, two dice were placed 25 mm apart in front and back as shown in FIGS. 5(a) and 5(b), and images modulated with various two-dimensional patterns as shown in FIG. A case will be described in which a moving average filter process is applied to the intensity distribution using seven pixel values in total. When this moving average filter is applied, the image I _M (x,y) at the pixel position M of the image sensor is obtained by changing the two-dimensional pattern N times in the two-dimensional pattern array P(x,y). If the array of the th two-dimensional pattern is P _n (x,y), and the reflected light intensity at the pixel position M of the image sensor measured at the time of the n th two-dimensional pattern is S _n,M , then the following formula (4) is obtained. Represented by

The intensity distribution as it is obtained at the pixel position on the image sensor surface as shown in FIG. 3 is shown in the second column (obtained intensity raw data) of FIG.
This intensity distribution is the intensity distribution obtained when each two-dimensional pattern in the first column of FIG. is the intensity value obtained at each pixel, and is expressed as a graph.
This intensity distribution shows a different distribution depending on each two-dimensional pattern, and variations in intensity values can also be confirmed between adjacent pixels.

The intensity distribution obtained by performing moving average filter processing on this intensity distribution is shown in the third column of FIG. 7 (after moving average filter processing). By performing this moving average filter processing, variations between adjacent pixels can be reduced.
Figure 7 shows an example of the intensity value distribution when four different two-dimensional patterns are used, but the moving average filter processing is applied to all of the many two-dimensional patterns used to obtain the intensity. do. A parallax image can be favorably reconstructed by using intensity values at each pixel position and a large number of two-dimensional patterns used to obtain the intensity for the intensity values subjected to moving average filter processing.

FIG. 8 shows a comparison of reconstructed images using intensity values before and after correction by moving average filter processing for the intensity distribution. Further, FIG. 9 shows a comparison of numerical values of reconstructed images using intensity values before and after correction by moving average filter processing. This reconstructed image was created using pixel values in the background whose intensity values are ideally zero.

The comparative evaluation was performed by calculating the average value and variance value of the intensity values of the 10th line (horizontal line position above each image) of each reconstructed image. According to FIG. 8, it is visually clear that the reconstructed image after moving average filter processing has reduced background noise compared to before moving average filter processing, and FIG. 9 also shows that the average value - Since both variance values are lower after the moving average filter processing than before the moving average filter processing, it can be confirmed that noise has been reduced.

Further, FIG. 10 shows comparison results between reconstructed images of adjacent pixels before and after the moving average filter processing. According to FIG. 10, the results of reconstruction using adjacent pixels show that there is variation in brightness values before moving average filter processing, but this variation is reduced after moving average filter processing. can do.

FIG. 11 shows a graph plotting the distribution of variance values in the 10th row of each reconstructed image. Comparing before and after moving average filter processing, the variation in the variance value is smaller after moving average filter processing than before moving average filter processing, and the unevenness of the average brightness distribution in the reconstructed image between adjacent pixels is reduced. You can confirm that.

As described above, by performing moving average filter processing on the acquired intensity values, it is possible to improve the SNR of the reconstructed parallax images, and it is also possible to reduce unevenness in brightness values between the reconstructed parallax images.

The above example shows the result of performing moving average filter processing on one predetermined row on the image sensor surface, but the same process is performed on other columns on the image sensor, and in the end, the intensity of the entire surface of the image sensor By performing the same processing as above on the distribution, it is possible to improve the SNR of the reconstructed parallax image and to reduce unevenness in brightness values between reconstructed elemental images. Further, as the number of pixel positions used in the image sensor increases, variations in brightness values can be reduced.

In the smoothing correction process of the second embodiment, the moving average filter used in the first embodiment is replaced with a high frequency cut (low pass) filter process to reduce variations in the obtained intensity distribution.
In Example 2, the intensity acquisition using patterns and the reconstruction using intensity values are the same as in Example 1 (see FIGS. 5(a), (b), and 7), so there is no duplication. To avoid this, we omit the explanation. Note that an example of the calculation results of the intensity distribution in Example 2 is shown in FIG. 12 (corresponding to FIG. 7 of Example 1).

FIG. 12 shows the comparison results of reconstructed images using intensity values before and after filter processing that cuts frequencies of spatial frequency 0.25 or more with respect to the intensity distribution.
Examples of frequency distributions before and after low-pass filter processing are shown in FIGS. 6(a) and 6(b).

Further, FIG. 13 shows a comparison of reconstructed images using intensity values before and after correction by low-pass filter processing for the intensity distribution. Further, FIG. 14 shows a comparison of numerical values of reconstructed images using intensity values before and after correction by low-pass filter processing. This reconstructed image was created using pixel values in the background whose intensity values are ideally zero.

The comparative evaluation was performed by calculating the average value and variance value of the intensity values of the 10th line (horizontal line position above each image) of each reconstructed image. According to FIG. 13, it is visually clear that the reconstructed image after low-pass filter processing has reduced background noise compared to before low-pass filter processing, and FIG. Since both values are lower after the low-pass filter processing than before the low-pass filter processing, it can be confirmed that the noise has been reduced.
Note that although this embodiment uses an intensity value distribution that has been subjected to a low-pass filter process in which frequencies of 0.25 or higher are cut, the frequencies to be cut are not limited to those used in this embodiment.

Further, FIG. 15 shows comparison results between reconstructed images of adjacent pixels before and after low-pass filter processing. According to FIG. 15, it can be confirmed that the result of reconstruction using adjacent pixels has variations in brightness values before low-pass filter processing, but this variation has been reduced after low-pass filter processing. Can be done.

FIG. 16 shows a graph plotting the distribution of variance values in the 10th row of each reconstructed image. Comparing before and after low-pass filtering, it was confirmed that the dispersion value was smaller after low-pass filtering than before low-pass filtering, and the unevenness of the average brightness distribution was reduced in reconstructed images between adjacent pixels. can do.

As described above, by performing low-pass filter processing on the acquired intensity values, it is possible to improve the SNR of the reconstructed parallax images, and to reduce unevenness in brightness values between the reconstructed parallax images.

The above example shows the result of low-pass filter processing performed on one predetermined row on the image sensor surface, but the same process is performed on other columns on the image sensor, and the result is that the intensity distribution over the entire surface of the image sensor is However, by performing the same processing as described above, it is possible to improve the SNR of the reconstructed parallax image and reduce the unevenness in brightness values between the reconstructed elemental images. Further, as the number of pixel positions used in the image sensor increases, variations in brightness values can be reduced.

In the smoothing correction process of the third embodiment, the moving average filter used in the first embodiment is replaced with a process using the least squares method to reduce variations in the obtained intensity distribution.
In Example 3, the intensity acquisition using patterns and the reconstruction using intensity values are the same as in Example 1 (see FIGS. 5(a), (b), and FIG. 7), so there is no duplication. To avoid this, we omit the explanation. An example of the calculation results of the intensity distribution in Example 3 is shown in FIG. 17 (corresponding to FIG. 7 of Example 1).

FIG. 18 shows a comparison result of reconstructed images using intensity values before and after correction by processing using the least squares method of a cubic function model for the intensity distribution. Further, FIG. 19 shows a comparison of numerical values of reconstructed images using intensity values before and after correction by processing using the least squares method. This reconstructed image was created using pixel values in the background whose intensity values are ideally zero.

The comparative evaluation was performed by calculating the average value and variance value of the intensity values of the 10th line (horizontal line position above each image) of each reconstructed image. According to FIG. 18, it is visually clear that background noise in the reconstructed image after processing using the least squares method is reduced compared to before processing using the least squares method. 19, both the mean and variance values are lower after processing using the least squares method than before processing using the least squares method, confirming that noise has been reduced. can do.

Note that although this embodiment employs processing using the least squares method of a cubic function model, the order of the function model may be other than cubic.
Further, FIG. 20 shows comparison results of reconstructed images of adjacent pixels before and after processing using the least squares method. According to FIG. 20, the result of reconstruction using adjacent pixels has variations in brightness values before processing using the least squares method, but this variation is reduced after processing using the least squares method. You can confirm that it is.

FIG. 21 shows a graph plotting the distribution of variance values in the 10th row of each reconstructed image. Comparing before and after processing using the least squares method, the variation in variance values is smaller after processing using the least squares method than before processing using the least squares method, and the average of the reconstructed image between adjacent pixels is smaller. It can be confirmed that the unevenness of the brightness distribution has been reduced.

From the above, it is possible to improve the SNR of the reconstructed parallax images by processing the obtained intensity values using the least squares method, and to reduce the unevenness of brightness values between the reconstructed parallax images. can.

The above example shows the result of processing using the least squares method on one predetermined row on the image sensor surface, but the same process is performed on other columns on the image sensor, and eventually the image sensor By performing the same processing as above on the intensity distribution over the entire surface, it is possible to improve the SNR of the reconstructed parallax image and reduce unevenness in brightness values between reconstructed elemental images. Further, as the number of pixel positions used in the image sensor increases, variations in brightness values can be reduced.

Note that the three-dimensional image imaging device and the three-dimensional image imaging display device of the present invention are not limited to those of the above embodiments, and various other modifications may be made.
For example, the smoothing correction processing performed by the smoothing correction processing means 9 and 109 is not limited to that of the above-mentioned embodiments, but may include the two-dimensional convolution filter processing described above, and other types of image shading (intensity) changes. Various image correction processes can be employed to smooth the image and reduce unnecessary shading variations such as noise included in the image.

Further, the display patterns 12a and 112a are not random patterns, but use DCT bases or Hadamard bases, thereby reducing the image reconstruction time.
Compressed sensing techniques can also be incorporated to further reduce time.
Furthermore, in the above embodiment, the object is a non-transparent body and the image is reconstructed using the reflected light from the object, but the object is a transparent body and the transmitted light from the object is used to reconstruct the image. It is also possible to reconstruct an image (for example, an image representing the density distribution within a transparent object) by using this.
Further, in the above embodiment, a binary display pattern of black and white is used, but it is also possible to use a pattern of three or more values, including values having gray levels in addition to black and white.

1, 101, 301 Object 1a, 101a, 301a Reflected light 2, 102, 302 Illumination light source 2a, 102a, 302a Illumination light 3, 103, 303 Lens 4, 104, 304 Image sensor 5, 105 Control processing unit 7, 107 Two-dimensional element image reconstruction means 8, 108, 308 Three-dimensional image reconstruction means 9, 109 Smoothing correction processing means 10, 110 Display means 11, 111 Spatial light modulators 12, 112 Two-dimensional pattern information provision means 12a, 112a Display pattern 12b 2-dimensional pattern 305 Lens array 306 2-dimensional element image 309 Arithmetic processing unit

Claims (8)

A spatial light modulator that displays a multivalued two-dimensional pattern is provided between an illumination light source that emits illumination light and an object that is a subject, and the illumination light patterned by the spatial light modulator forms an image on the object. an optical system consisting of a lens arranged so that Prepare,
multi-value two-dimensional pattern variable instruction means for temporally changing the multi-value two-dimensional pattern displayed on the spatial light modulator;
A multi-value two-dimensional pattern displayed on the spatial light modulator according to instructions from the multi-value two-dimensional pattern variable instruction means, and each of the unit elements of the image pickup device when the multi-value two-dimensional pattern is displayed. The relationship of the light intensity information acquired in is captured for each display of the multi-valued two-dimensional pattern, and a two-dimensional element image is obtained for each unit element based on the values captured for each of the multi-valued two-dimensional patterns. Elemental image reconstruction means;
A correction process is performed such that the intensity distribution of the two-dimensional elemental image for each unit element obtained by the two-dimensional elemental image reconstruction means is smoothed, and the intensity distribution of the two-dimensional elemental image is smoothed. a smoothing correction processing means for outputting a dimensional element image;
3-dimensional image reconstruction means for reconstructing a 3-dimensional image of the object based on the 2-dimensional element image of each unit element subjected to correction processing by the smoothing correction processing means;
An imaging device for three-dimensional images, comprising: a spatial light modulator that displays a multivalued two-dimensional pattern, which is provided on the light exit side of an object that is a subject that is irradiated with illumination light from an illumination light source; and reflected light or transmitted light of the illumination light from the object. an image sensor in which unit elements are two-dimensionally arranged to obtain image information of the object patterned by the spatial light modulator; and reflected light from the object carrying image information of the object; an optical system configured with a lens arranged to image transmitted light on the spatial light modulator,
multi-value two-dimensional pattern variable instruction means for temporally changing the multi-value two-dimensional pattern displayed on the spatial light modulator;
A multi-value two-dimensional pattern displayed on the spatial light modulator according to instructions from the multi-value two-dimensional pattern variable instruction means, and each of the unit elements of the image pickup device when the multi-value two-dimensional pattern is displayed. The relationship of the light intensity information acquired in is captured for each display of the multi-valued two-dimensional pattern, and a two-dimensional element image is obtained for each unit element based on the values captured for each of these multi-valued two-dimensional patterns. Elemental image reconstruction means;
A correction process is performed such that the intensity distribution of the two-dimensional elemental image for each unit element obtained by the two-dimensional elemental image reconstruction means is smoothed, and the intensity distribution of the two-dimensional elemental image is smoothed. a smoothing correction processing means for outputting a dimensional element image;
3-dimensional image reconstruction means for reconstructing a 3-dimensional image of the object based on the 2-dimensional elemental image of each unit element subjected to correction processing by the smoothing correction processing means;
An imaging device for three-dimensional images, comprising: The smoothing correction processing performed by the smoothing correction processing means is any one of moving average filter processing, low-pass filter processing, two-dimensional convolution filter processing, and processing using the least squares method. The imaging device for three-dimensional images according to claim 1 or 2. 4. The imaging device for three-dimensional images according to claim 3, wherein the smoothing correction process is a low-pass filter process that cuts frequencies of a spatial frequency of 0.25 or more with respect to the intensity distribution. 4. The imaging device for three-dimensional images according to claim 3, wherein the smoothing correction process is a process using a least squares method of a cubic function model. In the smoothing correction processing performed by the smoothing correction processing means, the multi-valued two-dimensional pattern is a binary two-dimensional pattern, and the two-valued pattern changes according to an instruction from the multi-valued two-dimensional pattern variable instruction means. The imaging device for three-dimensional images according to any one of claims 1 to 5, wherein the two-dimensional pattern changes in a random manner. The two-dimensional elemental image reconstruction means uses the calculation results for each multivalued two-dimensional pattern to reconstruct the two-dimensional elemental image I _i,j (x, 7. The imaging device for three-dimensional images according to claim 6, characterized in that y) is obtained.

However, P _n (x,y) is the nth pattern array when the pattern is changed N times in the two-dimensional pattern array P(x,y), and S _i,j,n is the image sensor This is the reflected light intensity measured for the nth pattern at pixel (i,j). 3, characterized in that it comprises the imaging device for three-dimensional images according to any one of claims 1 to 7, and display means for displaying the three-dimensional image reconstructed by the three-dimensional image reconstruction means. Imaging display device for dimensional images.

Images