JP2024077420A - Imaging device - Google Patents

Abstract

[Problem] To provide an imaging device capable of improving the image quality of a reconstructed image by eliminating loss of pixel information in an encoded image and reducing crosstalk between reconstructed images. [Solution] An imaging device that restores a plurality of images along a time axis from an encoded captured image is characterized by comprising: a light modulation section that divides an image of a subject and modulates each divided light with an encoding pattern a plurality of times to generate a plurality of encoded images, an imaging section that obtains a plurality of the encoded images for each divided light with a single exposure and generates the encoded captured image, and a reconstruction processing section that reconstructs images in the number of encoding patterns along the time axis based on the encoded captured image and the encoding pattern. [Selected Figure] Figure 1

Description

The present invention relates to an imaging device, and in particular to an imaging device that uses a coding pattern.

In recent years, high frame rate video capture technology using coding patterns has been developed as a method for capturing video at rates faster than the frame rate of a camera. With this technology, a coding pattern is inserted between the subject and the camera, and the subject is captured through the coding pattern. During one exposure of the camera, the coding pattern is changed multiple times at regular intervals to modulate the subject image, and multiple coded images (coded images) are captured in one exposure to obtain coded captured images. By using a computer to reconstruct images from one coded captured image and multiple coding patterns, it is possible to restore images from one coded captured image along the time axis as many as the number of coding patterns, and capture video at rates faster than the frame rate of the camera.

In this reconstruction process, the inverse problem Y = AX is solved. The image captured by the camera is Y, and the high frame rate factor expressed using the encoded pattern is A. By solving the equation Y = AX that corresponds to the optical system, image information X for the number of patterns can be obtained, enabling a high frame rate.

In solving the above equations, compressed sensing is used for the calculation because the number of conditional expressions in the equations is less than the unknowns. In general, in simultaneous equations, more conditional expressions than the number of unknowns are required to find the unknowns. In compressed sensing, however, when the unknowns are sparse, the unknowns can be estimated even if the conditional expressions are less than the number of unknowns. In the case of images, the unknowns can be made sparse by converting them into frequency components using a basis that can compress some amount of information, such as discrete cosine transform (DCT), discrete wavelet transform (DWT), or discrete Fourier transform (DFT). Then, by performing conversion processing such as inverse DCT, inverse DWT, or inverse DFT on the sparse solution obtained using compressed sensing, images can be reconstructed for the number of coding patterns along the time axis.

Conventionally, various methods have been attempted for superimposing the coded pattern and for the reconstruction process in order to improve the image quality of the reconstructed image. For example, there is a method of mechanically moving a fixed pattern mask in a direction perpendicular to the optical axis to provide the subject with a coded pattern (Non-Patent Document 1). FIG. 12 shows an example of the configuration of an imaging device of the conventional technology described in Non-Patent Document 1. Light from the subject is collected by a lens 1 and imaged on a coded pattern mask (Coded Aperture) 2. The coded pattern mask 2 is moved in a direction perpendicular to the optical axis by, for example, a piezoelectric element (not shown), and provides the image of the subject with a coded pattern. The coded image is imaged on a camera 4 by a lens 3. A calculation processing device 5 reconstructs the image based on the obtained captured image and the coded pattern. In Non-Patent Document 1, the image is reconstructed by sparse estimation using a DCT basis.

Non-Patent Document 2 proposes a method of modulating a subject image by displaying an encoded pattern on a spatial light phase modulator. Non-Patent Document 2 reconstructs an image by sparse estimation using a basis that can provide a solution more sparsely than Non-Patent Document 1 by dictionary learning. Non-Patent Document 3 also discloses a method of providing an encoded pattern to a subject image using a digital micromirror device (DMD). Non-Patent Document 3 reconstructs an image by deep learning. Non-Patent Document 4 configures a camera using an image sensor having a circuit unit that records a signal output from one pixel of a photoelectric conversion unit in two memories. By recording the signal amount while switching between both memories multiple times during one exposure, a complementary encoded pattern is provided to the subject image on the image sensor side.

Patrick Llull, et al., “Coded aperture compressive temporal imaging”, Optics Express, Vol. 21, No. 9, pp. 10526-10545, (2013) Dengyu Liu, et al., “Efficient Space-Time Sampling with Pixel-Wise Coded Exposure for High-Speed Imaging”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.36, No.2, pp.248-260, (2013) Mu Qiao, et al., “Deep learning for video compressive sensing”, APL Photonics, Vol.5, No.3, 030801, (2020) Rahul Gulve, et al., “A 39,000 Subexposures/s CMOS Image Sensor with Dual-tap Coded-exposure Data-memory Pixel for Adaptive Single-shot Computational Imaging”, 2022 IEEE Symposium on VLSI Technology and Circuits, Digest of Technical Papers pp.78-79, (2022)

In Non-Patent Documents 1 to 3, one coded image is generated and captured by one camera. As a result, information in the areas of the subject that are masked by the coding pattern is lost in the coded image, and information in the lost areas is restored by estimation during reconstruction, but a decrease in image quality is unavoidable. Furthermore, if the masked areas of the coding pattern are reduced to reduce loss in the coded image in order to improve image quality, the amount of information that can be obtained from one coding pattern increases, but the similarity between the coding patterns increases, reducing the separability between the patterns and increasing crosstalk between the reconstructed images.

Figure 13 shows the relationship between the aperture ratio of a pattern mask and a reconstructed image according to conventional technology. Specifically, an image was restored (reconstructed) using four coding patterns using the method described in Non-Patent Document 1. (a) is the true value (image of the original subject), and (b) is a reconstructed image obtained by simulation when the aperture ratio of the coding pattern is 50%. Also, (c) is a reconstructed image obtained by simulation when the aperture ratio is 100%. It can be seen that the image quality is low at an aperture ratio of 50%, and that the images cannot be separated at an aperture ratio of 100% (full transmission).

On the other hand, the image sensor in Non-Patent Document 4 has a pixel count of 320 x 320 and can only capture images in the QVGA (Quarter Video Graphics Array) or lower standard, so its uses are limited due to display standards. Since a special camera is used for capturing images, it is not easy to change the pixel count. In addition, since two memories are arranged per pixel, it is not possible to simultaneously apply three or more encoding patterns to the subject image.

Therefore, in consideration of the above problems, the object of the present invention is to provide an imaging device that can eliminate loss of pixel information in an encoded image while simultaneously reducing crosstalk between reconstructed images, thereby improving the image quality of the reconstructed images.

In order to solve the above problems, an imaging device according to the present invention comprises:
(1) An imaging device that restores a plurality of images along a time axis from an encoded captured image, the imaging device comprising: a light modulation unit that divides an image of a subject into a plurality of divided lights and modulates each divided light a plurality of times with an encoding pattern to generate a plurality of encoded images; an imaging unit that obtains a plurality of the encoded images for each divided light with a single exposure and generates the encoded captured image; and a reconstruction processing unit that reconstructs images equal to the number of encoding patterns along the time axis based on the encoded captured image and the encoding pattern.

(2) In the imaging device of (1) above, it is further preferable that the light modulation unit modulates at least two of the multiple split light beams with complementary coding patterns.

(3) In the imaging device of (1) or (2) above, it is further preferable that the optical modulation unit is configured by connecting in multiple stages splitting and encoding units that split the incident light into two and modulate each of the light with a complementary encoding pattern.

(4) In any of the imaging devices (1) to (3) above, it is further preferable that the light modulation unit is configured with a digital micromirror device.

(5) In any of the imaging devices (1) to (3) above, it is preferable that the light modulation unit is further composed of a beam splitter and a spatial light modulator.

(6) In any of the imaging devices described above in (1) to (5), it is preferable that the reconstruction processing unit converts the image to be reconstructed into pixel components using a basis function obtained by discrete cosine transform, discrete wavelet transform, discrete Fourier transform, or dictionary learning, and generates a reconstructed image by solving a linear equation that estimates the pixel components using an optimization problem.

(7) In any of the imaging devices (1) to (6) above, it is preferable that the reconstruction processing unit further reconstructs the image using compressed sensing, CGI (computer ghost imaging), or machine learning.

The imaging device of the present invention can eliminate loss of pixel information in the encoded image while reducing crosstalk between reconstructed images, improving the image quality of the reconstructed images.

1 is a conceptual diagram of an imaging apparatus according to a first embodiment. 1 is a diagram showing an example of an actual configuration of the optical system of the image pickup apparatus according to the first embodiment. FIG. 2 is a diagram illustrating the principle of generating an encoded image by a DMD. 4A and 4B are diagrams showing the relationship between the display pattern of the DMD and the projected images of each camera. FIG. 2 is a diagram showing a linear equation for estimating an original image from an encoded captured image. 1A and 1B are diagrams illustrating a comparison of reconstructed images of the present invention and the prior art. FIG. 13 is a conceptual diagram of an imaging apparatus according to a second embodiment. 13 is a diagram showing an example of an actual configuration of an optical system of an image pickup apparatus according to a second embodiment. FIG. 11 is a conceptual diagram of an imaging device according to a modified example of the first embodiment. FIG. 13 is a conceptual diagram of an imaging device according to a modified example of the second embodiment. FIG. 1 is a conceptual diagram of an imaging device when the present invention is generalized. 1 is a configuration example of an imaging device according to a conventional technique. FIG. 1 is a diagram showing the relationship between the aperture ratio of a pattern mask and a reconstructed image according to the conventional technology.

The following describes an embodiment of the present invention with reference to the drawings.

First Embodiment
1 is a conceptual diagram of an imaging device according to a first embodiment. The imaging device includes a division/encoding unit 10, an imaging unit 20, a synchronization controller 30, and an arithmetic processing device (computer) 40. In this conceptual diagram, a subject is depicted as a moving image (multiple images). Each component will be described below.

The splitting/encoding unit 10 functions as a light modulation unit that splits an image from a subject into two, modulates each image (split light) with an encoding pattern, and generates multiple encoded images. Note that an image modulated with an encoding pattern (an image to which an encoding pattern has been applied) is called an encoded image. In this embodiment, it is desirable for the encoding patterns applied to the two images to be complementary patterns (patterns in which the valid and invalid parts are in an inverted relationship).

The imaging unit 20 is made up of multiple (two) imaging devices 21 and 22, and captures the encoded image output from the division and encoding unit 10 to generate an encoded captured image. Here, the encoded captured image is an image obtained by capturing multiple (a predetermined number of) encoded images generated along the time axis in a single exposure.

The synchronization controller 30 controls the synchronization of the division/encoding unit 10 and the operations of the imagers 21 and 22. Specifically, it controls so that the imagers 21 and 22 can capture a corresponding predetermined number of encoded images with a single exposure. Note that the synchronization controller 30 is not essential, and the calculation processing device 40 described below may also perform the functions of the synchronization controller 30.

The arithmetic processing device 40 receives the encoded captured image captured by the imaging unit 20 as input, and also acquires information on the encoding pattern (used for modulation) generated by the division/encoding unit 10, performs arithmetic processing, and generates reconstructed images corresponding to the number of encoding patterns. In other words, the arithmetic processing device 40 functions as a reconstruction processing unit that reconstructs (restores) a predetermined number of images along the original time axis.

Figure 2 shows an example of an actual configuration of the optical system of the imaging device of the first embodiment. The optical system in Figure 2 uses a DMD (digital micromirror device) 11 as the division and encoding unit 10, and includes the DMD 11, first and second cameras (imaging devices) 21, 22, first to third lenses 51 to 53, and an arithmetic processing device 40.

The first lens 51 forms an image of the subject on the DMD 11.

The DMD 11 is controlled by a synchronous controller 30 (or a processor 40) and can display any coding pattern, which encodes the image of the subject. Figure 3 is a diagram showing the principle of generating a coded image by the DMD 11. When a coding pattern is displayed on the DMD 11, light reflected by pixels (mirrors) 12 that are ON is imaged on the first camera 21, and light reflected by pixels (mirrors) 13 that are OFF is imaged on the second camera 22. In other words, the DMD 11 functions as a splitting and coding unit 10 (light modulation unit) that splits the subject image into two and codes each image.

Figure 4 is a diagram showing the relationship between the display pattern of the DMD 11 and the projected images of each camera. Each pixel (mirror) of the DMD 11 is either ON or OFF, and as shown in Figure 3, the image is distributed to the first and second cameras depending on whether it is ON or OFF. Therefore, as shown in Figure 4, the coded image (coded subject image) projected to the first camera 21 and the coded image projected to the second camera 22 are coded with an inverted pattern (complementary pattern). Therefore, the light imaged on the DMD 11 is projected to either the first camera 21 or the second camera 22, and no missing pixels are generated. Note that in this embodiment, a random pattern is used to control the ON and OFF of the DMD, but any pattern may be used, including an orthogonal basis such as a Hadamard pattern.

The second lens 52 projects and forms an image of the image encoded by the positive pattern of the DMD 11 on the first camera 21. The third lens 53 projects and forms an image of the image encoded by the negative pattern of the DMD 11 on the second camera 22. In this embodiment, a lens is used as an image transfer means for transferring the encoded subject image, but other optical elements may be used as a means for transferring the encoded image to the camera and forming an image.

The first camera (image capture device 1) 21 captures an image encoded with the positive pattern of the DMD 11 transferred by the second lens 52. The second camera (image capture device 2) 22 captures an image encoded with the negative pattern of the DMD 11 transferred by the third lens 53. It is possible to set as appropriate which of the first camera 21 and the second camera 22 captures the positive pattern and which of the negative patterns, respectively. The encoded captured images acquired by the cameras 21 and 22 are transferred to the calculation processing device 40.

In this embodiment, a synchronization signal is issued from the synchronization controller 30, and during one exposure of the cameras 21 and 22, p coded patterns are displayed on the DMD at high speed to modulate the light from the subject, and the cameras 21 and 22 simultaneously capture coded images for p patterns in one exposure, obtaining one coded captured image from each camera.

The arithmetic processing device 40 performs reconstruction processing based on the encoded captured image and the encoding pattern information, and generates reconstructed images equal to the number of encoding patterns (p).

In this embodiment, by using complementary patterns during encoding, even if there is an area (pixels in the OFF state) where subject information cannot be obtained by encoding using only the first camera 21, the second camera 22 can always obtain subject information in that area. This prevents the problem of not being able to obtain all subject information in the encoded image, resulting in errors (missing pixels) in reconstruction (see FIG. 13(b)). In addition, two encoded images (positive and negative) can be generated simultaneously using one DMD encoding pattern, improving the image quality of the reconstructed image.

(Reconstruction process)
Next, the image reconstruction process performed by the processor 40 will be described.

When the encoded captured image acquired by the first camera 21 is _Y1 , the matrix of the encoding pattern on the first camera side is _C1 , the encoded captured image acquired by the second camera 22 is _Y2 , the matrix of the encoding pattern on the second camera side is _C2 , and the number of reconstructed images corresponding to the number of encoding patterns along the time axis is O, the following formula (1) is satisfied when the optical system is applied to the formula. Note that _Y1 and _Y2 are combined in the column direction to form one matrix Y, and _C1 and _C2 are combined in the column direction to form one matrix C. Here, binary encoding patterns are used for the encoding pattern matrices _C1 and _C2 .

Next, if the DCT base is B and the DCT component of the reconstructed image is X, then O = BX, and the transformation can be performed as shown in the following equation (2). Note that CB = A.

The structure of each matrix is explained below. The encoded image generated by the segmentation and encoding unit 10 and the encoded captured images acquired by the cameras 21 and 22 can be set as two-dimensional images (M, N) [number of pixels M × N].

5 is a diagram showing a linear equation for estimating an original image (multiple reconstructed images) from an encoded captured image. When an image is converted into a one-dimensional signal (two-dimensional image components _a0,0 to _aM-1,N-1 are arranged in a row), one encoded captured image _Y1 is an MxN row and one column matrix { _Y1 , ..., _YMxN }, and two encoded captured images Y ( _Y1 and _Y2 ) are combined into an MxNx2 row and one column matrix { _Y1 , ..., _YMxNx2 }. Similarly, the reconstructed images O, the number of encoding patterns (p images) along the time axis, are one-dimensional signals with MxNxp rows and one column, which are expressed as a product of a DCT basis B with MxNxp rows and MxNxp columns and a DCT component X of a reconstructed image with MxNxp rows and one column, { _X1 , ..., _XMxNxp }.

One coding pattern is a block in which all pixels displayed on the DMD are converted into one-dimensional signals, with effective pixels (reflective mirrors) being set to 1 and ineffective pixels (non-reflective mirrors) being set to 0, and arranged on the diagonal of a matrix of M×N rows and M×N columns. The p coding patterns are p such blocks arranged in the row direction. The coding pattern matrix _C1 on the first camera side and the coding pattern matrix _C2 on the second camera side are then combined in the column direction.

In this way, the relationship between the p reconstructed images (number of pixels: M×N) and the encoded captured images (number of pixels: M×N) from the two cameras can be expressed as a linear matrix product as shown in FIG. 5 by treating X, _Y1 , and _Y2 as one-dimensional signals.

In reconstruction using p coding patterns, the coded captured images _Y1 , _Y2 , coding pattern matrices _C1 and _C2 , and DCT basis B are known, so if two two-dimensional vectors C and B are grouped together into vector A (=CB), the optical system can be expressed by a linear equation Y = AX, and Y and A are known, resulting in an inverse problem of finding X = { _X1 , ..., _XM×N×p } from these two elements.

In general, in a simultaneous equation with M×N×p unknowns (i.e., a simultaneous equation with M×N×p unknowns), M×N×p or more conditional expressions are required. However, by using compressed sensing, if the solution X is sparse (many of the components are almost close to 0), it is possible to obtain the conditional expressions even if the number of unknowns is less than the number of unknowns. In this embodiment, the image is made into a sparse solution X by DCT, so this is applied. The image reconstruction process using the DCT basis B of M×N×p rows and M×N×p columns may be performed by treating the basis B as a three-dimensional basis function that takes into account the time component and converting multiple two-dimensional images into a sparse solution by simultaneous processing at once, or by treating the basis B as multiple two-dimensional basis functions and repeating the process of converting each two-dimensional image with the two-dimensional basis function for multiple images. Depending on the characteristics of the subject image, a more effective process can be selected. In this embodiment, DCT basis (basis function of discrete cosine transform) is used as the image transformation basis for converting the image into pixel components with sparse values. However, other image transformation bases such as DWT basis (basis function of discrete wavelet transform) and DFT basis (basis function of discrete Fourier transform) as well as bases obtained by dictionary learning may also be used.

When solving the simultaneous equations, the optimization problem of the following equation (3), called LASSO (Least Absolute Shrinkage and Selection Operator) regression, was solved using the ADMM (Alternating Direction Method of Multipliers) method. There are several algorithms for solving optimization problems, and in addition to the ADMM method, other methods such as Newton's method, quasi-Newton method, coordinate descent method, ISTA (Iterative Shrinkage-thresholding algorithm), and FISTA (Fast Iterative Shrinkage-thresholding algorithm) may also be used.

In this embodiment, a simulation was performed to generate a reconstructed image with the number of coding patterns set to p = 4 and the coding patterns set to complementary patterns with an aperture ratio of 50%. FIG. 6 is a diagram showing a comparison of reconstructed images of the present invention and the prior art. (a) is the true value (image of the original subject), (b) is a reconstructed image obtained by simulation based on the prior art described in Non-Patent Document 1, and (c) is a reconstructed image obtained by simulation based on the present invention (first embodiment). In addition, the PSNR (Peak Signal to Noise Ratio) was calculated and displayed for each image as an evaluation of the image quality of each image. It was confirmed that the image quality of the reconstructed image was improved by performing reconstruction processing using an encoded captured image generated from multiple encoded images modulated with complementary patterns.

In addition, in this embodiment, the DMD and the cameras are aligned so that the pixels of each camera form a one-to-one image on the pixels of the DMD. However, if there is a positional deviation, this information can be expressed using the resampling filter R, and the equation to be solved can be the following equation (4), making it possible to correct for the positional deviation.

Second Embodiment
7 is a conceptual diagram of an imaging device according to the second embodiment. The imaging device includes a division unit 60, an encoding unit 70, an imaging unit 20, a synchronization controller 30, and an arithmetic processing device (computer) 40. Compared to the first embodiment, the division unit 60 and the encoding unit 70 are independent in the second embodiment. Each component will be described below, but the description of the components common to FIG. 1 will be simplified.

The splitting unit 60 splits the image from the subject into two. This splitting means generating (duplicating) two images of the same subject. The generated images are output to the encoding means 71 and 72 of the encoding unit 70.

The encoding unit 70 has encoding means 71, 72 that modulate each input image (split light) with an encoding pattern. The encoding unit 70 outputs the encoded image (an image given an encoding pattern) to the imaging unit 20. In this embodiment, it is preferable that the encoding patterns given to the images by the encoding means 71, 72 are complementary patterns (patterns in which the valid and invalid parts are in an inverted relationship). In this embodiment, the splitting unit 60 and the encoding unit 70 correspond to a light modulation unit that splits the subject image into multiple split lights, modulates each split light multiple times with the encoding pattern, and generates multiple encoded images.

The imaging unit 20 is made up of multiple (two) imaging devices 21 and 22, and captures the encoded image output from the encoding unit 70 to generate an encoded captured image.

The synchronous controller 30 controls the synchronous operation of the encoding means 71, 72 and the imagers 21, 22. Specifically, it controls the imagers 21, 22 so that they can capture a corresponding predetermined number of encoded images with a single exposure. The arithmetic processing device 40 may also function as the synchronous controller 30.

The arithmetic processing device 40 receives the encoded captured image captured by the imaging unit 20 as input, and also acquires information on the encoding pattern generated by the encoding unit 70 (used for modulation), performs arithmetic processing, and generates reconstructed images corresponding to the number of encoding patterns. In other words, the arithmetic processing device 40 functions as a reconstruction processing unit that reconstructs (restores) a predetermined number of images along the original time axis.

Figure 8 shows an example of an actual configuration of the optical system of the imaging device of the second embodiment. The optical system in Figure 8 uses a beam splitter (BS) 61 as the splitting section 60 and spatial light modulators as the encoding means 71, 72, and includes the beam splitter 61, the spatial light modulators 71, 72, first and second cameras (imaging elements) 21, 22, first to fifth lenses 51-55, and an arithmetic processing device 40.

The first lens 51 focuses the light from the subject onto the beam splitter 61.

The beam splitter 61 splits the light into two and directs each of the split lights toward the spatial light modulators 71 and 72. The second lens 52 focuses one of the split lights onto the spatial light modulator 71, and the third lens 53 focuses the other split light onto the spatial light modulator 72.

The spatial light modulators 71 and 72 each modulate the split light with an encoding pattern. The transmissive spatial light modulators 71 and 72 can be configured using, for example, a liquid crystal lens. The spatial light modulators 71 and 72 can each generate an arbitrary encoding pattern, but in this embodiment, the encoding patterns are complementary to each other.

The fourth lens 54 projects and forms an image of the image encoded by the spatial light modulator 71 on the first camera 21. The fifth lens 55 projects and forms an image of the image encoded by the spatial light modulator 72 on the second camera 22. Although a lens is used as an image transfer means for transferring the encoded subject image, other optical elements may be used.

The first camera (image capture device 1) 21 captures an image that has been transferred by the fourth lens 54 and encoded by the spatial light modulator 71. The second camera (image capture device 2) 22 captures an image that has been transferred by the fifth lens 55 and encoded by the spatial light modulator 72. The encoded captured images acquired by the cameras 21 and 22 are transferred to the processor 40.

In this embodiment, a synchronization signal is issued from the synchronization controller 30, p coding patterns are displayed at high speed on the spatial light modulators 71 and 72 during one exposure of the cameras 21 and 22 to modulate the light from the subject, and the cameras 21 and 22 simultaneously capture coded images for p patterns in one exposure, acquiring one coded captured image from each camera. The arithmetic processing device 40 performs reconstruction processing based on the coded captured images and the information on the coding patterns, and generates reconstructed images for the number of coding patterns (p).

In this embodiment, a transmissive spatial light modulator (liquid crystal lens) is used to provide the encoding pattern to the subject image, but a method of mechanically moving a fixed pattern mask or a reflective spatial light modulator may also be used. A random pattern is used as the encoding pattern, but any pattern including a Hadamard pattern may also be used as the encoding pattern.

In this embodiment, complementary coding patterns are displayed on the spatial light modulators 71 and 72, and modulation is performed so that the mask portions of the patterns are complemented by each other without excess or deficiency. However, the subject image may be modulated by increasing the opening of one or both of the complementary coding patterns. By increasing the opening, more pixel information can be obtained. In other words, even if the coding patterns are not strictly complementary, coding patterns can be created to the extent that they can eliminate loss of pixel information and reduce crosstalk between reconstructed images.

Also, in this embodiment, if there is a positional misalignment between the pixels of each camera and the pixels of the spatial light modulator, the information is represented using the resampling filter R, and the equation to be solved is the above-mentioned equation (4), making it possible to correct the positional misalignment.

For the second embodiment, a simulation was performed in which the number of coding patterns was set to p = 4 and complementary coding patterns with an aperture ratio of 50% were used. The results were similar to those shown in Fig. 6, and it was confirmed that the image quality of the reconstructed image was improved by performing reconstruction processing using an encoded captured image generated from multiple encoded images modulated with complementary patterns.

(Modification of the first embodiment)
9 is a conceptual diagram of an imaging device according to a modification of the first embodiment. In this modification, the division/encoding units 10 of the first embodiment are connected in multiple stages to construct an optical system and increase the number of generated encoded images. The imaging device includes division/encoding units 10 ₁ , 10 ₂ , and 10 ₃ , an imaging unit 20 (imagers 21 to 24), a synchronization controller 30, and a processing unit (computer) 40. Each component will be described below, but the description of elements common to FIG. 1 will be simplified.

The division/encoding units 10 ₁ , 10 ₂ , and 10 ₃ each have a function of dividing an input image (incident light) into two and modulating the two images (divided light) with an encoding pattern. In this embodiment, it is preferable that the encoding patterns given to the two images are complementary patterns. If the aperture ratio of the encoding pattern given by each division/encoding unit is 50%, the effective pixels of the encoded images output from the division/encoding units 10 ₂ and 10 ₃ are about 25% each because two stages of division/encoding are performed in FIG. 9. In this embodiment, the division/encoding units 10 ₁ , 10 ₂ , and 10 ₃ as a whole correspond to a light modulation unit.

The imaging section 20 is made up of a plurality of (four) imagers 21 to 24, and captures the encoded images output from the division and encoding sections 10 ₂ and 10 ₃ to generate an encoded captured image.

The synchronous controller 30 controls the synchronous operation of the division/encoding units 10 ₁ , 10 ₂ , 10 ₃ and each of the imagers 21 to 24, and controls each of the imagers 21 to 24 so that a predetermined number of encoded images can be captured by one exposure. The arithmetic processing device 40 may also function as the synchronous controller 30.

The arithmetic processing device 40 receives as input the encoded captured image captured by the imaging unit 20, and also acquires information on the encoding patterns generated by the division/encoding units ₁₀₁ , ₁₀₂ , and ₁₀₃ , performs arithmetic processing, and generates a reconstructed image corresponding to the number of encoding patterns.

In this modified example, by dividing the effective pixels into four coding patterns, the similarity between the coding patterns can be reduced, and crosstalk between the reconstructed images can be reduced.

In the modified example shown in FIG. 9, the segmentation/encoding units 10 are connected in two stages to separate the subject image into four encoding patterns, but it is also possible to connect the segmentation/encoding units 10 in three or more stages to separate the subject image into a greater number of encoding patterns.

(Modification of the second embodiment)
10 is a conceptual diagram of an imaging device according to a modification of the second embodiment. In this modification, a plurality of optical systems according to the second embodiment are formed in parallel to increase the number of splits of light and the number of generated encoded images.

The imaging device comprises a splitting unit 60, an encoding unit 70, an imaging unit 20, a synchronization controller 30, and an arithmetic processing unit (computer) 40. In a variation of the second embodiment, the splitting unit 60 splits the light from the subject into four, the encoding means 71-74 each modulate the split light with an encoding pattern, and the imaging devices 21-24 capture the image. Each component will be described below, but the description of the elements common to FIG. 2 will be simplified.

The division unit 60 divides the image from the subject into four. This division means generating (duplicating) four images of the same subject. The generated images are output to the encoding means 71 to 74 of the encoding unit 70.

The encoding unit 70 has encoding means 71-74 that modulate each input image with an encoding pattern. The encoding means 71-74 output the encoded images (images given an encoding pattern) to the imagers 21-24, respectively. In this embodiment, it is preferable that the encoding patterns given to the images by at least one set of encoding means (e.g., encoding means 71, 72) are complementary patterns. Furthermore, the encoding pattern given to the image by another set of encoding means (encoding means 73, 74) may be another complementary pattern. In this embodiment, the splitting unit 60 and the encoding unit 70 correspond to a light modulation unit that splits the subject image into multiple split lights, modulates each split light multiple times with the encoding pattern, and generates multiple encoded images.

The imaging unit 20 is made up of multiple (four) imaging devices 21 to 24, which each capture an encoded image output from the encoding unit 70 to generate an encoded captured image.

The synchronous controller 30 controls the synchronous operation of the encoding means 71-74 and each of the imagers 21-24, and controls each of the imagers 21-24 so that a predetermined number of encoded images can be captured with a single exposure. The arithmetic processing device 40 may also function as the synchronous controller 30.

The arithmetic processing device 40 receives the encoded captured image captured by the imaging unit 20 as input, and also acquires information on the encoding pattern generated by the encoding unit 70, performs arithmetic processing, and generates a reconstructed image corresponding to the number of encoding patterns.

In this modified example, by constructing an optical system that increases the number of light divisions and increasing the number of encoded images generated, the number of effective pixels in the encoded image can be increased without increasing crosstalk, thereby improving the image quality of the reconstructed image.

As described above, a modified version of the first embodiment and a modified version of the second embodiment have been described, but it is also possible to have a configuration in which the first embodiment and the second embodiment (or the modified version) are combined.

(Modification of reconstruction method)
The reconstruction method can use solutions other than compressed sensing, such as computerized ghost imaging (CGI) or machine learning.

(Generalized form of the present invention)
11 is a conceptual diagram of an imaging device when the present invention is generalized. The imaging device includes a division/encoding unit 10, an imaging unit 20, a synchronization controller 30, and a processing unit (computer) 40. Each component will be described below.

The splitting and encoding unit 10 has the function of splitting an image from a subject into N pieces (N is an integer of 2 or more) and modulating each image with a different encoding pattern. It is desirable that the encoding patterns given to at least two of these images are complementary to each other (patterns in which the valid and invalid parts are in an inverted relationship). This prevents missing pixels from the subject image. When splitting an image from a subject into N pieces (N is 3 or more), it is not necessary to use complementary patterns as the encoding patterns. If there are no missing pixels (loss of image information of the subject) in the entire N encoding patterns and the encoding patterns are set to be different from each other, a reconstructed image with good image quality can be obtained. The splitting and encoding unit 10 corresponds to an optical modulation unit.

The imaging unit 20 is made up of multiple (N) imaging devices (1 to N), captures the encoded image output from the division and encoding unit 10, and generates an encoded captured image.

The synchronous controller 30 controls the synchronization of the division/encoding unit 10 and the imaging unit 20, and controls each imaging device (1 to N) so that a predetermined number of encoded images can be captured with a single exposure. The arithmetic processing device 40 may also function as the synchronous controller 30.

The arithmetic processing device 40 receives the encoded captured image captured by the imaging unit 20 as input, and also acquires information on the encoding patterns generated by the division and encoding unit 10, performs arithmetic processing, and generates a reconstructed image corresponding to the number of encoding patterns. The arithmetic processing device 40 functions as a reconstruction processing unit.

In this way, by using multiple encoded images, the accuracy of reconstruction can be increased, leading to improved image quality.

The imaging device of the present invention can eliminate loss of pixel information in the encoded image while reducing crosstalk between reconstructed images, improving the image quality of the reconstructed image. Furthermore, cameras of any structure can be used as the imaging device (camera) in the imaging section, allowing for a high degree of freedom in constructing an optical system. When the imaging device of the present invention is used for capturing moving images, moving images can be obtained at a frame rate higher than the frame rate of the camera.

In the above embodiment, the configuration and operation of the imaging device have been described, but the present invention is not limited to this, and may be configured as an imaging method for restoring multiple images along a time axis from an encoded captured image. In other words, the present invention may be configured as an imaging method for acquiring encoded captured images and generating reconstructed images according to the image and data flows in Figures 1, 7, and 9 to 11.

The above-mentioned embodiments have been described as representative examples, but it will be apparent to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present invention. Therefore, the present invention should not be interpreted as being limited by the above-mentioned embodiments, and various modifications or changes are possible without departing from the scope of the claims. For example, the functions contained in each block, step, etc. described in the embodiments can be rearranged so as not to cause logical inconsistencies, and multiple constituent blocks, steps, etc. can be combined into one or divided.

1 Lens 2 Encoded pattern mask 3 Lens 4 Camera 5 Processing device 10 Division and encoding unit 11 DMD
20 Imaging unit 21 to 24 Imaging device (camera)
30 Synchronization controller 40 Processing device 51 to 55 Lens 60 Splitting section 61 Beam splitter 70 Encoding section 71 to 74 Encoding means

Claims (7)

An imaging device that reconstructs a plurality of images along a time axis from an encoded captured image, comprising:
a light modulation unit that divides an object image into a plurality of divided light beams and modulates each divided light beam a plurality of times with an encoding pattern to generate a plurality of encoded images;
an imaging unit that acquires a plurality of the encoded images for each of the divided beams in one exposure and generates the encoded captured image;
a reconstruction processing unit that reconstructs images corresponding to the number of encoding patterns along a time axis based on the encoded captured image and the encoding pattern;
An imaging device comprising: 2. The imaging device according to claim 1,
The imaging device, wherein the light modulation section modulates at least two of the plurality of split light beams with complementary coding patterns. 3. The imaging device according to claim 2,
An imaging device, wherein the light modulation section is configured by connecting in multiple stages division/encoding sections which divide incident light into two and modulate each of the light with a complementary encoding pattern. 3. The imaging device according to claim 2,
11. An imaging device, wherein the light modulation section is composed of a digital micromirror device. 3. The imaging device according to claim 2,
11. An imaging device, wherein the light modulation section is composed of a beam splitter and a spatial light modulator. 6. The imaging device according to claim 1,
The imaging device, characterized in that the reconstruction processing unit converts the image to be reconstructed into pixel components using a basis function obtained by discrete cosine transform, discrete wavelet transform, discrete Fourier transform, or dictionary learning, and generates a reconstructed image by solving a linear equation that estimates the pixel components by an optimization problem. 6. The imaging device according to claim 1,
The imaging device, wherein the reconstruction processing unit reconstructs an image using compressed sensing, CGI (computer ghost imaging), or machine learning.

Images