JP2025123735A - small optical device - Google Patents

Abstract

To provide a small optical device that improves the contrast ratio by introducing a differential detection method into an imaging reconstruction process using a photodetector, is equipped with SPI, and suppresses various types of noise using deep learning.SOLUTION: A differential detection method is introduced into a reconstruction process of imaging using a single-pixel photodetector. This makes it possible to improve the contrast ratio and obtain clearer reconstructed images. Furthermore, image quality is improved by incorporating single-pixel imaging (SPI) into the optical system and suppressing noise in the obtained detection data using deep learning.SELECTED DRAWING: Figure 1

Description

Detailed Description of the Invention

The present invention relates to a compact optical device that is highly accurate, high contrast, and compact, and that improves the contrast ratio by introducing a differential detection method into the imaging reconstruction process using a photodetector.The present invention also relates to a compact optical device that is equipped with SPI and suppresses various types of noise through deep learning.

Single-pixel imaging (SPI), an imaging technology using a single-pixel detector, has attracted attention (Non-Patent Document 1). SPI filters an object with a spatially distributed coding pattern, acquires the filtered light intensity as an optical correlation signal, and then sequentially changes the coding pattern to acquire the optical correlation signal with each coding pattern. In principle, if the obtained optical correlation signal is and the matrix where each pattern corresponds to each matrix is , the relationship with the target object x can be expressed as I = HX. Information about the target object x can be reconstructed by solving the inverse matrix of x. Alternatively, a well-known method is to analytically solve the inverse problem using a coding pattern generated from an orthogonal basis matrix. SPI has advantages such as imaging with a wide wavelength range from X-rays to THz and with weak light, and being more noise-resistant than 2D image sensors. However, imaging with high noise levels results in poor image accuracy. For this reason, optical correlation imaging was developed, and a proposal was made to improve noise resistance by incorporating deep learning into the reconstruction process (Patent Document 1). Adaptive optics is also a noise-correcting technology. The basic components of adaptive optics are a wavefront sensor and a wavefront correction element, and the wavefront state is measured by the wavefront sensor and the wavefront correction element is controlled based on the measurement (Non-Patent Document 2). However, these devices have drawbacks such as being very expensive and making it difficult to measure the wavefront state in dark environments.

Duarte, M. F., Davenport, M. A., Takhar, D., Laska, J. N., Sun, T., Kelly, K. F., & Baraniuk, R. G. (2008). Single-pixel imaging via compressive sampling. IEEE signal processing magazine, 25(2), 83-91.

H. W. John, “Adaptive Optics for Astronomical Telescopes”. New York: Oxford University Press, 1998.

Patent Publication No. 2021-132330

Conventional SPI (Single-pixel imaging) technology has difficulty obtaining clear images from weak signals due to the influence of noise. To correct this noise, adaptive optics was used, and noise reduction technology existed using the wavefront sensor and wavefront correction element that comprise its basic components. However, these devices were very expensive, and had issues such as difficulty in measuring the wavefront state in dark environments.

To solve this problem, we introduce a differential detection method into the reconstruction process of imaging using a single-pixel photodetector. This improves the contrast ratio and enables the acquisition of clearer reconstructed images. Furthermore, we improve image quality by incorporating SPI into the optical system and suppressing noise in the obtained detection data using deep learning.

The deep learning-based SPI system of this invention consists of an optical correlation process in which SPI is installed in various optical systems, and a deep learning-based reconstruction process. In the optical correlation process, the target object is captured by the optical system, and the object image is illuminated onto an encoded pattern to obtain an optical correlation signal using a photodetector. In the reconstruction process, a signal obtained by taking the difference between the optical correlation signals obtained in the optical correlation process is input into deep learning. Using this method has the effect of suppressing image degradation caused by various noise components that affect the imaging of the target object, and is expected to enable the acquisition of highly accurate reconstructed images. Furthermore, since the optical system used does not need to be configured with high precision, it can be made smaller and less expensive.

FIG. 1 is a diagram showing an example of an overall system configuration including a compact optical device according to an embodiment. FIG. 2 is a software configuration diagram of the compact optical device according to the embodiment. 10A to 10C are diagrams showing a specific technique for manufacturing a compact optical device according to an embodiment. FIG. 1 is a diagram showing a deep learning technique according to the present invention. A conceptual diagram for demonstrating the validity of the patent of the present invention. FIG. 1 shows the patent validity of the present invention. FIG. 1 illustrates another embodiment of the present patent.

Modes for carrying out the invention

Embodiments of the present invention will be described below with reference to the drawings. Identical components throughout the drawings will be assigned the same reference numerals, and duplicate explanations will be omitted. In the following embodiments, a compact optical device according to the present invention will be described as an example.

Figure 1 shows an example of the overall configuration including a compact optical device 1 according to an embodiment of the present invention. The compact optical device 1 in Figure 1 is composed of an optical system 11 and a system control/processing function 12. The optical system 11 in the figure is an example of a reflecting telescope-type optical system that magnifies light from a distant object using a reflective optical system, but is not limited to this. It can also be a binocular type, a microscope type such as a transmission type, or a camera device that captures images. The optical system in this compact optical device 1 is composed of a Maksutov-Cassegrain telescope, a DMD (Digital Mirror Device) 10, and two photodetectors 8a and 8b. In this case, the target object 4 is assumed to be a distant star. A weak optical signal from the star enters a focusing lens 5, is reflected by a primary mirror 6, and then by a secondary mirror 7, and is incident on the focal plane. Three lenses 9a, 9b, and 9c are inserted between the focal plane of the Maksutov-Cassegrain telescope and the DMD 10, and these lenses are movable. This makes it possible to arbitrarily adjust the size and focal position of the image of the target object 4 illuminated on the DMD 10. In addition, by taking advantage of the fact that light incident on the DMD 10 is reflected in two directions, photodetectors 8a and 8b are installed at the end of each reflection.

The system control and processing function 12 controls the operation of the above-mentioned functions and performs calculations. It is composed of a CPU 13, ROM 14, RAM 15, external memory 16, and system bus 18. It also includes a display unit 17 that displays detailed images of the target object. The CPU 13 turns the power on and off for the photodetectors 8a, 8b, and DMD 10 in the optical system 11, performs detailed control, sets various settings, controls the program that performs the image processing calculations for the target object 4 described below, and sets various commands. The ROM 14 stores system software, processing calculation programs, and their initial values. The RAM 15 is a memory space for executing various software programs performed by the CPU 13, or for recording data and settings temporarily recorded during calculations, etc. The external memory 16 is memory for retaining data that cannot be stored in the ROM 14 or RAM 15, as well as previous data. It can also be used to store software updates, new settings, etc. Image data of the target object 4 processed and calculated by the CPU 13 (described below) is displayed on the display unit 17. This data is sent and received via the system bus 18. The data may be sent and received externally via the Internet 19.

FIG. 2 is a diagram showing the software configuration of the compact optical device 1 of this embodiment, which comprises a CPU 13, a ROM 14, a RAM 15, an external memory 16, and a system bus 18.
The CPU 13 is a microprocessor unit that controls the entire compact optical device 1 in accordance with a predetermined program. The system bus 18 is a data communication path for transmitting and receiving data between the CPU 13 and each part in the compact optical device 1.

ROM (Read Only Memory) 14 is memory that stores basic operating programs such as the operating system and other application programs. Rewritable ROM such as EEPROM (Electrically Erasable Programmable ROM) or flash ROM is used, and by updating the programs stored in ROM 14, it is possible to upgrade or expand the functionality of the basic operating programs and other application programs.

RAM (Random Access Memory) 15 serves as a work area when the basic operation program and other application programs are executed. Specifically, for example, basic operation program 14a stored in ROM 14 is expanded into RAM 15, and the CPU 13 then executes the expanded basic operation program to form a basic operation execution unit 15a. For simplicity's sake, the following description will be given assuming that the basic operation execution unit 15a controls each unit when the CPU 13 expands the basic operation program 14a stored in ROM 14 into RAM 15 and executes it. Similar descriptions will be used for other application programs.

The light intensity processing execution unit 15b performs various software processes on the electrical signals converted from the light intensities of the two photodetectors 8a and 8b, including correction of the light intensity, which is expressed as the level of the electrical signal, and edge processing and noise removal to make the converted electrical signals clearer. The optical correlation calculation execution unit 15c, deep learning calculation execution unit 15d, and reconstruction calculation execution unit 15e will be described in detail later. These units perform various light propagation calculations on the light intensities obtained by the photodetectors 8a and 8b, thereby creating two optical correlation signals from each photodetector of the target object 4 in new detail. The deep learning calculation execution unit 15d uses the two optical correlation signals obtained by the optical correlation calculation execution unit 14c, calculates the difference between the two optical correlation signals, and applies this to deep learning. Regarding various data collection tasks in deep learning, various necessary data is obtained from the Internet 19 via the system bus 18 and used for calculations. This data does not always need to be obtained via the Internet 19; it may be stored in advance in external memory 16. New data may also be re-registered through learning. The reconstruction calculation execution unit 15e performs a process of reconstructing an image of the original object using the results obtained by the deep learning execution unit 15d. The setting value execution unit 15f for various components controls various setting values and sets initial values for the photodetectors 8a, 8b, lenses 9a, 9b, 9c, DMD 10, display unit 17, etc. The image display execution unit 15g performs a process of displaying on the display unit 17 a detailed image of the target object reconstructed from the data calculated by the reconstruction calculation execution unit 15e (described below). The temporary storage area 15h is an area for temporarily storing data and the like during each of the above processes.

2, the operation of the object shape measuring apparatus 1 of this embodiment is controlled by a light intensity processing program 16b, an optical correlation program 16c, a deep learning program 16d, a reconstruction calculation program 16e, a setting program 16f for various components, and an image display program 16g, which are mainly stored in the external memory 16, expanded into the RAM 15, and executed by the CPU 13: a light intensity calculation execution unit 15b, an optical correlation calculation execution unit 15c, an optical correlation calculation execution unit 15c, a deep learning calculation execution unit 15d, a reconstruction calculation execution unit 15e, a setting value execution unit 15f for various components, and an image display execution unit 15g. The various information/data storage area 16a is an area for storing initial values of various functions, etc. The aforementioned light intensity calculation execution unit 15b, optical correlation calculation execution unit 15c, optical correlation calculation execution unit 15d, deep learning calculation execution unit 15d, reconstruction calculation execution unit 15e, and image display execution unit 15g may be performed by hardware blocks that realize some or all of their operations using hardware.
The ROM 14 and RAM 15 may be integrated with the CPU 13. The ROM 14 may not be an independent configuration as shown in Fig. 2, but may use a partial storage area within the external memory 16. The RAM 15 is also provided with a temporary storage area for temporarily storing data as necessary when various application programs are executed.

The external memory 16 may store various operational setting values for the compact optical device 1, position information and various other information for the compact optical device 1, some or all of the data such as images and information captured by the compact optical device 1, and other programs.

A portion of the external memory 16 may replace all or part of the functions of the ROM 14. Furthermore, the external memory 16 must retain the stored information even when power is not supplied to the compact optical device 1. Therefore, devices such as flash ROM, SSD (Solid State Drive), and HDD (Hard Disc Drive) are used.

Next, the basic technology of the present invention will be explained using equations and Figure 3. The optical correlation process is configured with a Maksutov-Cassegrain telescope 2, a DMD 10, two photodetectors 8a and 8b, and three lenses 9a, 9b, and 9c. In Figure 3, optical signals from the target object 4 enter the two photodetectors 8a and 8b through the telescope 2, and optical correlation signals _S1 and _S2 are formed. In the optical system for the optical correlation process, lens 9c is inserted between the focal plane of the Maksutov-Cassegrain telescope 2 and the DMD 10. This lens 9c is movable, allowing the size of the object image projected onto the DMD 10 to be adjusted as desired. This control is performed by the CPU 13. Taking advantage of the fact that light incident on the DMD 10 is reflected in two directions, photodetectors 8a and 8b are installed at the ends of each reflection.

The deep learning-based SPI system in Figure 1 consists of an optical correlation process using an optical system equipped with SPI 3 on telescope 2, and a deep learning reconstruction process. In the optical correlation process, a distant target object 4 is captured by telescope 2, and the object image is illuminated onto an encoded pattern to obtain an optical correlation signal using photodetectors 8a and 8b. In the reconstruction process, a signal obtained by calculating the difference between the optical correlation signals obtained in the optical correlation process is input to deep learning. This method suppresses image degradation caused by noise such as atmospheric turbulence and vibration of the telescope body, which affect the imaging of distant target objects, making it possible to obtain highly accurate reconstructed images. In the reconstruction process in Figure 3, an optical correlation signal , calculated by calculating the difference between the optical correlation signals obtained from two photodetectors 8a and 8b in the optical correlation process, is input.

Here, _S1 and _S2 are the optical correlation signals acquired by the respective photodetectors 8a and 8b. Deep learning is performed using the DNN (Deep Neural Network) method. A conceptual diagram is shown in Figure 4. This system consists of five transposed convolutional layers and three convolutional layers, and learns to reduce the effects of noise, such as atmospheric turbulence and telescope vibration, from the acquired image so as to approach the target image. Regarding noise caused by atmospheric turbulence and telescope vibration, deep learning can identify and remove random noise components such as atmospheric turbulence and specific frequency components characteristic of telescope vibration, based on the characteristics and information of various noise components obtained via the Internet 19. For example, in the case of image stabilization and vibration noise correction, conventional methods have involved applying inverse component correction based on vibration sensor information to optical lenses, etc., to achieve image stabilization and vibration noise reduction. In contrast to this, the present invention collects the characteristics of camera shake and target vibration noise, and the characteristics of random noise such as atmospheric turbulence, which are already available on the Internet 19, and by performing deep learning on this, it becomes possible to identify and remove noise components without a sensor. Furthermore, once a certain level of learning has been achieved, the results can be saved in the external memory 16, etc., enabling faster processing.

Next, we present a specific example to demonstrate the effectiveness of this patent. This will be explained using Figure 5. Figure 5 is a conceptual diagram of an optical system that mimics the propagation of light during astronomical observation. Using this optical system for the optical correlation process, we evaluated a system using differential detection. We used 2,048 MNIST (Mixed National Institute of Standards and Technology database) images as the target objects and 1,024 Hadamard patterns as the encoding patterns. MNIST is a dataset consisting of pixel numbers from 0 to 9, totaling 60,000 images. For the evaluation, we expanded the MNIST data to 1024 pixels and performed binarization. We also generated a spatial phase distribution of atmospheric turbulence (noise components) based on Kolmogorov turbulence theory and varied it over time to represent atmospheric turbulence. We simulated observations using a 51-cm telescope at the summit of Mauna Kea in Hawaii. Figure 6 shows the reconstruction results of this imaging system. _S1 and _S2 are the results of reconstruction using each photodetector, while _Sdiff is the result using differential detection. For comparison, we show the reconstruction results using the SPI reconstruction method without deep learning. In the reconstructed image using SPI, the signal-to-noise ratio was improved by using the differential detection method, and the contrast of the target object was improved compared to when only _S1 or _S2 was used ( _Sdiff ). Furthermore, when atmospheric turbulence was added, the use of deep learning made it possible to reconstruct the target object to the extent that its shape could be identified. Furthermore, we were able to confirm that the differential detection method ( _Sdiff ) showed improved reconstruction accuracy compared to images reconstructed from the optical correlation signal of only one photodetector ( _S1 , _S2 ).

Next, another embodiment of the present invention will be described with reference to FIG. 7. In FIG. 7, the optical system is assumed to be, for example, binoculars or a surveillance camera, and the target object 4 is assumed to be, for example, a human. In this case, the optical system is not the reflecting telescope type shown in FIG. 1, so light entering the objective lens 20 passes through a filter 20 toward the focal plane. While white light entered the focal plane in FIG. 1, in FIG. 7, the filter directs only light of specific wavelengths toward the focal plane. This filter may not only transmit a single wavelength, but may also transmit only the three primary colors of light, i.e., red, green, and blue, with time delays between each filter. In this case, the filter replacement is controlled by the CPU 13, using a three-filter selection function (not shown). This selection period may be as fast as 1/30 seconds or 1/60 seconds. For example, a color image of the target object 4 can be created from three images, one reconstructed with red light as shown in FIG. 1 and FIG. 2, as well as the other reconstructed with green and blue light. Although the processing speed of the reconstructed image depends in part on the CPU power and the communication capacity with the Internet 19, high-speed processing is possible and color moving images can also be handled by storing the characteristics of the noise components shown above in advance in the RAM 15 or external memory 16. This is particularly effective for applications such as night vision cameras and surveillance cameras where improved contrast is required.

1: Small optical device, 2: Telescope, 3: SPI, 4: Target object, 5: Condenser lens, 6: Primary mirror, 7: Secondary mirror, 8a: Photodetector 1, 8b: Photodetector 2, 9a: Lens 1, 9b: Lens 2, 9c: Lens 3, 10: DMD, 11: Optical system, 12: System control and processing function, 13: CPU, 14: ROM, 15: RAM, 16: External memory, 17: Display unit, 18: System bus, 19: Internet, 20: Objective lens, 21: Filter

Claims (7)

A compact optical device having the function of receiving an optical signal from a target object using an optical system consisting of multiple lenses or mirrors, etc., and at least two single-pixel photodetectors, acquiring the light intensity of the target object from the single-pixel photodetectors as an optical correlation signal, and reconstructing an image of the target object based on the obtained optical correlation signal, characterized in that in reconstructing the image, the light intensities of at least two single-pixel photodetectors are used and the image is reconstructed by detecting the difference between them.
A compact optical device that receives an optical signal from a target object using an optical system consisting of multiple lenses or mirrors, etc., and at least two single-pixel photodetectors, acquires the light intensity of the target object from the single-pixel photodetectors as an optical correlation signal, and reconstructs an image of the target object based on the obtained optical correlation signal, characterized in that, in reconstructing the image, noise components of the target object contained in the image are removed using deep learning.
3. The compact optical device according to claim 1, wherein both difference detection and deep learning are performed in image reconstruction.
The compact optical device according to claims 2 and 3, characterized in that the deep learning data used in image reconstruction is acquired via a network.
The compact optical device according to claims 2 and 3, characterized in that deep learning used in image reconstruction is performed using data stored in a memory within the compact optical device.
6. The compact optical device according to any one of claims 1 to 5, wherein a filter is disposed in the optical path, and an image is reconstructed using monochromatic light.
7. The compact optical device according to claim 6, wherein the filters arranged in the optical path are red, green, and blue filters, and the image is colorized using these three monochromatic lights.