WO2023074759A1

WO2023074759A1 - Imaging apparatus, vehicle lamp fitting, and vehicle

Info

Publication number: WO2023074759A1
Application number: PCT/JP2022/040006
Authority: WO
Inventors: 輝明鳥居; 祐太春瀬
Original assignee: 株式会社小糸製作所
Priority date: 2021-10-27
Filing date: 2022-10-26
Publication date: 2023-05-04

Abstract

In an imaging apparatus 100, an illuminating apparatus 110 sequentially emits, M times (M≥2), illuminating light having a spatially random intensity distribution. An arithmetic processing apparatus 130: divides the M emissions into a plurality k (k≥s) of units; performs a correlation calculation for each divided unit to generate an intermediate image; and combines the k intermediate images obtained for the k units to generate a restored image G (x, y).

Description

Imaging device, vehicle lamp, vehicle

The present disclosure relates to an imaging device using ghost imaging.

An object identification system that senses the position and type of objects around the vehicle is used for automated driving and automatic control of headlamp light distribution. An object identification system includes a sensor and a processor that analyzes the output of the sensor. Sensors are selected from cameras, LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), millimeter-wave radar, ultrasonic sonar, etc., taking into consideration the application, required accuracy, and cost.

As one of the imaging devices (sensors), one that uses the principle of ghost imaging is known. Ghost imaging illuminates an object while randomly switching the intensity distribution (pattern) of illumination light, and measures the light detection intensity of the reflected light for each pattern. The light detection intensity is the integral of energy or intensity over a plane, not the intensity distribution. Correlation calculation between the corresponding pattern and the light detection intensity is then performed to reconstruct a restored image of the object.

Japanese Patent No. 6412673 International publication WO2020/218282 International publication WO2021/079810

As a result of studying the imaging apparatus, the inventors came to recognize the following problems. The correlation function of Equation (1) is used for correlation in ghost imaging. _Ir of the illumination light is the r-th (r=1, 2, . . . , M) intensity distribution, and _br is the detected intensity value obtained when the illumination light having the r-th intensity distribution is irradiated.

FIG. 1 is a time chart showing sensing of one frame of an imaging device. As can be seen from equation (1), correlation calculation requires an average value of M detection intensities b ₁ to b _M obtained for M illumination light irradiations.

Therefore, after completing M irradiations, the average value is calculated and the correlation calculation is started. Here, a photodetector is used for measuring the photodetection intensity. Ambient light (ie, noise component) enters the photodetector in addition to the reflected light (ie, signal component) from the object. If the intensity of ambient light is constant during the irradiation period T during which irradiation is performed M times, the influence of noise is cancelled. However, when the intensity of ambient light fluctuates during the irradiation period T, the effect of noise is not canceled, and the image quality of the reconstructed image deteriorates. For example, when the imaging apparatus is mounted on a moving body such as an automobile, this problem becomes significant when the distance between the noise light source and the moving body changes.

The present disclosure has been made in this context, and one exemplary purpose of certain aspects thereof is to provide an imaging apparatus that is more tolerant of temporally varying noise.

An imaging device according to one aspect of the present disclosure includes an illumination device that illuminates a field of view with illumination light having a spatially random intensity distribution, a photodetector that measures reflected light from an object, and an output of the photodetector. and an arithmetic processing unit that performs correlation calculation between the detected intensity and the intensity distribution of the illumination light and reconstructs a restored image of the object, and is switchable between a first mode and a second mode, and performs arithmetic processing in the first mode. The apparatus generates an intermediate image by performing correlation calculation each time the illumination light is irradiated n times (n≧2), synthesizes k intermediate images (k≧2), generates a restored image, and generates a second In the mode, the illumination light contains pairs of patterns whose intensity distributions are complementary.

According to an aspect of the present disclosure, dynamic switching between the two modes according to the measurement scene can improve resistance to temporally varying noise.

4 is a time chart showing sensing of one frame by the imaging device; 1 is a diagram showing an imaging device according to Embodiment 1; FIG. 4 is a time chart for explaining the operation of the imaging apparatus in TDGI mode; FIG. 11 is a diagram illustrating another sequence in TDGI mode; FIG. 5(a) is a diagram showing the influence of noise in batch restoration, and FIG. 5(b) is a diagram showing the influence of noise in division restoration. FIG. 6(a) shows a target image, and FIG. 6(b) shows an image obtained by batch restoration and division restoration when noise does not exist. FIG. 10 is a diagram showing a restored image G(x, y) obtained by collective restoration and divisional restoration when linear noise exists; 4 is a time chart for explaining the IPGI mode operation of the imaging device; FIGS. 9A and 9B are diagrams showing restored images obtained by the TDGI mode and the IPGI mode. 2 is a block diagram of an imaging device according to Embodiment 2; FIG. FIGS. 11A and 11B are diagrams showing restored images obtained by the TDGI mode, IPGI mode, and TDGI-IPGI hybrid mode. 1 is a block diagram of an object identification system; FIG. 1 is a block diagram of a vehicle with an object identification system; FIG. 1 is a block diagram showing a vehicle lamp equipped with an object detection system; FIG.

An overview of some exemplary embodiments of the present disclosure is provided. This summary presents, in simplified form, some concepts of one or more embodiments, as a prelude to the more detailed description that is presented later, and for the purpose of a basic understanding of the embodiments. The size is not limited. This summary is not a comprehensive overview of all possible embodiments, and it is intended to neither identify key elements of all embodiments nor delineate the scope of some or all aspects. For convenience, "one embodiment" may be used to refer to one embodiment (example or variation) or multiple embodiments (examples or variations) disclosed herein.

An imaging device according to one embodiment includes an illumination device that illuminates a field of view with illumination light having a spatially random intensity distribution, a photodetector that measures reflected light from an object, and detection based on the output of the photodetector. and an arithmetic processing unit that performs correlation calculation between the intensity and the intensity distribution of the illumination light and reconstructs the restored image of the object, and is switchable between the first mode and the second mode. In the first mode, the arithmetic processing unit performs correlation calculation each time illumination light is irradiated n times (n≧2) to generate an intermediate image, and synthesizes the latest k intermediate images (k≧2). , to generate the restored image. In the second mode, the illumination light comprises pairs of patterns whose intensity distributions are complementary.

In the first mode, the correlation calculation is performed by dividing n irradiations into units, so that the amount of noise change per correlation calculation can be reduced, the restoration accuracy can be increased, and the image quality can be improved. In addition, since the correlation calculation can be started without waiting for the completion of M irradiations, the restoration image generation time can be shortened. In the second mode, it is possible to cancel the noise when measuring with illumination light having a certain intensity distribution and when measuring with illumination light complementary thereto, thereby improving the image quality.

In this specification, the first mode is called "Time Divisional Ghost Imaging (TDGI) mode", and the second mode is also called "Inverse Pattern Ghost Imaging (IPGI) mode".

"The intensity distribution is random" in this specification does not mean that it is completely random, but it is sufficient if it is random enough to reconstruct an image in ghost imaging. Therefore, "random" in this specification can include a certain degree of regularity therein. Also, "random" does not require unpredictability, but may be predictable and reproducible.

In one embodiment, in the first mode, each time a new intermediate image is generated, the arithmetic processing unit may synthesize the latest k intermediate images to generate a restored image. This makes it possible to increase the update rate of the restored image in the first mode.

In one embodiment, in the second mode, the processing unit performs correlation calculation each time the illumination light is emitted n times to generate an intermediate image, synthesizes the latest k intermediate images, and generates a restored image. may be generated. This makes it possible to increase the update rate of the restored image even in the second mode.

In one embodiment, it may be possible to switch between the first mode and the second mode based on the output of the photodetector. Image quality is improved by selecting the first mode when the noise included in the output of the photodetector is relatively small and selecting the second mode when the noise included in the output of the photodetector is relatively large. It can be improved.

In one embodiment, it may be possible to switch between the first mode and the second mode based on the quality of the restored image. When the image quality of the restored image obtained in the first mode deteriorates, the image quality can be improved by switching to the second mode.

In one embodiment, it may be possible to switch between the first mode and the second mode based on the identification rate of a classifier (classifier) based on the restored image. When the identification rate based on the restored image obtained by the first mode is lowered, it is assumed that the image quality is degraded, and switching to the second mode can improve the image quality and thus the identification rate.

(embodiment)
BEST MODE FOR CARRYING OUT THE INVENTION The present invention will be described below based on preferred embodiments with reference to the drawings. The same or equivalent constituent elements, members, and processes shown in each drawing are denoted by the same reference numerals, and duplication of description will be omitted as appropriate. Moreover, the embodiments are illustrative rather than limiting the invention, and not all features and combinations thereof described in the embodiments are necessarily essential to the invention.

(Embodiment 1)
FIG. 2 is a diagram showing the imaging device 100 according to the first embodiment. Imaging device 100 is a correlation function image sensor that uses the principle of ghost imaging (also called single-pixel imaging), and includes illumination device 110 , photodetector 120 and arithmetic processing device 130 . Imaging device 100 is also referred to as a quantum radar camera.

The illumination device 110 is a pseudo thermal light source, generates illumination light S1 having a spatial intensity distribution I(x, y) that can be regarded as substantially random, and irradiates the object OBJ. The illumination light S1 is sequentially irradiated a plurality of M times while randomly changing its intensity distribution. The number of times of irradiation M is the number of times that the original image can be restored in a conventional imaging method (batch restoration described later).

Illumination device 110 includes light source 112 , patterning device 114 and pattern generator 132 . Light source 112 produces light S0 having a uniform intensity distribution. A laser, a light emitting diode, or the like may be used as the light source 112 . The wavelength and spectrum of the illumination light S1 are not particularly limited, and may be white light having multiple or continuous spectra, or monochromatic light including a predetermined wavelength. The wavelength of the illumination light S1 may be infrared or ultraviolet.

The patterning device 114 has a plurality of pixels arranged in a matrix, and is configured to be able to spatially modulate the light intensity distribution I based on a combination of ON and OFF of the plurality of pixels. In this specification, a pixel in an ON state is called an ON pixel, and a pixel in an OFF state is called an OFF pixel. In the following description, for ease of understanding, it is assumed that each pixel takes only two values (1, 0) of ON and OFF, but it is not limited to this and may take intermediate gradations.

A reflective DMD (Digital Micromirror Device) or a transmissive liquid crystal device can be used as the patterning device 114 . A pattern signal PTN (image data) generated by a pattern generator 132 is applied to the patterning device 114 .

The pattern generator 132 generates a pattern signal PTN _r that specifies the intensity distribution _Ir of the illumination light S1, and switches the pattern signal PTN _r over time for each time slot (r=1, 2, . . . M). Thereby, the light S0 is spatially modulated with a different pattern for each time slot to generate the illumination light S1. Sensing by the imaging apparatus 100 is performed with M patterned irradiations as one set, and one restored image is generated corresponding to the M patterned irradiations. One sensing based on M pattern irradiations is called one frame. That is, one frame includes M time slots.

Photodetector 120 measures reflected light from object OBJ and outputs detection signal _Dr. The detection signal _Dr is a spatial integrated value of light energy (or intensity) incident on the photodetector 120 when the object OBJ is irradiated with illumination light having the intensity distribution _Ir . Therefore, the photodetector 120 can use a single-pixel photodetector. The photodetector 120 outputs a plurality of detection signals D ₁ to D _M respectively corresponding to a plurality of M intensity distributions I ₁ to I _M .

The processor 130 includes a pattern generator 132 , a reconstruction processor 134 and a mode controller 136 . The reconstruction processing unit 134 performs correlation calculation between a plurality of intensity distributions (also referred to as random patterns) I ₁ to I _M and a plurality of detected intensities b ₁ to b _M to generate a restored image G(x, y) of the object OBJ. to reconfigure.

The detected intensities b ₁ -b _M are based on the detected signals D ₁ -D _M . The relationship between the detected intensity and the detected signal may be determined in consideration of the type and system of the photodetector 120 .

Assume that illumination light S1 having a certain intensity distribution _Ir is emitted for a certain irradiation period. It is also assumed that the detection signal _Dr represents the amount of light received at a certain time (or minute time), that is, an instantaneous value. In this case, the detection signal _Dr may be sampled multiple times during the irradiation period, and the detection strength b _r may be the integrated value, average value, or maximum value of all sampled values of the detection signal _Dr. Alternatively, some of all sampled values may be selected, and the integrated value, average value, or maximum value of the selected sampled values may be used. Selection of a plurality of sampled values may be performed, for example, by extracting the order x-th to y-th counting from the maximum value, excluding sampled values lower than an arbitrary threshold, or You may extract the sampling value of the small range.

When a device such as a camera whose exposure time can be set is used as the photodetector 120, the output _Dr of the photodetector 120 can be directly used as the detection intensity _br .

The conversion from the detection signal _Dr to the detection intensity b _r may be performed by the processing unit 130 or may be performed outside the processing unit 130 .

The arithmetic processing unit 130 can be implemented by combining a processor (hardware) such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), or microcomputer, and a software program executed by the processor (hardware). Processing unit 130 may be a combination of multiple processors. Alternatively, the arithmetic processing unit 130 may be composed only of hardware.

The functions of the arithmetic processing unit 130 may be realized by software processing, hardware processing, or a combination of software processing and hardware processing. Specifically, software processing is implemented by combining processors (hardware) such as CPUs (Central Processing Units), MPUs (Micro Processing Units), microcomputers, and software programs executed by the processors (hardware). Note that the arithmetic processing unit 130 may be a combination of multiple processors. Specifically, hardware processing is implemented by hardware such as ASIC (Application Specific Integrated Circuit), controller IC, and FPGA (Field Programmable Gate Array).

The imaging apparatus 100 is configured to be switchable between a first mode (TDGI mode) and a second mode (IPGI mode). The mode controller 136 adaptively switches between the TDGI mode and the IPGI mode according to the sensing environment.

　The TDGI mode and the IPGI mode will be described below.

(TDGI mode)
The reconstructed image generation processing in the reconstruction processing unit 134 of the arithmetic processing unit 130 is performed by dividing M times of irradiation into a plurality of k (k≧2) units. Specifically, the reconstruction processing unit 134 performs correlation calculation for each divided unit to generate an intermediate image M _j (x, y). Then, the k intermediate images M ₁ to M _k (x, y) obtained for the k units are combined to generate the final restored image G _TDGI (x, y).

For simplicity, it is assumed that the number of irradiation times n of each unit is equal to M/k. The restored image G _TDGI (x, y) in this case is represented by Equation (2). I _r is the r-th intensity distribution, b _r is the value of the r-th detected intensity, and is the average value of the detected intensity b _r measured in the j-th unit. be.

The j-th term on the right side of equation (2) represents the intermediate image M _j (x, y) of the j-th unit. Therefore, the restored image G _TDGI (x, y) is synthesized by simply adding the corresponding pixels of the plurality of intermediate images M _j (x, y). Note that the combining method is not limited to simple addition, and weighted addition or other processing may be performed.

In other words, the processor 130 performs the correlation calculation every n=M/k irradiations. Then, a restored image G _TDGI (x, y) is generated by synthesizing the k intermediate images M(x, y).

(IPGI mode)
In the second mode, the pattern generator 132 generates a pattern signal PTN such that the illumination light S1 contains a pair of patterns I(x,y), Î(x,y) whose intensity distributions are complementary. .

In IPGI mode, a first pattern group containing M patterns and a second pattern group containing M patterns are irradiated. The pattern I _r (x, y) included in the first pattern group and the pattern I r (x, y) corresponding to the pattern I _r (x, y) in the second pattern group (referred to as an inverted pattern) I _r (x, y) are Complementary relationship. Complementary means that corresponding pixels are complementary.

When the intensity distribution has two gradations (binary values of 1 and 0), when the pixel value of a pixel with the pattern I _r (x, y) is 1, the same pixel of the inverted pattern I _r (x, y) is 0, and when the pixel value of a pixel in the pattern I _r (x, y) is 0, the pixel value of the same pixel in the inverse pattern I _r (x, y) is 1.

When the intensity distribution is multi-gradation (0 to MAX), when the pixel value of a pixel with the pattern I _r (x, y) is A (0≦A≦MAX), the inverted pattern I _r ^(x, y ) is MAX-A.

The irradiation order _of the first pattern group and the second pattern group is _not particularly limited. may be physically adjacent to each other.

Alternatively, the first pattern group may be irradiated first, followed by the second pattern group.

Let b _r be the detected intensity obtained as a result of irradiation with the non-inverted pattern I _r (x, y), and let b _r ^ be the detected intensity obtained as a result of irradiation with the inverted pattern I _r ^(x, y). In the second mode, the reconstruction processing unit 134 of the arithmetic processing device 130 generates the restored image G _IPGI (x, y) by the correlation calculation represented by Equation (3).

That is, a correlation calculation is performed between a plurality of non-inverted patterns I ₁ (x, y) to I _M (x, y) and a plurality of detection intensities b ₁ to b _M , and a plurality of inverted patterns I ₁ ^(x, y) to A correlation calculation is performed between I _M ̂(x,y) and a plurality of detected intensities b ₁ ̂˜b _M ̂, and the sum of the two correlation results represents the reconstructed image G _IPGI (x,y).

The configuration of the imaging apparatus 100 is as described above. Next, operations in the TDGI mode and the IPGI mode will be described.

FIG. 3 is a time chart explaining the operation of the imaging apparatus 100 in TDGI mode. In FIG. 3, the total number of pixels of the patterning device 114 is represented by p=4×4=16 pixels, and M random pattern I ₁ to I _M are generated.

The reconstruction image is generated by dividing it into k units each containing n irradiations.

The first unit includes the 1st to nth irradiations, the detected intensities b ₁ to _bn corresponding to the n irradiations are generated, and their average value is generated. Then, correlation calculation is performed using the detected intensities b ₁ to b _n , their average value , and the intensity distributions I _n+1 to I _2n , and the intermediate image M ₁ (x, y) is generated.

The second unit contains the n+1 to 2n exposures, and the detected intensities b _n+1 to b _2n corresponding to the n exposures are generated and their average value is generated. Then, correlation calculation is performed using the detected intensities b _n+1 to b _2n , their average values , and the intensity distributions I _n+1 to I _2n , and the intermediate image M ₂ (x, y) is generated.

Similarly, the j-th unit includes (j-1)n+1 to jn-th irradiation, and the detection intensities b _(j-1)n+1 to b _jn corresponding to the n-th irradiation are generated, and their average A value is generated. Then, correlation calculation is performed using the detected intensities b _(j−1)n+1 to b _jn , their average values , and the intensity distributions I _(j−1)n+1 to I _jn , An intermediate image M _j (x,y) is generated.

The last k-th unit contains the (k−1)n+1 to kn-th irradiations, and the detected intensities b _(k−1)n+1 to b _kn corresponding to the n irradiations are generated, and their average is generated. Then, correlation calculation is performed using the detected intensities b _(k−1)n+1 to b _kn , their average values , and the intensity distributions I _(k−1)n+1 to I _kn , An intermediate image M _k (x,y) is generated.

A final restored image G(x, y) is generated by synthesizing the k intermediate images M ₁ (x, y) to M _k (x, y).

The above is the operation of the TDGI mode. According to the TDGI mode, by dividing the correlation calculation into units and performing it, the amount of noise change per correlation calculation can be reduced, the restoration accuracy can be increased, and the image quality can be improved. In addition, since correlation calculation can be started without waiting for the completion of M irradiations, the time required to generate a restored image can be shortened.

　In the following, the improvement of noise resistance in the TDGI mode will be described. Imaging using correlation calculation for each unit based on Equation (2) is hereinafter referred to as segmentation reconstruction. Also, conventional imaging using correlation calculation based on equation (1) is referred to as collective reconstruction.

For example, consider noise that monotonously increases over time (referred to as linear noise). Such linear noise can occur in cases where the distance between the noise source and the photodetector approaches over time.

FIG. 4 is a diagram explaining another sequence in the TDGI mode. In the example of FIG. 4, the illumination light is continuously emitted, and the correlation calculation is performed every n times of irradiation to generate the intermediate image M(x, y). Then, every time an intermediate image M(x, y) is generated, a restored image G(x, y) is generated by synthesizing the latest k intermediate images M(x, y). According to this sequence, the restored image G(x, y) can be updated every n times of irradiation, so the update rate can be increased.

Next, noise in the TDGI mode will be explained. FIG. 5(a) is a diagram showing the influence of noise in batch restoration, and FIG. 5(b) is a diagram showing the influence of noise in division restoration. The horizontal axis represents the irradiation pattern number, that is, the time. σ _r represents the noise intensity when illuminating the r-th pattern, where it increases according to σ _r =α×r.

Please refer to FIG. When _collective restoration _is _performed , the absolute value _| Δσ _r _| The integrated value over irradiation corresponds to the hatched area and is expressed by the following equation (4).
Σ|Δσ _r |=2×1/2×(kn/2)×(knα/2)=n ² αk ² /4 (4)

Please refer to FIG. Here, k=5. In the case of division restoration, the integrated value Σ|Δσ r | of the absolute value | _Δσ _r | of the difference Δσ _r (=σ _r −<σ _r ^[j] >) in the j-th unit is the hatched area. It corresponds and is represented by Formula (5).
Σ|Δσ _r |=2×1/2×(n/2)×(nα/2)=n ² α/4 (5)
The integrated value over all irradiation can be obtained by multiplying the value of equation (5) by k, which is kn ² α/4.

In this way, by performing division and restoration, the integrated value of the absolute value |Δσ _r | of the noise intensity difference can be reduced, and the noise immunity can be improved.

Next, the simulation results for collective restoration and divided restoration will be described.
FIG. 6(a) shows a target image, and FIG. 6(b) shows an image obtained by batch restoration and division restoration when noise does not exist. The simulation was performed for M=1000, M=10000 and M=100000. Also, the number of units k is assumed to be 10.

In batch restoration, M = 100000 is required in order to restore the original target image to a recognizable degree, and the same is true for segmented restoration. When M=100000, it can be said that the accuracy of the finally obtained restored image is the same between batch restoration and divisional restoration.

FIG. 7 is a diagram showing a restored image G(x, y) obtained by collective restoration and divisional restoration when linear noise exists. The total number of irradiation times is M=100000, and the average value of the detected intensity is =1000. The noise coefficient α is calculated in four ways: 0, 0.01, 0.1, and 1. α=0 corresponds to no noise. As for split reconstruction, calculations are made for k=10, 100, and 1000.

As can be seen from the simulation results in FIG. 7, the higher the number of divisions k, the higher the noise resistance.

(IPGI)
FIG. 8 is a time chart for explaining the operation of the imaging apparatus 100 in IPGI mode.

A first pattern group _{I 1} ₍ x, y) to I _M ₍ x, y) and the second pattern group I ₁ ̂(x, y) to I _M ̂(x, y) are alternately irradiated in a time division manner.

Each irradiation with the non-inverted pattern I _r (x, y) produces a detected intensity b _r , and each irradiation with the inverted pattern I _r ̂(x, y) produces a detected intensity b _r ̂. Each detection strength b _r , b _r ^ includes a noise component σ. The reconstruction processing unit 134 obtains detection intensities b ₁ to b _M , detection intensities b ₁ ^ to b _M ^, non-inverted patterns I ₁ (x, y) to I _M (x, y ), I ₁ ̂(x, y) to I _M ̂(x, y), calculate the synthetic correlation formula of equation (3), and generate the final restored image G _IPGI (x, y) .

Next, I will explain the advantages of IPGI mode.

Assume now that the intensity distribution takes two values of 0 and 1. At this time, the non-inverted pattern I _r (x, y) and the inverted pattern I _r ^(x, y) hold the formula (6).

Substituting equation (3) into equation (6) yields equation (7).

Since is the average value of b _r ^, Equation (8) holds.

By substituting equation (8) into equation (7) and arranging, equation (9) is obtained.

Here, b _r and b _r ^ are represented by the sum of the true signal component (subscripted with (true)) due to the reflected light and the noise component σ. Suppose b _r and b _r ^ contain the same noise component σ _r .
b _r =b _r(true) +σ _r
b _r ^ = b _{r (true)} ^ + σ _r

Substituting this into equation (9) yields equation (10).

As can be seen from Equation (9), in the finally obtained image G _IPGI (x, y), the noise components cancel each other out, so only the signal components b _r(true) and b _r(true) ̂ are contains. Therefore, image quality can be improved.

　Figs. 9(a) and (b) are diagrams showing restored images obtained by the TDGI mode and the IPGI mode. For comparison, a conventional image (GI) based on equation (1) is shown. In FIG. 9A, the number of pattern irradiation times M in the TDGI mode is 10000, and the number n of one unit is four. The number M of pattern irradiations in the IPGI mode is 5000, and the total number of irradiations is the same as 10000 times in the TDGI mode. It is assumed that sine wave noise of 120 Hz is generated when the pattern switching frequency is 20 kHz. The noise SD value (standard deviation) is varied between 1000 and 10000 in 10 steps.

The image quality in TDGI mode and IPGI mode is improved regardless of the size of the noise compared to the image quality obtained by normal correlation calculation based on Equation (1).

In a region where the noise level is low, when comparing the image quality of the TDGI mode and the IPGI mode, the image quality of the TDGI mode is higher than that of the IPGI mode, although the total number of irradiation times is the same. . This is because the information obtained in the non-inverted pattern and the inverted pattern in the IPGI mode is mathematically equivalent (redundant), so the actual number of irradiations is half that in the TDGI mode. .

On the contrary, in areas where the noise level is high, the image quality in IPGI mode is higher than that in TDGI mode. This is because the effect of noise cancellation in the IPGI mode is remarkable.

In FIG. 9(b), the number of times of irradiation in TDGI mode is 500, the number of times of irradiation in IPGI mode is 250×2, and the noise SD value is changed in 10 steps between 100 and 1000.

In the situation of FIG. 9(b), the noise level is small, so the result is that the image quality is better in the TDGI mode.

According to this embodiment, a high-quality restored image can be obtained by adaptively selecting the TDGI mode and the IPGI mode according to the sensing environment.

The method of mode selection by the mode controller 136 is not particularly limited, and a mode that provides higher image quality under certain circumstances may be selected.

For example, the mode controller 136 may monitor the detection signal D or the detection intensity b output from the photodetector 120 and select a mode according to the characteristics of the noise contained therein. As for the characteristics of the noise, the mode can be selected based on the noise intensity, noise waveform (sine wave, random noise, rectangular noise, impulse noise), and noise frequency. Preliminary measurements and simulations allow us to know the tendency of which mode should be selected for what kind of noise. Alternatively, mode controller 136 may be implemented by machine learning.

The mode controller 136 may select a mode based on the image quality of the restored image generated by the reconstruction processing unit 134. For example, when a good image quality is obtained in a certain mode, that mode may be continued, and when the image quality deteriorates, the opposite mode may be selected. If the TDGI mode is selected by default and the quality of the resulting decompressed image is degraded, the IPGI mode may be selected.

When the restored image generated by the imaging apparatus 100 is input to a discriminator or classifier and the type (class/category) of the object included in the restored image is detected, the discriminator or classifier should be detected. When the object can no longer be detected, it may be assumed that the image quality has deteriorated, and the mode may be switched.

(Embodiment 2)
FIG. 10 is a block diagram of an imaging device 100A according to the second embodiment. The imaging apparatus 100A can switch between two modes, a TDGI mode as a first mode and a TDGI-IPGI hybrid mode as a second mode.

　The TDGI-IPGI hybrid mode will be explained. In the TDGI-IPGI hybrid mode, the irradiation pattern of the illumination light S1 is the same as the irradiation pattern in the IPGI mode, and the arithmetic processing by the reconstruction processing unit 134 is the same as the arithmetic processing in the TDGI mode.

That is, the pattern of illumination light S1 in the TDGI-IPGI mode includes a set of non-inverted patterns and inverted patterns. On the other hand, the reconstruction processing unit 134 performs correlation calculation each time the illumination light is emitted n times to generate an intermediate image, and synthesizes the latest k intermediate images to generate a restored image.

FIGS. 11(a) and (b) are diagrams showing restored images obtained by the TDGI mode, IPGI mode, and TDGI-IPGI hybrid mode. In the TDGI-IPGI hybrid mode, it is possible to obtain the same image quality as in the IPGI mode.

Also, in the TDGI-IPGI hybrid mode, by adopting the processing shown in FIG. 4, the restored image can be updated each time the measurement of one unit is completed. That is, the frame rate can be improved compared to the IPGI mode.

Also, when switching between the TDGI mode and the IPGI mode in Embodiment 1, it is necessary to switch both the irradiation pattern and the arithmetic processing. On the other hand, switching between the TDGI mode and the TDGI-IPGI hybrid mode in the second embodiment has the advantage that only the irradiation pattern needs to be changed, and there is no need to switch the arithmetic processing.

(Embodiment 3)
In the second embodiment, the TDGI mode and the TDGI-IPGI hybrid mode are switched depending on the situation, but in the third embodiment, the two modes can be seamlessly switched continuously. That is, there is an intermediate stage between the TDGI mode and the TDGI-IPGI hybrid mode.

That is, let k be the number of units in the TDGI mode and the TDGI-IPGI hybrid mode. In an intermediate stage, x units out of k units are assigned to TDGI mode and the remaining (kx) units are assigned to TDGI-IPGI hybrid mode. Then, the restored image may be generated by synthesizing the x intermediate images obtained for the x units and the (kx) intermediate images obtained for the (kx) units. . In other words, the TDGI mode and the TDGI-IPGI hybrid mode are selected by time division, and the intermediate images obtained in the two modes are synthesized to generate the restored image.

The mode controller 136 changes the parameter x, that is, the ratio of time division, between 0 and k so as to obtain higher image quality according to the sensing environment.

It should be understood by those skilled in the art that this embodiment is an example, and that various modifications can be made to combinations of each component and each treatment process, and such modifications are also within the scope of the present invention. . Such modifications will be described below.

(Modification 1)
In the embodiment, the number of times of irradiation for each unit is equal, but this is not the only option, and the number of times of irradiation for each unit may not be equal.

(Modification 2)
Also, in the embodiment, the unit number k is fixed, but the unit number k may be dynamically controlled. Image quality can be further improved by selecting the optimum number of units k according to the noise fluctuation speed and noise waveform.

(Modification 3)
In the embodiment, illumination device 110 is configured by a combination of light source 112 and patterning device 114, but this is not the only option. For example, the illumination device 110 is composed of an array of multiple semiconductor light sources (LEDs (light emitting diodes) and LDs (laser diodes)) arranged in a matrix, and can control the on/off (or brightness) of each semiconductor light source. can be configured to

(Application)
Next, the application of the imaging apparatus 100 will be described. FIG. 12 is a block diagram of the object identification system 10. As shown in FIG. This object identification system 10 is mounted on a vehicle such as an automobile or a motorcycle, and determines types (categories) of objects OBJ existing around the vehicle.

The object identification system 10 includes an imaging device 100 and an arithmetic processing device 40 . As described above, the imaging apparatus 100 generates the restored image G of the object OBJ by irradiating the object OBJ with the illumination light S1 and measuring the reflected light S2.

The arithmetic processing device 40 processes the output image G of the imaging device 100 and determines the position and type (category) of the object OBJ.

The classifier 42 of the arithmetic processing unit 40 receives the image G as an input and determines the position and type of the object OBJ contained therein. Classifier 42 is implemented based on a model generated by machine learning. Although the algorithm of the classifier 42 is not particularly limited, YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), R-CNN (Region-based Convolutional Neural Network), SPPnet (Spatial Pyramid Pooling), Faster R-CNN , DSSD (Deconvolution-SSD), Mask R-CNN, etc., or algorithms that will be developed in the future.

The above is the configuration of the object identification system 10. By using the imaging device 100 as a sensor for the object identification system 10, the following advantages can be obtained.

By using the imaging device 100, that is, the quantum radar camera, the noise immunity is greatly improved. For example, when it is raining, snowing, or driving in fog, it is difficult to recognize the object OBJ with the naked eye. A restored image G can be obtained.

Also, by using the imaging apparatus 100, the calculation delay can be reduced. This can provide low latency sensing. Particularly in in-vehicle applications, there are cases where the object OBJ moves at high speed.

FIG. 13 is a block diagram of an automobile equipped with the object identification system 10. FIG. Automobile 300 includes

headlights

302L and 302R. Imaging device 100 is built into at least one of

headlights

302L and 302R. The headlight 302 is positioned at the extreme end of the vehicle body and is the most advantageous location for installing the imaging apparatus 100 in terms of detecting surrounding objects.

FIG. 14 is a block diagram showing a vehicle lamp 200 including an object detection system 210. FIG. The vehicle lamp 200 constitutes a lamp system 310 together with a vehicle-side ECU 304 . A vehicle lamp 200 includes a light source 202 , a lighting circuit 204 and an optical system 206 . Furthermore, the vehicle lamp 200 is provided with an object detection system 210 . Object detection system 210 corresponds to object identification system 10 described above and includes imaging device 100 and arithmetic processing device 40 .

Information on the object OBJ detected by the processing unit 40 may be used for light distribution control of the vehicle lamp 200 . Specifically, the lamp-side ECU 208 generates an appropriate light distribution pattern based on the information about the type and position of the object OBJ generated by the arithmetic processing unit 40 . The lighting circuit 204 and the optical system 206 operate so as to obtain the light distribution pattern generated by the lamp-side ECU 208 .

Information regarding the object OBJ detected by the processing unit 40 may be transmitted to the vehicle-side ECU 304 . The vehicle-side ECU may perform automatic driving based on this information.

Although the present invention has been described using specific terms based on the embodiments, the embodiments merely show one aspect of the principle and application of the present invention, and the embodiments are defined in the scope of claims. Many modifications and changes in arrangement are permitted without departing from the spirit of the present invention.

The present disclosure relates to an imaging device using ghost imaging.

OBJ... Object 10... Object identification system 40... Arithmetic processor 42... Classifier 100... Imaging device 110... Illuminator 112... Light source 114... Patterning device 120... Photodetector 130... Arithmetic processing Apparatus 132 Pattern generator 134 Reconfiguration processor 136 Mode controller 200 Vehicle lamp 202 Light source 204 Lighting circuit 206 Optical system 300 Automobile 302 Headlamp 304: Vehicle-side ECU, 310: Lamp system.

Claims

an illumination device that illuminates a field of view with illumination light having a spatially random intensity distribution;
a photodetector that measures reflected light from an object;
an arithmetic processing unit that performs correlation calculation between the detected intensity based on the output of the photodetector and the intensity distribution of the illumination light, and reconstructs a restored image of the object;
and can be switched between the first mode and the second mode,
In the first mode, the arithmetic processing unit performs correlation calculation each time the illumination light is irradiated n times (n≧2) to generate an intermediate image, and synthesizes k intermediate images (k≧2). to generate the restored image,
The imaging apparatus, wherein in the second mode, the illumination light includes a pair of patterns having complementary intensity distributions.
2. The method according to claim 1, wherein, in said first mode, each time a new intermediate image is generated, said arithmetic processing unit synthesizes the latest k intermediate images to generate said restored image. imaging equipment.
In the second mode, the arithmetic processing unit performs correlation calculation every time the illumination light is emitted n times to generate an intermediate image, synthesizes the latest k intermediate images, and generates the restored image. 2. The imaging apparatus of claim 1, wherein:
The imaging apparatus according to any one of claims 1 to 3, wherein switching between the first mode and the second mode is possible based on the output of the photodetector.
The imaging apparatus according to any one of claims 1 to 3, wherein switching between the first mode and the second mode is possible based on the image quality of the restored image.
The imaging apparatus according to any one of claims 1 to 3, wherein switching between the first mode and the second mode is possible based on a discrimination rate by a classifier based on the restored image.
4. The imaging apparatus according to claim 3, wherein the first mode and the second mode are selected in a time division manner, intermediate images obtained in the two modes are synthesized, and the restored image is generated.
The imaging apparatus according to claim 7, wherein the ratio of time division between the first mode and the second mode is adaptively changed.
A vehicle lamp comprising the imaging device according to any one of claims 1 to 8.
A vehicle comprising the imaging device according to any one of claims 1 to 8.