WO2023085328A1

WO2023085328A1 - Imaging device, imaging method, vehicular lamp, and vehicle

Info

Publication number: WO2023085328A1
Application number: PCT/JP2022/041755
Authority: WO
Inventors: 輝明鳥居; 祐太春瀬
Original assignee: 株式会社小糸製作所
Priority date: 2021-11-12
Filing date: 2022-11-09
Publication date: 2023-05-19

Abstract

A light source 112 in an illumination device 110 generates frequency-modulated continuous waves S0. A patterning device 114 modulates the frequency-modulated continuous waves S0 using patterns that differ for each time slot to generate illumination light S1. A beat detection unit 120 generates a detection signal based on reflected light S2 received from an object and multiplexed light S3 of the frequency-modulated continuous waves S0. A calculation processing device 130 calculates a correlation, for each beat in the detection signal, between the intensity of the beat and the intensity distribution of the illumination light S1 to reconstruct a restored image of the object.

Description

IMAGING APPARATUS AND IMAGING METHOD, VEHICLE LAMP, VEHICLE

The present disclosure relates to imaging apparatus and methods.

An object identification system that senses the position and type of objects around the vehicle is used for automated driving and automatic control of headlamp light distribution. An object identification system includes a sensor and a processor that analyzes the output of the sensor. Sensors are selected from cameras, LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), millimeter-wave radar, ultrasonic sonar, etc., taking into consideration the application, required accuracy, and cost.

As one of the imaging devices (sensors), one that uses the principle of ghost imaging is known. Ghost imaging illuminates an object while randomly switching the intensity distribution (pattern) of illumination light, and measures the light detection intensity of the reflected light for each pattern. The light detection intensity is the integral of energy or intensity over a plane, not the intensity distribution. Correlation calculation between the corresponding pattern and the light detection intensity is then performed to reconstruct a restored image of the object.

Japanese Patent No. 6412673 International publication WO2020/218282 International publication WO2021/079810

FMCW (Frequency Modulated Continuous Wave)-LiDAR is known as a sensor capable of generating distance information. FMCW-LiDAR is a ranging technology conventionally used for radar, and has excellent characteristics such as high resolution and high detection sensitivity. Relative velocity can also be measured using the Doppler effect.

In order to acquire 3D information with FMCW-LiDAR, it is necessary to scan the light in the horizontal and vertical directions, which increases the cost. Moreover, since scanning is required, there is a problem that the measurement time becomes long.

The present disclosure has been made in such a situation, and one exemplary purpose of certain aspects thereof is to provide an imaging device capable of generating distance information in a short period of time.

An imaging apparatus according to an aspect of the present disclosure includes a light source that generates a frequency-modulated continuous wave, and a spatial light modulator that spatially modulates the frequency-modulated continuous wave with a different pattern for each time slot. a lighting device that irradiates a field of view with a modulated continuous wave as illumination light; a beat detector that generates a detection signal based on a signal obtained by combining the reflected light from an object and the frequency-modulated continuous wave; and an arithmetic processing unit that performs correlation calculation between the intensity of the beat and the intensity distribution of the illumination light and reconstructs the restored image of the object.

An imaging device according to an aspect of the present disclosure includes a light source that generates a frequency-modulated continuous wave, an illumination device that irradiates a field of view with the frequency-modulated continuous wave as illumination light, and a spatial light reflected from an object for each time slot. a spatial light modulator that randomly patterns the spatial light modulator; a beat detector that generates a detection signal based on the combined light of the reflected light modulated by the spatial light modulator and the frequency-modulated continuous wave; and an arithmetic processing unit that performs correlation calculation between the intensity of the illumination light and the intensity distribution of the illumination light, and reconstructs a restored image of the object.

Another aspect of the present disclosure is an imaging method. This method comprises the steps of: generating a frequency-modulated continuous wave; modulating the frequency-modulated continuous wave with a different pattern for each time slot to generate illumination light; a step of generating a detection signal based on the combined light of continuous waves; and a step of performing a correlation calculation between the intensity of the beat and the intensity distribution of the illumination light for each beat of the detection signal, and reconstructing a restored image of the object. Prepare.

Another aspect of the present disclosure is also an imaging method. This method comprises the steps of: irradiating a field of view with a frequency-modulated continuous wave as illumination light; modulating the reflected light of the illumination light reflected by an object with a different pattern for each time slot; a step of generating a detection signal based on the combined light of the modulated continuous waves, performing a correlation calculation between the intensity of the beat and the intensity distribution of the illumination light for each beat of the detection signal, and reconstructing a restored image of the object; Prepare.

According to one aspect of the present disclosure, distance information can be generated in a short time.

1 is a diagram showing an imaging device according to Embodiment 1; FIG. FIG. 4 is a diagram for explaining beats included in the detection signal _Di ; FIGS. 3A and 3B are diagrams for explaining beat intensity. 2A and 2B are diagrams for explaining the operation of the imaging apparatus of FIG. 1; FIG. 2A and 2B are diagrams for explaining the operation of the imaging apparatus of FIG. 1; FIG. FIG. 10 is a diagram for explaining synthesis of a plurality of restored images; FIG. Fig. 3 shows spectra obtained in a plurality of time slots; Fig. 3 shows spectra obtained in a plurality of time slots; 4 is a time chart for explaining the operation of the imaging apparatus in TDGI mode; FIG. 11 is a diagram illustrating another sequence in TDGI mode; FIG. 11(a) is a diagram showing the influence of noise in batch restoration, and FIG. 11(b) is a diagram showing the influence of noise in division restoration. FIG. 12(a) is a diagram showing a target image, and FIG. 12(b) is a diagram showing an image obtained by batch restoration and division restoration when noise does not exist. FIG. 10 is a diagram showing a restored image G(x, y) obtained by collective restoration and divisional restoration when linear noise exists; 2 is a block diagram of an imaging device according to Embodiment 2; FIG. 1 is a block diagram of an object identification system; FIG. 1 is a block diagram of a vehicle with an object identification system; FIG. 1 is a block diagram showing a vehicle lamp equipped with an object detection system; FIG.

An overview of some exemplary embodiments of the present disclosure is provided. This summary presents, in simplified form, some concepts of one or more embodiments, as a prelude to the more detailed description that is presented later, and for the purpose of a basic understanding of the embodiments. The size is not limited. This summary is not a comprehensive overview of all possible embodiments, and it is intended to neither identify key elements of all embodiments nor delineate the scope of some or all aspects. For convenience, "one embodiment" may be used to refer to one embodiment (example or variation) or multiple embodiments (examples or variations) disclosed herein.

An imaging apparatus according to one embodiment includes a light source that generates a frequency-modulated continuous wave and a spatial light modulator that spatially modulates the frequency-modulated continuous wave with a different pattern for each time slot. a lighting device that irradiates a field of view with a wave as illumination light; a beat detector that generates a detection signal based on a signal obtained by combining the reflected light from an object and a frequency-modulated continuous wave; and an arithmetic processing unit that performs correlation calculation between the intensity and the intensity distribution of the illumination light and reconstructs a restored image of the object.

According to this configuration, it is possible to generate three-dimensional distance information by single-pixel imaging using correlation calculation without scanning illumination light.

In one embodiment, the arithmetic processing unit may Fourier transform the detection signal and detect the strength of the beat.

In one embodiment, the arithmetic processing unit may apply different colors to a plurality of restored images and synthesize them. This makes it possible to generate an image that represents the distance and brightness (reflectance) of an object.

In one embodiment, the processor may calculate the velocity of the object based on the Doppler shift, and estimate the distance to the object in the next time slot based on the velocity.

In one embodiment, the processor may estimate the beat frequency of the next time slot based on the estimated distance to the object. As a result, when there are a plurality of beats with different frequencies, the beat of the previous time slot can be appropriately associated with the beat of the current time slot.

In one embodiment, the imaging device may notify the outside when the number of beats of the detection signal changes.

In one embodiment, when the number of beats in the detected signal changes, a new frame measurement may be started from there. As a result, beats and objects can be associated on a one-to-one basis, and image accuracy can be improved.

In one embodiment, the light source and spatial light modulator may be an array of light emitting elements.

"The intensity distribution is random" in this specification does not mean that it is completely random, but it is sufficient if it is random enough to reconstruct an image in ghost imaging. Therefore, "random" in this specification can include a certain degree of regularity therein. Also, "random" does not require unpredictability, but may be predictable and reproducible.

(embodiment)
BEST MODE FOR CARRYING OUT THE INVENTION The present invention will be described below based on preferred embodiments with reference to the drawings. The same or equivalent constituent elements, members, and processes shown in each drawing are denoted by the same reference numerals, and duplication of description will be omitted as appropriate. Moreover, the embodiments are illustrative rather than limiting the invention, and not all features and combinations thereof described in the embodiments are necessarily essential to the invention.

(Embodiment 1)
FIG. 1 is a diagram showing an imaging apparatus 100 according to Embodiment 1. FIG. Imaging device 100 is a correlation function image sensor that uses the principle of ghost imaging (also called single pixel imaging), and includes illumination device 110 , beat detection section 120 and arithmetic processing device 130 .

The illumination device 110 is a pseudo thermal light source, generates illumination light S1 having a spatial intensity distribution I(x, y) that can be regarded as substantially random, and illuminates the object OBJ. The illumination light S1 is sequentially irradiated a plurality of M times while randomly changing its intensity distribution. The number of times of irradiation M is the number of times that can restore the original image.

Illumination device 110 includes light source 112 , patterning device 114 and pattern generator 132 .

The light source 112 generates a frequency-modulated continuous wave S0 that has a uniform intensity distribution and whose frequency changes over time. For example, the frequency of the frequency-modulated continuous wave S0 changes (increases or decreases) with a constant slope over time.

A semiconductor laser with a frequency modulation function can be used for the light source 112 . The wavelength and spectrum of the frequency-modulated continuous wave S0 are not particularly limited, and may be white light having multiple or continuous spectra, or monochromatic light containing a predetermined wavelength. The wavelength of the illumination light S1 may be infrared or ultraviolet.

The patterning device 114 has a plurality of pixels arranged in a matrix, and is configured to be able to spatially modulate the light intensity distribution I based on a combination of ON and OFF of the plurality of pixels. In this specification, a pixel in an ON state is called an ON pixel, and a pixel in an OFF state is called an OFF pixel. In the following description, for ease of understanding, it is assumed that each pixel takes only two values (1, 0) of ON and OFF, but it is not limited to this and may take intermediate gradations.

A reflective DMD (Digital Micromirror Device) or a transmissive liquid crystal device can be used as the patterning device 114 . A pattern signal PTN (image data) generated by a pattern generator 132 is applied to the patterning device 114 .

The pattern generator 132 generates a pattern signal PTN _r that specifies the intensity distribution _Ir of the illumination light S1, and switches the pattern signal PTN _r for each time slot TS (r=1, 2, . . . M). Thereby, the frequency-modulated continuous wave S0 is spatially modulated with a different pattern for each time slot to generate the illumination light S1. Within one time slot TS, the frequency of the frequency-modulated continuous wave S0 is preferably swept at least once, preferably twice or more. Sensing by the imaging apparatus 100 is performed with M patterned irradiations as one set, and one restored image is generated corresponding to the M patterned irradiations. One sensing based on M pattern irradiations is called one frame. That is, one frame includes M time slots TS ₁ to TS _M .

Imaging apparatus 100 receives reflected light S2 reflected by object OBJ when illumination light S1 hits object OBJ. The beat detector 120 detects the reflected light S2 by heterodyne detection and outputs a detection signal _Dr. The detection signal _Dr is a spatially integrated value of light energy (or intensity) incident on the beat detection section 120 when the object _OBJ is irradiated with illumination light having an intensity distribution Ir.

The beat detector 120 includes a multiplexer 122 and a photodetector 124 . The combining unit 122 combines the frequency-modulated continuous wave S0 and the reflected light S2 to generate combined light S3. The combined light S3 will contain a beat according to the frequency difference between the two lights S0 and S2. The photodetector 124 detects the combined light S3 and generates a detection signal D according to the beat contained in the combined light S3. The photodetector 124 can use a single-pixel photodetector (photodetector) 124 . Note that an image sensor may be used as the photodetector 124 to synthesize the values of a plurality of pixels.

The beat detector 120 outputs a plurality of detection signals D ₁ to D _M respectively corresponding to a plurality of M intensity distributions I ₁ to I _M .

A processor 130 receives a plurality of detection signals D ₁ to D _M . For ease of understanding, consider a scene with only a single object in the field of view. While the illumination light S1 of the r-th pattern _Ir is emitted, the detection signal D _i includes a frequency beat corresponding to the distance to the object OBJ, similarly to FMCW-LiDAR.

FIG. 2 is a diagram for explaining beats included in the detection signal _Di. The horizontal axis represents time, and the vertical axis represents the frequency of the optical continuous wave. FIG. 2 shows the frequencies of the frequency-modulated continuous wave S0, the illumination light S1, and the reflected light S2. The reflected light S2 is shifted to the right by the time τ when the illumination light S1 hits the object OBJ and returns. When the distance to the object OBJ is d,
τ=2×d/c
is. c is the speed of light.

The frequency f ₀ (t) of the frequency-modulated continuous wave S0 and the illumination light S1 is expressed by the relational expression f ₀ (t)=f ₀ +a×t
shall increase (chirp) linearly according to At this time, the frequency f ₂ (t) of the reflected light S2 is
f ₂ (t)=f ₀ (t−τ)=f ₀ +a×(t−τ)
becomes. When light beams with two frequencies f ₀ and f ₂ with a small frequency difference are multiplexed (interfered), a beat is generated based on the frequency difference Δf=f ₀ −f ₂ between the two frequencies f ₀ and f ₂ .
In this example, the beat frequency Δf is
Δf=a×τ=a×2×d/c
Therefore, it depends on the distance d to the object. This is the reason why the beat appears in the detection signal _Di. When multiple objects exist at different distances, multiple beats with different frequencies appear corresponding to the multiple distances.

Return to FIG. The arithmetic processing unit 130 includes a reconstruction processing unit 134 . The reconstruction processing unit 134 detects the intensity b ₁ to b _M of each beat of the detection signals D ₁ to D _M . When there are a plurality of beats with different frequencies, the intensity of the k-th beat corresponding to the r-th pattern irradiation is expressed as b ^k _r .

For example, the reconstruction processing unit 134 acquires the waveform of the detection signal _Dr obtained while the illumination light S1 corresponding to the r-th (r=1, 2, . . . M) intensity distribution _Ir is irradiated. Then, the captured waveform of the detection signal _Dr is converted into spectral data in the frequency domain by fast Fourier transform. The spectral data indicates the beat frequency f ^k and the beat intensity b ^k _r .

FIGS. 3A and 3B are diagrams for explaining beat intensity. In FIG. 3(a), the detection signal (also referred to as beat signal) _Dr that indicates the beat contains a single frequency ^f1 . By Fourier transforming this detection signal Dr, the intensity b ¹ _r of f ¹ is obtained.

In FIG. 3(b), the beat signal D _r includes multiple frequencies f ¹ and f ² . By Fourier transforming this detection signal _Dr , the intensity b ¹ _r of the beat with frequency f ¹ and the intensity b ² _r of the beat with frequency f ² are obtained.

Return to FIG. The reconstruction processing unit 134 performs correlation calculation of the intensity distributions I _{1 to I M of the illumination light S1 with the intensities b k 1} ^to ^b _k ^M _of the beats having different frequencies f ¹ . . _. reconstruct the restored image G ^k (x, y) of . Equation (1) can be used for correlation calculation.

The arithmetic processing unit 130 can be implemented by combining a processor (hardware) such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), or microcomputer, and a software program executed by the processor (hardware). Processing unit 130 may be a combination of multiple processors. Alternatively, the arithmetic processing unit 130 may be composed only of hardware.

The functions of the arithmetic processing unit 130 may be realized by software processing, hardware processing, or a combination of software processing and hardware processing. Specifically, software processing is implemented by combining processors (hardware) such as CPUs (Central Processing Units), MPUs (Micro Processing Units), microcomputers, and software programs executed by the processors (hardware). Note that the arithmetic processing unit 130 may be a combination of multiple processors. Specifically, hardware processing is implemented by hardware such as ASIC (Application Specific Integrated Circuit), controller IC, and FPGA (Field Programmable Gate Array).

The configuration of the imaging apparatus 100 is as described above. Next, the operation will be explained.

FIG. 4 is a diagram for explaining the operation of the imaging apparatus 100 of FIG. Here, for ease of understanding, consider a case where only a single object exists and the distance between the imaging apparatus 100 and the object does not change. In this case, the detection signal Dr contains a beat with a single frequency ^f1 . The pattern of the irradiation light _S1 is switched in order like _I1 , _I2 , _. . . , _IM for each of the time slots _TS1 , TS2, . Detection signals D ₁ , D _{2 .} _{. . , DM} _are obtained according to the irradiation patterns I ₁ , I ₂ . The beats represented by the detection signals D ₁ , D _{2 .} . . , D _M have different amplitudes depending on the pattern, but equal frequency f ¹ depending on the distance to the object. As described above, the amplitudes of the beats _of frequency ^f1 contained _{in the detection signals D1, D2, . . . , DM} _are _detected ^as _intensities ^b11 ^, _b12 , .

_, b ¹ _M _obtained ⁱⁿ a plurality of _time slots TS ₁ _, TS ₂ _, ^. , the restored image G ¹ (x, y) is calculated based on equation (1).

FIG. 5 is a diagram explaining the operation of the imaging apparatus 100 of FIG. Here, consider a case where two objects OBJ ¹ and OBJ ² are present at different distances, and the distances between the imaging apparatus 100 and each of the objects OBJ ¹ and OBJ ² do not change. In this case, the detection signal _Dr includes a beat of a first frequency ^f1 corresponding to the distance to the object ^OBJ1 and a beat of a second frequency ^f2 corresponding to the distance to the object ^OBJ2 .

The pattern of the irradiation light S1 is switched in order like I ₁ , I _{2 .} . . , _IM for each time slot. Detection signals D ₁ , D _{2 . .} _{. , DM} _are obtained according to the irradiation patterns I ₁ , I ₂ . The detection signals D ₁ , D _{2 .} . . , D _M each contain two frequency components f ¹ , f ² . The amplitude of the frequency f ¹ of each of the detection signals D ₁ , D _{2 .} . . , D _M changes based on the shape of the object OBJ ¹ and the patterns I ₁ to I _M . The amplitude of the frequency f ² of each of the detection signals D ₁ , D _{2 .} . . , D _M changes based on the shape of the object OBJ ² and the patterns I ₁ to I _M .

By Fourier transforming the detection signal _Dr , the intensity b ¹ _r at the frequency f ¹ and the intensity b ² _r at the frequency f ² can be obtained. ^Then , for ^the frequency f ¹ _, _the restored image _G ¹ ⁽ _x , y) can be generated. Similarly, the restored image G ² (x, y) of the object OBJ ² can be generated by performing correlation calculation using I ₁ to I _M and b ² ₁ to b ² _M for the frequency f ² .

The above is the operation of the imaging apparatus 100 . According to this imaging apparatus 100, by combining FMCW-LiDAR and ghost imaging, a three-dimensional image can be generated without scanning the illumination light S1. Since the imaging apparatus 100 does not require scanning as compared with the conventional FMCW-LiDAR, it can generate a two-dimensional range image in a short period of time. Also, since no mechanical or electronic control is required for scanning, costs can be reduced.

When multiple objects are present at different distances, the imaging apparatus 100 can generate an independent restored image G ^k (x, y) for each object. As a result, when object detection is performed by a classifier (discriminator) in the latter stage, since the objects are separated from the beginning, the recognition rate is reduced compared to the case where one image contains multiple objects. can be increased.

(Synthesis of images)
The imaging apparatus 100 may synthesize a plurality of restored images G ¹ (x, y), G ² (x, y), . . . obtained for a plurality of frequencies to generate one synthesized image IMG. FIG. 6 is a diagram for explaining combining of a plurality of restored images. Each image G ^k (x, y) is a monochrome multi-tone image.

The reconstruction processing unit 134 synthesizes the restored images G ¹ (x, y), G ² (x, y), . . . with different colors. Image G ¹ is assigned a first color C ₁ (eg red), image G ² is assigned a second color C ₂ (eg green) and eg image G ³ is assigned a third color C ₃ (eg blue).

Assume that the pixel value of each pixel of the original restored image is 8 bits (256 gradations) from 0 to 255. In this case, the pixel value of the restored image G ¹ (x, y) is the R component of the color composite image, and the pixel value of the restored image G ² (x, y) is the G component of the color composite image. The pixel values of the image G ³ (x, y) may be used as the B component of the color composite image.

It should be noted that the technique of adding color to and synthesizing multiple monochrome images is not limited to this. In the synthesized image, the colors represent the distance to the object, and the brightness of each color represents the reflectance of the object.

(change in distance to object)
In the discussion so far, the distance to the object was unchanged during one frame of sensing, so the frequency of one beat was constant during sensing. On the other hand, in in-vehicle applications, the distance to an object is not always constant during one frame for sensing an object. For example, if the vehicle is running and the object is stopped, the distance to the object will decrease over time. As the distance to the object changes, the frequency of the beat changes. That is, the beat frequency can change from time slot to time slot.

FIG. 7 is a diagram showing spectra obtained in a plurality of time slots. The frequency of the beat increases as the distance to the object increases with each time slot. In other words, the position of the peak included in the spectrum shifts to the high frequency side.

If the spectral data contains only a single peak, it is possible to perform correlation calculations assuming that they have the same beat frequency.

On the other hand, if the spectrum data contains multiple peaks, that is, multiple beats, the problem is how to associate them.

The arithmetic processing unit 130 calculates the velocity of the object based on the Doppler shift. Velocity estimation based on Doppler shift is a technique used in FMCW-LiDAR, so detailed description of processing is omitted. Processing unit 130 then estimates the distance to the object in the next time slot based on this velocity.

In other words, assume that the beat frequency is f in a certain time slot. Then the distance to the object is
d=f×c/(2×a)
is obtained by a is the frequency modulation chirp rate.

If the length of the time slot is Ts and the relative velocity of the object calculated from the Doppler shift is v, then the estimated distance d to the object in the next time slot is
d^=d−v×Ts
It is estimated to be.

Arithmetic processing unit 130 estimates the beat frequency based on the estimated distance d^ to the object. For example, the beat frequency f^ can be calculated from the following equation.
f^=a*2*d^/c

By using beat frequency estimation, beats in the previous time slot and beats in the current time slot can be appropriately associated when there are multiple beats with different frequencies.

FIG. 8 is a diagram showing spectra obtained in a plurality of time slots. Suppose there are two objects OBJ ¹ and OBJ ² and the relative velocity of one object OBJ ¹ is zero and the relative velocity of the other object OBJ ² is v. In this case, since the distance to the object ^OBJ1 does not change, the beat frequency of the reflected light is constant. On the other hand, the beat frequency of the reflected light from object ^OBJ2 changes with time. As the time slot elapses, the beat frequency ^f2 of the object ^OBJ2 approaches the beat frequency ^f1 of the object ^OBJ1 .

Assume that the velocity of the object OBJ ¹ was measured to be 0 at the time slot TS _i−1 based on past measurement results. Then the frequency f ¹ ^ of the beat of object OBJ ¹ in the next time slot TS _i is estimated to be the same as f ¹ . Also, assume that the velocity v of the object ^OBJ2 can be measured. Then the frequency f ² of the beat of object OBJ ² in the next time slot TS ₃ can be estimated.

The next time slot TS _i uses the frequencies f ¹ ̂, f ² ̂ predicted in the previous time slot TS _i−1 . That is, of the two beats included in the spectrum of the current time slot TS _i , the one close to f ¹ ^ is associated with the beat of object OBJ ¹ , and the one close to f ² ^ is associated with the beat of object OBJ ² . correspond to Also in this time slot TSi, the beat frequencies f ¹ ̂, f ² ̂ of the next time slot TS _i+1 are estimated.

The next time slot TS _i+1 utilizes the frequencies f ¹ ̂, f ² ̂ predicted in the previous time slot TS _i . That is, of the two beats included in the spectrum of the current time slot TS _i+1 , the one close to f ¹ ^ is associated with the beat of the object OBJ 1, and the one close to f ² ^{^} is associated with the beat of the object OBJ ² . correspond to

By estimating the beat frequency of future time slots using the information of past time slots, it is possible to appropriately associate beats caused by the same object over multiple time slots.

It should be noted that if the number of beats changes in multiple time slots within one frame, there is a possibility that the correspondence between objects and beats will be erroneous. Therefore, when the processing unit 130 detects a change in the number of beats, it may notify the outside. This notification can be used as an alert indicating the appearance of a new object in the field of view, or the disappearance of an object from the field of view, or indicates that the reliability of the reconstructed image obtained in that frame may be declining. do.

The imaging apparatus 100 may start measuring a new frame when the number of beats of the detection signal, that is, the number of peak frequencies changes. When the number of beats changes, the correspondence between beats and objects is lost, and the reliability of the restored image decreases. Therefore, when the number of peak frequencies changes, by starting measurement of a new frame from that point, it becomes possible to associate beats and objects one-to-one, and the accuracy of images can be improved.

(TDGI mode)
The reconstructed image generation processing in the reconstruction processing unit 134 of the arithmetic processing unit 130 may be performed by dividing M times of irradiation into a plurality of k (k≧2) units. Specifically, the reconstruction processing unit 134 performs correlation calculation for each divided unit to generate an intermediate image M _j (x, y). Then, the k intermediate images M ₁ to M _k (x, y) obtained for the k units are combined to generate the final restored image G _TDGI (x, y).

For simplicity, it is assumed that the number of irradiation times n of each unit is equal to M/k. The restored image G _TDGI (x, y) in this case is represented by Equation (2). I _r is the r-th intensity distribution, b _r is the value of the r-th detected intensity, and is the average value of the detected intensity b _r measured in the j-th unit. be. Note that the detection intensity _br is generated individually for each beat, and the restored image is also generated for each beat.

The j-th term on the right side of equation (2) represents the intermediate image M _j (x, y) of the j-th unit. Therefore, the restored image G _TDGI (x, y) is synthesized by simply adding the corresponding pixels of the plurality of intermediate images M _j (x, y). Note that the combining method is not limited to simple addition, and weighted addition or other processing may be performed.

In other words, the processor 130 performs the correlation calculation every n=M/k irradiations. Then, a restored image G _TDGI (x, y) is generated by synthesizing the k intermediate images M(x, y).

FIG. 9 is a time chart for explaining the operation of the imaging apparatus 100 in TDGI mode. In FIG. 9, the total number of pixels of the patterning device 114 is represented by p=4×4=16 pixels, and M random patterns are used for generating one restored image G(x, y) (capturing one frame). I ₁ to I _M are generated.

The reconstruction image is generated by dividing it into k units each containing n irradiations (that is, n time slots).

The first unit includes the 1st to nth irradiations, the detected intensities b ₁ to _bn corresponding to the n irradiations are generated, and their average value is generated. Then, correlation calculation is performed using the detected intensities b ₁ to b _n , their average value , and the intensity distributions I _n+1 to I _2n , and the intermediate image M ₁ (x, y) is generated.

The second unit contains the n+1 to 2n exposures, and the detected intensities b _n+1 to b _2n corresponding to the n exposures are generated and their average value is generated. Then, correlation calculation is performed using the detected intensities b _n+1 to b _2n , their average values , and the intensity distributions I _n+1 to I _2n , and the intermediate image M ₂ (x, y) is generated.

Similarly, the j-th unit includes (j-1)n+1 to jn-th irradiation, and the detection intensities b _(j-1)n+1 to b _jn corresponding to the n-th irradiation are generated, and their average A value is generated. Then, correlation calculation is performed using the detected intensities b _(j−1)n+1 to b _jn , their average values , and the intensity distributions I _(j−1)n+1 to I _jn , An intermediate image M _j (x,y) is generated.

The last k-th unit contains the (k−1)n+1 to kn-th irradiations, and the detected intensities b _(k−1)n+1 to b _kn corresponding to the n irradiations are generated, and their average is generated. Then, correlation calculation is performed using the detected intensities b _(k−1)n+1 to b _kn , their average values , and the intensity distributions I _(k−1)n+1 to I _kn , An intermediate image M _k (x,y) is generated.

A final restored image G(x, y) is generated by synthesizing the k intermediate images M ₁ (x, y) to M _k (x, y).

The above is the operation of the TDGI mode. According to the TDGI mode, by dividing the correlation calculation into units and performing it, the amount of noise change per correlation calculation can be reduced, the restoration accuracy can be increased, and the image quality can be improved. In addition, since correlation calculation can be started without waiting for the completion of M irradiations, the time required to generate a restored image can be shortened.

　In the following, the improvement of noise resistance in the TDGI mode will be described. Imaging using correlation calculation for each unit based on Equation (2) is hereinafter referred to as segmentation reconstruction. Also, conventional imaging using correlation calculation based on equation (1) is referred to as collective reconstruction.

For example, consider noise that monotonously increases over time (referred to as linear noise). Such linear noise can occur in cases where the distance between the noise light source and the beat detector approaches over time.

FIG. 10 is a diagram explaining another sequence in TDGI mode. In the example of FIG. 10, illumination light continues to be emitted, correlation calculation is performed every n times of illumination, and an intermediate image M(x, y) is generated. Then, every time an intermediate image M(x, y) is generated, a restored image G(x, y) is generated by synthesizing the latest k intermediate images M(x, y). According to this sequence, the restored image G(x, y) can be updated every n times of irradiation, so the update rate can be increased.

Next, noise in the TDGI mode will be explained. FIG. 11(a) is a diagram showing the influence of noise in batch restoration, and FIG. 11(b) is a diagram showing the influence of noise in division restoration. The horizontal axis represents the irradiation pattern number, that is, the time. σ _r represents the noise intensity when illuminating the r-th pattern, where it increases according to σ _r =α×r.

Please refer to FIG. When _collective restoration _is _performed , the absolute value _| Δσ _r _| The integrated value over irradiation corresponds to the hatched area and is expressed by the following equation (4).
Σ|Δσ _r |=2×1/2×(kn/2)×(knα/2)=n ² αk ² /4 (4)

Please refer to FIG. Here, k=5. When segmentation restoration is performed, the integrated value Σ|Δσ r | of the absolute value | _Δσ _r | of the difference Δσ _r (=σ _r −<σ _r ^[j] >) in the j-th unit is the hatched area. It corresponds and is represented by Formula (5).
Σ|Δσ _r |=2×1/2×(n/2)×(nα/2)=n ² α/4 (5)
The integrated value over all irradiation can be obtained by multiplying the value of equation (5) by k, which is kn ² α/4.

In this way, by performing division and restoration, the integrated value of the absolute value |Δσ _r | of the noise intensity difference can be reduced, and the noise immunity can be improved.

Next, the simulation results for collective restoration and divided restoration will be described.
FIG. 12(a) is a diagram showing a target image, and FIG. 12(b) is a diagram showing an image obtained by batch restoration and division restoration when noise does not exist. The simulation was performed for M=1000, M=10000 and M=100000. Also, the number of units k is assumed to be 10.

In batch restoration, M = 100000 is required in order to restore the original target image to a recognizable degree, and the same is true for segmented restoration. When M=100000, it can be said that the accuracy of the finally obtained restored image is the same between batch restoration and divisional restoration.

FIG. 13 is a diagram showing a restored image G(x, y) obtained by collective restoration and divisional restoration when linear noise exists. The total number of irradiations is M=10000, and the average value of the detected intensity is =1000. The noise coefficient α is calculated in four ways: 0, 0.01, 0.1, and 1. α=0 corresponds to no noise. As for split reconstruction, calculations are made for k=10, 100, and 1000.

As can be seen from the simulation results in FIG. 13, the higher the number of divisions k, the higher the noise resistance.

(Embodiment 2)
FIG. 14 is a block diagram of an imaging device 100A according to the second embodiment. Embodiment 2 differs from Embodiment 1 in the arrangement of spatial light modulators 140 . In FIG. 14, illumination light S1 is frequency-modulated continuous wave S0 and has a spatially uniform intensity distribution.

The spatial light modulator 140 spatially modulates the reflected light S2 from the object OBJ. The multiplexer 122 multiplexes the light S4 spatially modulated by the spatial light modulator 140 and the frequency-modulated continuous wave S0. Subsequent processing is the same as in the first embodiment.

According to the second embodiment, effects similar to those of the first embodiment can be obtained.

It should be understood by those skilled in the art that the embodiments are examples, and that various modifications can be made to the combination of each component and each treatment process, and that such modifications are within the scope of the present invention. Such modifications will be described below.

(Modification 1)
In TDGI, the number of times of irradiation for each unit is assumed to be equal, but this is not the case, and the number of times of irradiation for each unit may not be equal.

(Modification 2)
Also, in TDGI, the unit number k is fixed, but the unit number k may be dynamically controlled. Image quality can be further improved by selecting the optimum number of units k according to the noise fluctuation speed and noise waveform.

(Modification 3)
In Embodiment 1, the illumination device 110 is configured by a combination of the light source 112 and the patterning device 114, but this is not the only option. For example, the illumination device 110 may be composed of an array of a plurality of semiconductor light sources (LEDs (light emitting diodes) and LDs (laser diodes)) arranged in a matrix, and configured so that the brightness of each semiconductor light source can be controlled. .

(Application)
Next, the application of the imaging apparatus 100 will be described. FIG. 15 is a block diagram of the object identification system 10. As shown in FIG. This object identification system 10 is mounted on a vehicle such as an automobile or a motorcycle, and determines types (categories) of objects OBJ existing around the vehicle.

The object identification system 10 includes an imaging device 100 and an arithmetic processing device 40 . As described above, the imaging apparatus 100 generates the restored image G of the object OBJ by irradiating the object OBJ with the illumination light S1 and measuring the reflected light S2.

The arithmetic processing device 40 processes the output image G of the imaging device 100 and determines the position and type (category) of the object OBJ.

The classifier 42 of the arithmetic processing unit 40 receives the image G as an input and determines the position and type of the object OBJ contained therein. Classifier 42 is implemented based on a model generated by machine learning. Although the algorithm of the classifier 42 is not particularly limited, YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), R-CNN (Region-based Convolutional Neural Network), SPPnet (Spatial Pyramid Pooling), Faster R-CNN , DSSD (Deconvolution-SSD), Mask R-CNN, etc., or algorithms that will be developed in the future.

The above is the configuration of the object identification system 10. By using the imaging device 100 as a sensor for the object identification system 10, the following advantages can be obtained.

By using the imaging device 100, that is, the quantum radar camera, the noise immunity is greatly improved. For example, when it is raining, snowing, or driving in fog, it is difficult to recognize the object OBJ with the naked eye. A restored image G can be obtained.

Also, by using TDGI, the calculation delay can be reduced. This can provide low latency sensing. Particularly in in-vehicle applications, there are cases where the object OBJ moves at high speed.

FIG. 16 is a block diagram of an automobile equipped with the object identification system 10. FIG. Automobile 300 includes

headlights

302L and 302R. Imaging device 100 is built into at least one of

headlights

302L and 302R. The headlamp 302 is positioned at the extreme end of the vehicle body and is the most advantageous location for installing the imaging device 100 in terms of detecting surrounding objects.

FIG. 17 is a block diagram showing a vehicle lamp 200 including an object detection system 210. FIG. The vehicle lamp 200 constitutes a lamp system 310 together with a vehicle-side ECU 304 . A vehicle lamp 200 includes a light source 202 , a lighting circuit 204 and an optical system 206 . Furthermore, the vehicle lamp 200 is provided with an object detection system 210 . Object detection system 210 corresponds to object identification system 10 described above and includes imaging device 100 and arithmetic processing device 40 .

Information on the object OBJ detected by the processing unit 40 may be used for light distribution control of the vehicle lamp 200 . Specifically, the lamp-side ECU 208 generates an appropriate light distribution pattern based on the information about the type and position of the object OBJ generated by the arithmetic processing unit 40 . The lighting circuit 204 and the optical system 206 operate so as to obtain the light distribution pattern generated by the lamp-side ECU 208 .

Information regarding the object OBJ detected by the arithmetic processing unit 40 may be transmitted to the vehicle-side ECU 304 . The vehicle-side ECU may perform automatic driving based on this information.

Although the present invention has been described using specific terms based on the embodiments, the embodiments merely show one aspect of the principles and applications of the present invention, and the embodiments are defined in the scope of claims. Many modifications and changes in arrangement are permitted without departing from the spirit of the present invention.

The present disclosure relates to imaging apparatus and methods.

OBJ... Object, S0... Frequency-modulated continuous wave, S1... Illumination light, S2... Reflected light, S3... Combined light, 10... Object identification system, 40... Arithmetic processing unit, 42... Classifier, 100... Imaging device, 110... Illuminating device 112 Light source 114 Patterning device 120 Beat detector 122 Combiner 124 Photodetector 130 Arithmetic processor 132 Pattern generator 134 Reconstruction processor 200 Vehicle lamp 202 Light source 204 Lighting circuit 206 Optical system 300 Automobile 302 Headlamp 310 Lamp system 304 Vehicle side ECU.

Claims

A light source that generates a frequency-modulated continuous wave, and a spatial light modulator that spatially modulates the frequency-modulated continuous wave with a different pattern for each time slot, wherein the spatially-modulated frequency-modulated continuous wave is used as illumination light in a field of view. a lighting device for irradiation;
a beat detector that generates a detection signal based on the light reflected from the object and the combined light of the frequency-modulated continuous wave;
an arithmetic processing unit that performs correlation calculation between the intensity of the beat of the detection signal and the intensity distribution of the illumination light for each beat of the detection signal, and reconstructs a restored image of the object;
An imaging device comprising:
an illumination device that includes a light source that generates a frequency-modulated continuous wave and irradiates a field of view with the frequency-modulated continuous wave as illumination light;
a spatial light modulator that modulates reflected light from an object with a different pattern for each time slot;
a beat detector that generates a detection signal based on the combined light of the reflected light modulated by the spatial light modulator and the frequency-modulated continuous wave;
an arithmetic processing unit that performs correlation calculation between the intensity of the beat of the detection signal and the intensity distribution of the illumination light for each beat of the detection signal, and reconstructs a restored image of the object;
An imaging device comprising:
The imaging apparatus according to claim 1, wherein the arithmetic processing unit Fourier-transforms the detection signal and detects the intensity of the beat.
3. The imaging apparatus according to claim 1 or 2, wherein the arithmetic processing unit adds different colors to the plurality of restored images and synthesizes them.
4. The processor of claim 1, wherein the arithmetic processing unit calculates the velocity of the object based on the Doppler shift, and estimates the distance to the object in the next time slot based on the velocity. An imaging device according to any one of the preceding claims.
The imaging apparatus according to claim 4, wherein the arithmetic processing unit estimates the beat frequency of the next time slot based on the estimated distance to the object.
The imaging apparatus according to any one of claims 1 to 5, wherein when the number of beats of the detection signal changes, it is notified to the outside.
The imaging apparatus according to any one of claims 1 to 5, wherein when the number of beats of the detection signal changes, measurement of a new frame is started from there.
The imaging apparatus according to any one of claims 1 to 7, wherein the light source and the spatial light modulator are arrays of light emitting elements.
A vehicle lamp comprising the imaging device according to any one of claims 1 to 8.
A vehicle comprising the imaging device according to any one of claims 1 to 8.
generating a frequency modulated continuous wave;
modulating the frequency-modulated continuous wave with a different pattern for each time slot to generate illumination light;
generating a detection signal based on the combined light of the reflected light of the illumination light reflected by the object and the frequency-modulated continuous wave;
performing a correlation calculation between the intensity distribution of the illumination light and the intensity of the beat for each beat of the detection signal, and reconstructing a restored image of the object;
An imaging method, comprising:
illuminating the field of view with a frequency-modulated continuous wave as illumination light;
modulating the light reflected by the object from the illumination light with a different pattern for each time slot;
generating a detection signal based on the combined light of the modulated reflected light and the frequency-modulated continuous wave;
performing a correlation calculation between the intensity distribution of the illumination light and the intensity of the beat for each beat of the detection signal, and reconstructing a restored image of the object;
An imaging method, comprising: