CN110493535B

CN110493535B - Image acquisition device and image acquisition method

Info

Publication number: CN110493535B
Application number: CN201910472706.0A
Authority: CN
Inventors: 於敏杰; 聂鑫鑫; 罗丽红
Original assignee: Hangzhou Hikvision Digital Technology Co Ltd
Current assignee: Hangzhou Hikvision Digital Technology Co Ltd
Priority date: 2019-05-31
Filing date: 2019-05-31
Publication date: 2021-09-28
Anticipated expiration: 2039-05-31
Also published as: WO2020238827A1; CN110493535A

Abstract

The disclosure provides an image acquisition device and an image acquisition method, and belongs to the technical field of videos. The image acquisition device comprises an image acquisition component and a light supplementing component, the image acquisition component comprises a light filtering component and an image sensor, the image sensor is used for sensing through multiple rolling shutter exposure and outputting a first image signal and a second image signal, the first image signal is an image signal generated according to the first exposure, the second image signal is an image signal generated according to the second exposure, the light filtering component comprises a first light filtering device used for enabling visible light and partial near infrared light to pass through, the light supplementing component comprises a first light supplementing device used for not performing near infrared light supplementing during the first exposure and performing near infrared light supplementing during the second exposure, the starting time of performing near infrared light supplementing during the second exposure is determined according to a first moment, and the ending time of performing near infrared light supplementing during the second exposure is determined according to a second moment. By adopting the method and the device, the realization difficulty can be reduced.

Description

Image acquisition device and image acquisition method

Technical Field

The present disclosure relates to the field of video technologies, and in particular, to an image capturing apparatus and an image capturing method.

Background

With the development of computer technology, human faces are more visual compared with physical features such as fingerprints and irises, so that the human face detection technology is gradually applied to security systems such as entrance guards.

In the related art, in order to prevent an illegal user from using a face picture of a legal user through a system such as an access control system, a face detection device is generally connected with a binocular camera, and the binocular camera includes a visible light camera and an infrared camera. When the human face detection is carried out, the detection equipment acquires a visible light image shot by a visible light camera and an infrared image shot by an infrared camera at the same moment, determines whether the human face is a living human face through the infrared images, and determines whether the human face is a legal user through the visible light images.

In this way, two cameras are required for obtaining the infrared image and the visible light image, but the installation of the two cameras requires higher processing precision, so that the realization difficulty is higher.

Disclosure of Invention

In order to solve the problem of high implementation difficulty in the related art, the embodiment of the disclosure provides an image acquisition device and an image acquisition method. The technical scheme is as follows:

in a first aspect, an image capturing apparatus is provided, the image capturing apparatus comprising:

the device comprises an image acquisition component and a light supplementing component, wherein the image acquisition component comprises a light filtering component and an image sensor;

the image sensor is used for sensing and outputting a first image signal and a second image signal through multiple rolling shutter exposures, wherein the first image signal is an image signal generated according to a first exposure, the second image signal is an image signal generated according to a second exposure, and the first exposure and the second exposure are two exposures in the multiple rolling shutter exposures;

the filtering component comprises a first filtering device and a second filtering device, wherein the first filtering device is used for allowing visible light and partial near infrared light to pass through;

the light supplementing component comprises a first light supplementing device, wherein the first light supplementing device is used for not performing near-infrared light supplementing during the first exposure and performing near-infrared light supplementing during the second exposure, the starting time of performing near-infrared light supplementing during the second exposure is determined at least according to a first moment, and the ending time of performing near-infrared light supplementing during the second exposure is determined at least according to a second moment;

the first image signal comprises a plurality of lines of effective image signals, the second image signal comprises a plurality of lines of effective image signals, and when the second exposure is the current exposure, the first time is the exposure starting time of the first line of effective image signals of the second image signal generated by the current exposure, and the second time is the exposure ending time of the last line of effective image signals of the second image signal generated by the current exposure.

As a possible implementation manner, the starting time of performing the near-infrared supplementary lighting during the second exposure is not later than the first time, and the ending time of performing the near-infrared supplementary lighting during the second exposure is not earlier than the second time.

As a possible implementation manner, the starting time of performing the near-infrared supplementary lighting at the second exposure is determined according to the first time and a third time, where the third time is an ending exposure time of a last line of effective image signals of a first image signal generated by the last first exposure before the current exposure, and the first time is no earlier than the third time.

As a possible implementation manner, the starting time of performing the near-infrared supplementary lighting at the second exposure is not earlier than the third time and not later than the first time.

As a possible implementation manner, the ending time of the near-infrared supplementary lighting during the second exposure is determined according to the second time and a fourth time, the fourth time is the starting exposure time of the first line effective image signal of the first image signal generated by the first exposure which is the latest time after the current exposure, and the second time is not later than the fourth time.

As a possible implementation manner, the ending time of the near-infrared supplementary lighting in the second exposure is not earlier than the second time and not later than the fourth time.

As a possible implementation manner, in the current exposure, the duration of the near-infrared supplementary lighting performed by the first supplementary lighting device is not less than the sum of the exposure duration of any line of the effective image signal of the current exposure and the readout duration of the effective image signal of the second image signal generated by the current exposure.

As a possible implementation manner, the fill-in duration of each row of effective image signals of the second image signal generated by the current exposure is the same.

As a possible implementation manner, the multiple rolling shutter exposures include multiple exposure periods, and each exposure period includes at least one of the first exposure and the second exposure.

As a possible implementation manner, the image acquisition component further includes a lens;

the light filtering component is positioned between the lens and the image sensor, and the image sensor is positioned on the light emitting side of the light filtering component; alternatively, the first and second electrodes may be,

the lens is located between the light filtering component and the image sensor, and the image sensor is located on the light emitting side of the lens.

As a possible implementation manner, the image sensor includes a plurality of photosensitive channels, the plurality of photosensitive channels include at least one of an R photosensitive channel, a G photosensitive channel, a B photosensitive channel, and a W photosensitive channel, and the plurality of photosensitive channels generate and output the first image signal and the second image signal through the multiple rolling shutter exposures;

the system comprises a light source, a light sensing channel, an R light sensing channel, a G light sensing channel, a B light sensing channel and a W light sensing channel, wherein the R light sensing channel is used for sensing light of a red light wave band and a near infrared wave band, the G light sensing channel is used for sensing light of a green light wave band and a near infrared wave band, the B light sensing channel is used for sensing light of a blue light wave band and a near infrared wave band, and the W light sensing channel is used for sensing light of a full wave band.

As a possible implementation manner, the image sensor is any one of a red, green, blue and white RGBW sensor, a red, white, blue RCCB sensor, a red, green, blue RGB sensor, or a red, yellow, blue RYYB sensor;

wherein, R represents an R photosensitive channel, G represents a G photosensitive channel, B represents a B photosensitive channel, and W represents a W photosensitive channel.

As a possible implementation, the first exposure and the second exposure have different at least one exposure parameter, the at least one exposure parameter is one or more of exposure time, exposure gain, aperture size, and the exposure gain includes analog gain, and/or digital gain.

As a possible implementation, the exposure gain of the second exposure is smaller than the exposure gain of the first exposure.

As a possible implementation, at least one exposure parameter of the first exposure and the second exposure is the same, the at least one exposure parameter includes one or more of exposure time, exposure gain, aperture size, the exposure gain includes analog gain, and/or digital gain.

As a possible implementation, the exposure time of the first exposure is equal to the exposure time of the second exposure.

As a possible implementation manner, the filtering component further includes a second filtering device and a switching component, and both the first filtering device and the second filtering device are connected to the switching component;

the switching component is used for switching the second light filtering device to the light inlet side of the image sensor;

after the second filtering device is switched to the light incident side of the image sensor, the second filtering device allows light in a visible light wave band to pass through and blocks light in a near infrared light wave band, and the image sensor is used for generating and outputting a third image signal through exposure.

As a possible implementation manner, the light supplement component further includes a second light supplement device;

the second light supplementing device is used for supplementing visible light in a normally bright mode; alternatively, the first and second electrodes may be,

the second light supplement device is used for supplementing visible light in a stroboscopic mode, wherein the supplementary visible light exists at least in part of the exposure time period of the second exposure, and the supplementary visible light does not exist in the whole exposure time period of the first exposure; alternatively, the first and second electrodes may be,

the second light supplement device is used for supplementing visible light in a stroboscopic mode, wherein the supplementary visible light does not exist at least in the whole exposure time period of the second exposure, and the supplementary visible light exists in part of the exposure time period of the first exposure.

As a possible implementation manner, when the central wavelength of the near-infrared supplementary lighting performed by the first supplementary lighting device is a set characteristic wavelength or falls within a set characteristic wavelength range, the central wavelength and/or the band width of the near-infrared light passing through the first filtering device reach the constraint condition.

As a possible implementation manner, the center wavelength of the near-infrared supplementary lighting performed by the first supplementary lighting device is any wavelength within a wavelength range of 750 ± 10 nanometers; or

The center wavelength of the near-infrared supplementary lighting performed by the first supplementary lighting device is any wavelength within the wavelength range of 780 +/-10 nanometers; or

The center wavelength of the first light supplement device for near-infrared light supplement is any wavelength within a wavelength range of 940 +/-10 nanometers.

As a possible implementation, the constraint condition includes: the difference value between the central wavelength of the near-infrared light passing through the first light filtering device and the central wavelength of the near-infrared light supplemented by the first light supplementing device is within a wavelength fluctuation range, and the wavelength fluctuation range is 0-20 nanometers; alternatively, the first and second electrodes may be,

the half bandwidth of the near infrared light passing through the first filtering device is less than or equal to 50 nanometers; alternatively, the first and second electrodes may be,

the first wave band width is smaller than the second wave band width; wherein the first wavelength band width refers to the wavelength band width of the near infrared light passing through the first filter device, and the second wavelength band width refers to the wavelength band width of the near infrared light blocked by the first filter device; alternatively, the first and second electrodes may be,

the third wave band width is smaller than the reference wave band width, the third wave band width refers to the wave band width of the near infrared light with the passing rate larger than the set proportion, and the reference wave band width is any wave band width in the wave band range of 50-150 nanometers.

In a second aspect, there is provided an image capturing method, applied to an image capturing apparatus, the image capturing apparatus including: the image sensor, light filling ware part and filtering part, light filling part includes first light filling device, filtering part includes first light filtering device, the method includes:

performing near-infrared supplementary lighting through the first supplementary lighting device, wherein the near-infrared supplementary lighting is not performed during the first exposure, the near-infrared supplementary lighting is performed during the second exposure, the starting time of the near-infrared supplementary lighting during the second exposure is determined at least according to the first time, and the ending time of the near-infrared supplementary lighting during the second exposure is determined at least according to the second time;

passing visible light and a portion of near-infrared light through the first filtering means;

performing multiple rolling shutter exposure sensing through the image sensor and outputting a first image signal and a second image signal, wherein the first image signal is an image signal generated according to the first exposure, and the second image signal is an image signal generated according to the second exposure;

the first image signal comprises a plurality of lines of effective image signals, the second image signal comprises a plurality of lines of effective image signals, when the second exposure is the current exposure, the first time is the starting exposure time of the first line of effective image signals of the second image signal generated by the current exposure, and the second time is the ending exposure time of the last line of effective image signals of the second image signal generated by the current exposure

The beneficial effects brought by the technical scheme provided by the embodiment of the disclosure at least comprise:

in the embodiment of the disclosure, the exposure time sequence of the image sensor is used for controlling the near-infrared light supplement time sequence of the light supplement component, so that a second image signal is generated through second exposure when near-infrared light supplement exists, and a first image signal is generated through first exposure when the near-infrared light supplement does not exist. Moreover, the first image signal and the second image signal are both generated and output by the same image sensor, so that the viewpoint corresponding to the first image signal is the same as the viewpoint corresponding to the second image signal, and therefore, the information of an external scene can be obtained through the first image signal and the second image signal together, and the phenomenon that images generated according to the first image signal and the second image signal are not aligned due to the fact that the viewpoint corresponding to the first image signal is different from the viewpoint corresponding to the second image signal does not exist.

Drawings

Fig. 1 is a schematic diagram of a rolling shutter exposure provided by an embodiment of the present disclosure;

fig. 2 is a schematic diagram of a rolling shutter exposure provided by an embodiment of the present disclosure;

fig. 3 is a schematic structural diagram of an image capturing device according to an embodiment of the present disclosure;

fig. 4 is a schematic diagram of a rolling shutter exposure provided by an embodiment of the present disclosure;

fig. 5 is a schematic diagram of a rolling shutter exposure provided by an embodiment of the present disclosure;

fig. 6 is a schematic diagram of a rolling shutter exposure provided by an embodiment of the present disclosure;

fig. 7 is a schematic diagram of a rolling shutter exposure provided by an embodiment of the present disclosure;

fig. 8 is a schematic diagram of a rolling shutter exposure provided by an embodiment of the present disclosure;

fig. 9 is a schematic structural diagram of an image capturing device according to an embodiment of the present disclosure;

FIG. 10 is a schematic diagram of a light throughput provided by an embodiment of the present disclosure;

fig. 11 is a schematic structural diagram of an image capturing device according to an embodiment of the present disclosure;

fig. 12 is a schematic structural diagram of an image capturing device according to an embodiment of the present disclosure;

FIG. 13 is a schematic diagram of a channel structure of an RGBW sensor provided by an embodiment of the present disclosure;

fig. 14 is a schematic diagram of a channel structure of an RCCB sensor according to an embodiment of the present disclosure;

FIG. 15 is a schematic diagram of a channel structure of an RGB sensor provided in an embodiment of the present disclosure;

fig. 16 is a schematic diagram of a channel structure of an RYYB sensor provided in an embodiment of the present disclosure;

FIG. 17 is a schematic diagram of a spectral response provided by an embodiment of the present disclosure;

FIG. 18 is a schematic illustration of an exposure sequence provided by an embodiment of the present disclosure;

FIG. 19 is a schematic illustration of an exposure sequence provided by an embodiment of the present disclosure;

FIG. 20 is a schematic illustration of an exposure sequence provided by an embodiment of the present disclosure;

fig. 21 is a schematic flowchart of a method for image acquisition according to an embodiment of the present disclosure.

Description of the figures

Image acquisition part 1 light filling part 2

Filter member 11 image sensor 12

First filter 111 of lens 13

Second filter 112 switching member 113

First light supplement device 21 and second light supplement device 22

Detailed Description

To make the objects, technical solutions and advantages of the present disclosure more apparent, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

First, technical terms related to the embodiments of the present disclosure will be briefly described.

Visible light is electromagnetic waves which can be perceived by human eyes, the visible spectrum has no precise range, and the wavelength of the electromagnetic waves which can be perceived by the human eyes is 400nm (nanometer) to 760 nm.

The visible light image is a color image in which only visible light signals are perceived, and the color image is only sensitive to a visible light band.

The infrared image is a luminance image for sensing a near-infrared light signal.

The rolling shutter exposure means that the exposure time of different lines of the image sensor is not completely overlapped, and the exposure starting time and the exposure ending time of each line are gradually shifted backwards for a period of time line by line.

As shown in fig. 1, the horizontal axis represents time, the vertical axis represents a pixel row number of the image sensor, for each row of pixels, the open dots represent exposure start time, the solid dots represent exposure end time, the first row of pixels of the image sensor starts exposure at time T1, ends exposure at time T3, the second row of pixels starts exposure at time T2, ends exposure at time T4, T2 is shifted backward by a period of time compared to T1, T4 is shifted backward by a period of time compared to T3, and so on for the other rows.

The readout time is a time from the end of exposure of the first row of pixels to the end of exposure of the last row of pixels and data output, and represents the readout time.

As shown in fig. 1, when the first row of pixels ends exposure at time T3, starts outputting data, ends outputting data at time T5, and the last row of pixels ends exposure at time T6, and ends outputting data at time T7, the time from T3 to T7 is the readout time, as shown by the rectangle in fig. 1.

As shown in fig. 2, a simplified schematic diagram of multiple rolling shutter exposures is shown, where the horizontal direction represents time, the dotted line represents the start time of each exposure (the start time of the exposure forms a slant line because the start time of each row of pixels is different), the solid line represents the end time of each exposure (the end time of the exposure forms a slant line because the end time of each row of pixels is different), and the rectangle represents the readout time.

In the correlation technique, obtain the infrared image and the visible light image of an object, need use the binocular camera, lead to realizing the degree of difficulty height, in order to reduce the realization degree of difficulty, this disclosure provides the image acquisition device of following structure.

Fig. 3 is a schematic structural diagram of an image capturing device provided in an embodiment of the present disclosure, where the image capturing device includes an image capturing component 1 and a light supplement component 2, and the image capturing component 1 includes a light filtering component 11, an image sensor 12, and a lens 13 (the lens 13 is not shown in fig. 3). The image sensor 12 is configured to generate and output a first image signal and a second image signal through multiple rolling shutter exposures, the first image signal being an image signal generated according to the first exposure, the second image signal being an image signal generated according to the second exposure, the first exposure and the second exposure being two exposures of the multiple rolling shutter exposures. The filter component 11 includes a first filter device 111 for passing visible light and part of near infrared light. The light supplement component 2 comprises a first light supplement device 21, the first light supplement device 21 is used for not performing near-infrared light supplement during first exposure and performing near-infrared light supplement during second exposure, the starting time of performing near-infrared light supplement during second exposure is determined at least according to the first time, and the ending time of performing near-infrared light supplement during second exposure is determined at least according to the second time. When the second exposure is the current exposure, the first time is the starting exposure time of the first line effective image signal of the second image signal generated by the current exposure, and the second time is the ending exposure time of the last line effective image signal of the second image signal generated by the current exposure

Wherein the first image signal and the second image signal each include a plurality of lines of effective image signals, the effective image signals being image signals finally read out. For example, a rolling shutter is exposed with 1000 rows of photosensitive channels, and the number of rows of pixel points of the finally output image is 800 rows, so that the effective image signal is an image signal output by 800 rows of photosensitive channels in the 1000 rows of photosensitive channels, and may be from row 1 to row 800, or from row 100 to row 900, which is not limited in the embodiment of the present disclosure.

In an implementation, the image capturing device may include an image capturing part 1 and a light supplementing part 2, and the image capturing part 1 may include a filtering part 11, an image sensor 12, and a lens 13. The light supplementing component 2 includes a first light supplementing device 21, the first light supplementing device 21 is used for supplementing near-infrared light, the light filtering component 11 includes a first light filtering device 111, and the first light filtering device 111 is used for only allowing visible light and near-infrared light to pass through.

The filter member 11 may be composed of a filter using a coating technique, and may be used to pass visible light and near-infrared light, or to pass visible light and part of near-infrared light, that is, the first filter device 111 passes visible light and near-infrared light, or to pass visible light and part of near-infrared light.

The image sensor 12 may be configured to sense and output a first image signal and a second image signal in a multi-shutter exposure, the first image signal being an image signal generated according to a first exposure, the second image signal being an image signal generated according to a second exposure, the first exposure and the second exposure being two exposures of the multi-shutter exposure.

When carrying out first exposure, near-infrared light filling can not be carried out to light filling part 2, if light filling part 2 is near-infrared light filling lamp, can close near-infrared light filling lamp etc. makes the exposure time quantum of first exposure pass through by the light that the object reflection got into light filtering part 11 of shooing be ambient light, and at this moment image sensor 12 can use the exposure parameter of first exposure, carries out the exposure of roll curtain formula shutter, outputs first image signal.

When the second exposure is performed, the light supplement component 2 may perform infrared light supplement, for example, the light supplement component 2 is a near-infrared light supplement lamp, and the near-infrared light supplement lamp may be turned on, and when the second exposure is the current exposure (which refers to the exposure being performed at the current time), the starting time of performing the near-infrared light supplement may be determined according to a first time, the first time may be the starting exposure time of the first row of effective image signals of the second image signal generated by the current exposure, and the ending time of performing the near-infrared light supplement may be determined according to a second time, and the second time may be the ending exposure time of the last row of effective image signals of the second image signal generated by the current exposure. Between the start time and the end time, the fill-in light unit 2 may perform near-infrared fill-in light, and the image sensor 12 may perform rolling shutter exposure with exposure parameters of the second exposure, and output a second image signal.

Note that the multiple rolling shutter exposure includes multiple first exposures and multiple second exposures, for example, both the first exposure and the second exposure are performed multiple times within one second, and the order of the first exposure and the second exposure may be configured in advance.

The light supplement component 2 may include a first light supplement device 21, and the first light supplement device 21 may be located in the image acquisition device or located outside the image acquisition device. The first light supplement device 21 may be a part of the image capturing device, or may be a device independent from the image capturing device. When the first light supplement device 21 is located outside the image acquisition device, the first light supplement device 21 can be in communication connection with the image acquisition device, so that the exposure time sequence of the image sensor 12 in the image acquisition device and the near-infrared light supplement time sequence of the first light supplement device 21 can be ensured to have a certain relationship.

As a possible implementation manner, in order to perform near-infrared supplementary lighting within the exposure time of the effective image signal during the second exposure, the near-infrared supplementary lighting is performed at the time of starting exposure of the first row of effective image signals during the second exposure, and the near-infrared supplementary lighting is also performed at the time of ending exposure of the last row of effective image signals during the second exposure, that is, the time of starting near-infrared supplementary lighting during the second exposure cannot be later than the first time, and the time of ending near-infrared supplementary lighting during the second exposure cannot be earlier than the second time.

For example, as shown in fig. 4 (a), the first exposure and the second exposure of the image sensor 12 are performed alternately, the first light supplement device 21 corresponds to a first light supplement state during the first exposure, the first light supplement device 21 corresponds to a second light supplement state during the second exposure, the first light supplement state is off, the first light supplement device 21 does not perform near-infrared light supplement, the second light supplement state is on, the first light supplement device 21 performs near-infrared light supplement, the previous frame image signal is the first image signal, the current frame image signal (i.e. the image signal generated by the current exposure) is the second image signal, the current exposure is the second exposure and generates the second image signal, when the second fill-in state corresponding to the second image signal turns on the near-red infrared light, the time when the first fill-in device 21 starts the near-infrared fill-in is no later than the exposure start time of the first row effective image signal of the current frame image signal. Further, the time when the first fill-in light device 21 ends the near-infrared fill-in light is not earlier than the time when the last row of effective image signals of the current frame of image signals ends the exposure.

As a possible implementation manner, when the current exposure is the second exposure, in order not to affect the first exposure, the starting time of performing the near-infrared light supplement during the second exposure is determined according to the first time and a third time, where the third time is the ending exposure time of the last line effective image signal of the first image signal generated by the last first exposure before the current exposure. In addition, in order to prevent the near-infrared supplementary lighting from affecting the first exposure, the first time is not earlier than the third time.

As a possible implementation manner, when the current exposure is the second exposure, in order to not affect the first exposure and enable the first line effective image signals of the second exposure to perform near-infrared supplementary lighting, the near-infrared supplementary lighting is not generated before the last line effective image of the first exposure of the previous frame is exposed, and the near-infrared supplementary lighting is generated when the first line effective image signals of the current exposure start to be exposed, so that the starting time of performing the near-infrared supplementary lighting in the second exposure is not earlier than the third time and is not later than the first time.

For example, as shown in fig. 4 (b), near-infrared supplementary lighting is started within two vertical dotted lines (between the third time and the first time), the dotted line near the left is 5 ms, the dotted line near the right is 8ms, and the near-infrared supplementary lighting may be started at 6 ms.

As a possible implementation manner, in order to enable the last line of effective image signals of the second exposure to perform near-infrared supplementary lighting without affecting the first exposure, an end time of the second exposure to perform near-infrared supplementary lighting may be determined using a second time and a fourth time, where the fourth time is a start exposure time of the first line of effective image signals of the first image signals generated by the first exposure of the next frame.

As a possible implementation manner, when the current exposure is the second exposure, in order to not affect the first exposure and enable the last line of effective image signals of the second exposure to perform near-infrared supplementary lighting, not only the near-infrared supplementary lighting is not available after the first line of effective image signals of the first exposure of the next frame starts to be exposed, but also the near-infrared supplementary lighting is available when the last line of effective image signals of the current exposure finishes to be exposed, so that the starting time of performing the near-infrared supplementary lighting in the second exposure is not earlier than the second time and not later than the fourth time.

For example, as shown in fig. 4 (c), the near-infrared fill-in light ends within two vertical dashed lines (between the second time and the fourth time), the dashed line near the left is 25 ms, the dashed line near the right is 28 ms, and the near-infrared fill-in light may end at 26 ms.

As a possible implementation manner, as shown in fig. 5, when the current exposure is the second exposure, in order to make each line of effective image signals completely perform near-infrared light compensation during the exposure time when the second exposure is performed (here, the near-infrared light compensation of the first line of effective image signals of the current exposure is affected), the starting time of performing the near-infrared light compensation is no later than the starting exposure time of the first line of effective image signals of the current exposure (if the starting exposure time of the first line of effective image signals is later than the starting exposure time of the first line of effective image signals, the first line of effective image signals is not completely perform the near-infrared light compensation during the exposure time), if the latest first exposure before the current exposure is not affected, the starting time of performing the near-infrared light compensation may be no earlier than the ending exposure time of the last line of effective image signals of the last frame of first exposure (if the ending exposure time of the last line of effective image signals is earlier than the ending exposure time of the last line of effective image signals, the first exposure of the previous frame will have near infrared fill). Therefore, the starting time of the near-infrared supplementary lighting during the second exposure is not earlier than the third time and not later than the first time. In order to completely perform near-infrared light supplement on each line of effective image signals within the exposure time during the second exposure (here, the influence is the near-infrared light supplement of the last line of effective image signals of the current exposure), the ending time of the near-infrared light supplement may be no earlier than the ending time of the last line of effective image signals of the current exposure (if the ending time of the last line of effective image signals is earlier than the ending time of the last line of effective image signals, the last line of effective image signals does not completely perform near-infrared light supplement within the exposure time), if the latest first exposure after the current exposure is not influenced, the ending time of the near-infrared supplementary lighting is not later than the starting exposure time of the first line effective image signal of the first exposure of the next frame (if the ending time of the near-infrared supplementary lighting is later than the starting exposure time of the first line effective image signal, the near-infrared supplementary lighting is provided at the first exposure of the next frame). Therefore, the ending time of the near-infrared supplementary lighting during the second exposure is not earlier than the second time and not later than the fourth time. Therefore, near-infrared supplementary lighting is completely carried out on each line of effective image signals of the current exposure within the exposure time, so that the supplementary lighting time of near-infrared light is longer.

And because the exposure time of any line of effective image signals is the same, the light filling duration of each line of effective image signals of the second image signals generated by the current exposure is the same.

It should be noted that, for more clearly describing the relationship between the fill-in light and the exposure, it may be considered that the exposure time and the fill-in light time satisfy a certain constraint:

and during the second exposure, the first light supplementing device 21 performs near-infrared light supplementing, if the first exposure and the second exposure are performed alternately, the current exposure is the second exposure, and the starting time of performing near-infrared light supplementing is not earlier than the ending exposure time of the last line of effective image signals of the previous frame and not later than the starting exposure time of the currently exposed first line of effective image signals. The ending time of the near-infrared supplementary lighting is not earlier than the ending exposure time of the last line of effective image signals of the current exposure and not later than the starting exposure time of the first line of effective image signals of the next frame.

It should be noted that the first and second concepts are merely relative concepts and may be interchanged with each other.

As a possible implementation manner, in the current exposure, the exposure start time of each line of effective image signals of the current exposure is different, the effective image signals are started at the latest at the exposure start time of the first line of effective image signals of the current exposure, and the effective image signals of the last line of effective image signals of the current exposure are ended at the earliest, the near-infrared light supplement is performed for the whole exposure time period of the effective image signals of the second exposure, so that the near-infrared light supplement is performed from the exposure start time of the first line of effective image signals to the exposure end time of the last line of effective image signals, and therefore, the duration of the near-infrared light supplement performed by the first light supplement device is not less than the sum of the exposure time of any line of effective image signals of the current exposure and the readout duration of the effective image of the second image signals generated by the current exposure.

In one example, as shown in fig. 6, the image sensor performs exposure alternately under a exposure and a exposure B, during the exposure a, the first fill-in device 21 is turned off to output an image signal without performing near-infrared fill-in (i.e., a first image signal), and during the exposure B, the first fill-in device 21 is turned on to output an image signal with a fill-in lamp performing near-infrared fill-in (i.e., a second image signal). The first light supplementing device 21 is turned on no earlier than the ending exposure time of the last line of effective image signals exposed by the previous frame a and no later than the starting exposure time of the first line of effective image signals exposed by the current frame B, and the first light supplementing device 21 is turned off no earlier than the ending exposure time of the last line of effective image signals exposed by the current frame B and no later than the starting exposure time of the first line of effective image signals exposed by the next frame a.

Further, as shown in fig. 7, the exposure time of the a exposure and the B exposure is 8ms, the readout time is 10ms, the exposure of the first line effective image signal is started when the a exposure is 22ms, the exposure of the last line effective image signal is ended when the a exposure is 40ms, the exposure of the first line effective image signal is started when the B exposure is 42ms, and the exposure of the last line effective image signal is ended when the B exposure is 60 ms. The first supplementary lighting device 21 turns on the near-infrared supplementary lighting at 41ms, and turns off the near-infrared supplementary lighting at 61ms, wherein the supplementary lighting time is 20 ms. Therefore, the exposure A without the near-infrared supplementary lighting can be obtained, the color information of the environment can be truly reflected, and the exposure B with the near-infrared supplementary lighting has better brightness, definition and signal-to-noise ratio.

Further, the exposure time of the exposure of a can be properly prolonged, better color information and signal to noise ratio can be obtained, the first light supplementing device 21 is turned on at the exposure starting time of the effective image signal of the first line exposed by the exposure of B and turned off at the exposure ending time of the effective image signal of the last line, the near-infrared light supplementing time can be reduced, and the power can be reduced. For example, as shown in fig. 8, the exposure time of the a exposure is 10ms, the exposure time of the B exposure is 8ms, and the readout times are 10ms, the exposure of the first line effective image signal is started when the a exposure is 20ms, the exposure of the last line effective image signal is ended when the a exposure is 40ms, the exposure of the first line effective image signal is started when the B exposure is 42ms, and the exposure of the last line effective image signal is ended when the B exposure is 60 ms. The first light supplement device 21 may turn on the near-infrared light supplement in 42ms, turn off the near-infrared light supplement in 60ms, and supplement the light for 18 ms.

The above-mentioned staggered order of AB exposures is only one implementation of the present disclosure, and other orders may be adopted, and the exposure conditions of the AB exposures may be different, and the exposure conditions include, but are not limited to, exposure time, digital gain, analog gain, and the like (described later). In addition, the linkage control relationship between the exposure and the light supplement unit 2 is only one implementation manner of the present disclosure, and other linkage control manners satisfying the present disclosure may be adopted.

As a possible implementation manner, as shown in fig. 9, the image capturing component 1 may further include a lens 13, and the lens 13 may be composed of a plurality of lenses and is used for focusing light and helping an object to be imaged on the image sensor 12. In (a) of fig. 9, the filter member 11 may be located between the lens 13 and the image sensor 12, and the image sensor 12 may be located on a light-emitting side of the filter member 11. As an example, the filter member 11 may be a filter film, and when the filter member 11 is located between the lens 13 and the image sensor 12, the filter member 11 may be attached to a surface of the light exit side of the lens 13.

Alternatively, as shown in fig. 9, the image capturing component 1 may further include a lens 13, and the lens 13 may be composed of a plurality of lenses for focusing light and helping the object to be imaged on the image sensor 12. In fig. 9 (b), the lens 13 may be located between the filter member 11 and the image sensor 12, and the image sensor 12 may be located on the light outgoing side of the lens 13. As an example, the filter member 11 may be a filter film, and when the lens 13 is located between the filter member 11 and the image sensor 12, the filter member 11 may be attached to a surface of the lens 13 on the light incident side.

Taking the structure of the image capturing device in fig. 9 (a) as an example, the filter component 11 may be located between the lens 13 and the image sensor 12, and the image sensor 12 is located on the light emitting side of the filter component 11 as an example, the process of capturing the first image signal and the second image signal by the image capturing device is as follows: when the image sensor 12 performs the first exposure, the first light supplement device 21 does not have near-infrared supplement light, at this time, after ambient light in a shooting scene passes through the lens 13 and the filtering component 11, the image sensor 12 generates a first image signal through the first exposure, when the image sensor 12 performs the second exposure, the first light supplement device 21 has near-infrared supplement light, at this time, the ambient light in the shooting scene and near-infrared light reflected by an object in the scene when the first light supplement device 21 performs the near-infrared supplement light pass through the lens 13 and the filtering component 11, the image sensor 12 generates a second image signal through the second exposure, M first exposures and N second exposures can be provided within a unit time period of image acquisition, various combinations of sequencing can be provided between the first exposure and the second exposure, and in a unit time period of image acquisition, values of M and N and a magnitude relationship of M and N can be set according to actual requirements, for example, M and N may or may not be equal in value.

First light filling device 21 is the device that can send near-infrared light, for example, near-infrared light filling lamp etc. and first light filling device 21 can carry out near-infrared light filling with the stroboscopic mode, also can carry out near-infrared light filling with other modes of similar stroboscopic, and this embodiment of this disclosure does not limit this. In some examples, when the first light supplement device 21 performs near-infrared light supplement in a stroboscopic manner, the first light supplement device 21 may be controlled in a manual manner to perform near-infrared light supplement in the stroboscopic manner, or the first light supplement device 21 may be controlled in the stroboscopic manner to perform near-infrared light supplement in a software program or a specific device, which is not limited in this disclosure.

The time period of the first light supplement device 21 for performing near-infrared light supplement may coincide with the exposure time period of the second exposure, or may be greater than the exposure time period of the second exposure or smaller than the exposure time period of the second exposure, as long as the near-infrared light supplement is performed in the whole exposure time period or part of the exposure time period of the second exposure, and the near-infrared light supplement is not performed in the exposure time period of the first exposure.

As an example, the first fill-in device 21 performs near-infrared fill-in at the time of the second exposure, and for the rolling shutter exposure mode, the exposure time period of the second exposure may be a time period between the start exposure time of the first line effective image signal and the end exposure time of the last line effective image signal of the second image signal, but is not limited thereto. For example, the exposure time period of the second exposure may also be an exposure time period corresponding to a target image signal in the second image signal, the target image signal is a plurality of lines of effective image signals corresponding to a target object or a target area in the second image signal, and a time period between the start exposure time and the end exposure time of the plurality of lines of effective image signals may be regarded as the exposure time period of the second exposure.

It should be noted that, when the first light supplement device 21 performs near-infrared light supplement on an external scene, near-infrared light incident on the surface of an object may be reflected by the object, and then enter the first filter device 111. And because the ambient light may include visible light and near-infrared light in a normal condition, and the near-infrared light in the ambient light is reflected by the object when being incident on the surface of the object, so as to enter the first filter device 111. Therefore, the near-infrared light passing through the first optical filter 111 during the near-infrared supplementary lighting may include near-infrared light entering the first optical filter 111 through reflection of an object when the first supplementary lighting device 21 performs the near-infrared supplementary lighting, and the near-infrared light passing through the first optical filter 111 when the first supplementary lighting device 21 does not perform the near-infrared supplementary lighting may include near-infrared light entering the first optical filter 111 through reflection of an object when the first supplementary lighting device 21 does not perform the near-infrared supplementary lighting. That is, the near-infrared light passing through the first optical filter 111 during the near-infrared supplementary lighting includes the near-infrared light emitted by the first supplementary lighting device 21 and reflected by the object and the near-infrared light reflected by the object in the ambient light, and the near-infrared light passing through the first optical filter 111 during the non-near-infrared supplementary lighting includes the near-infrared light reflected by the object in the ambient light.

In addition, since the intensity of the near-infrared light in the ambient light is lower than the intensity of the near-infrared light emitted by the first light supplement device 21, the intensity of the near-infrared light passing through the first optical filter 111 when the first light supplement device 21 performs the near-infrared light supplement is higher than the intensity of the near-infrared light passing through the first optical filter 111 when the first light supplement device 21 does not perform the near-infrared light supplement.

The wavelength range of the near-infrared supplementary lighting performed by the first supplementary lighting device 21 may be a second reference wavelength range, and the second reference wavelength range may be 700 nm to 800 nm, or 900 nm to 1000 nm, and the like, which is not limited in this embodiment. In addition, the wavelength band range of the near infrared light incident to the first filter 111 may be a first reference wavelength band range, and the first reference wavelength band range is 650 nm to 1100 nm.

When the near-infrared light compensation exists, the near-infrared light passing through the first filter device 111 may include near-infrared light reflected by the object and entering the first filter device 111 when the first light compensation device 21 performs near-infrared light compensation, and near-infrared light reflected by the object in the ambient light. The intensity of the near infrared light entering the first filter 111 is stronger at this time. However, in the absence of the near-infrared light compensation, the near-infrared light passing through the first filter 111 includes only near-infrared light reflected by an object in the ambient light and entering the first filter 111. Since the near infrared light supplemented by the first light supplementing device 21 is not present, the intensity of the near infrared light passing through the first optical filter 111 is weak at this time. Therefore, the intensity of near-infrared light included in the second image signal generated and output in accordance with the second exposure is higher than the intensity of near-infrared light included in the first image signal generated and output in accordance with the first exposure.

The central wavelength and/or the band range of the near-infrared supplementary lighting performed by the first supplementary lighting device 21 can be selected in various ways, in order to better match the first supplementary lighting device 21 with the first filtering device 111, the central wavelength of the near-infrared supplementary lighting performed by the first supplementary lighting device 21 can be designed, and the characteristics of the first filtering device 111 can be selected, so that when the central wavelength of the near-infrared supplementary lighting performed by the first supplementary lighting device 21 is set at the characteristic wavelength or falls within the set characteristic wavelength range, the central wavelength and/or the band width of the near-infrared light passing through the first filtering device 111 can reach the constraint condition. The constraint conditions are mainly used for constraining the center wavelength of the near infrared light passing through the first filter device 111 to be as accurate as possible, and the band width of the near infrared light passing through the first filter device 111 to be as narrow as possible, so as to avoid the introduction of wavelength interference caused by too wide band width of the near infrared light.

The central wavelength of the near-infrared light supplement by the first light supplement device 21 may be an average value in a wavelength range with the largest energy in the spectrum of the near-infrared light emitted by the first light supplement device 21, or may be a wavelength at a middle position in a wavelength range with energy exceeding a certain threshold in the spectrum of the near-infrared light emitted by the first light supplement device 21.

The set characteristic wavelength or the set characteristic wavelength range may be preset. As an example, the center wavelength of the near-infrared supplementary lighting performed by the first supplementary lighting device 21 may be any wavelength within a wavelength range of 750 ± 10 nanometers; or, the center wavelength of the near-infrared supplementary lighting performed by the first supplementary lighting device 21 is any wavelength within the wavelength range of 780 ± 10 nanometers; alternatively, the first fill-in light device 21 fills in near-infrared light with any wavelength within a wavelength range of 940 ± 10 nm. That is, the set characteristic wavelength range may be a wavelength range of 750 ± 10 nanometers, or a wavelength range of 780 ± 10 nanometers, or a wavelength range of 940 ± 10 nanometers. Illustratively, the central wavelength of the near-infrared supplementary lighting performed by the first supplementary lighting device 21 is 940 nm, the relationship between the wavelength and the relative intensity of the near-infrared supplementary lighting performed by the first supplementary lighting device 21, and the wavelength band range of the near-infrared supplementary lighting performed by the first supplementary lighting device 21 is 900 nm to 1000 nm, where at 940 nm, the relative intensity of the near-infrared light is the highest.

Since most of the near-infrared light passing through the first light supplement device 111 is near-infrared light reflected by the object and entering the first light filter 111 when the first light supplement device 21 performs near-infrared light supplement when the near-infrared light supplement exists, in some embodiments, the constraint condition may include: the difference between the central wavelength of the near-infrared light passing through the first filtering device 111 and the central wavelength of the near-infrared light supplemented by the first light supplementing device 21 is within a wavelength fluctuation range, which may be 0 to 20 nanometers, as an example.

The central wavelength of the near-infrared supplementary light passing through the first filter device 111 may be a wavelength at a peak position in a near-infrared band range in a near-infrared light transmittance curve of the first filter device 111, or may be a wavelength at a middle position in the near-infrared band range where a transmittance exceeds a certain threshold in the near-infrared light transmittance curve of the first filter device 111.

In order to avoid introducing wavelength interference due to too wide a band width of the near infrared light passing through the first filter 111, in some embodiments, the above constraint may include: the first band width may be less than the second band width. The first wavelength band width refers to a wavelength band width of the near infrared light passing through the first filter 111, and the second wavelength band width refers to a wavelength band width of the near infrared light blocked by the first filter 111. It should be understood that the band width refers to the width of the wavelength range in which the wavelength of the light is located. For example, the wavelength of the near infrared light passing through the first filter 111 is in the wavelength range of 700 nm to 800 nm, and then the first wavelength band width is 800 nm minus 700 nm, i.e., 100 nm. In other words, the band width of the near infrared light passing through the first filter 111 is smaller than the band width of the near infrared light blocked by the first filter 111.

For example, referring to fig. 10, fig. 10 is a schematic diagram of a relationship between a wavelength of light that can pass through the first filter 111 and a transmittance. The band of the near-infrared light incident to the first filter 111 is 650 nm to 1100 nm, and the first filter 111 can pass visible light with a wavelength of 380 nm to 650 nm, pass near-infrared light with a wavelength of 900 nm to 1100 nm, and block near-infrared light with a wavelength of 650 nm to 900 nm. That is, the first band width is 1000 nanometers minus 900 nanometers, i.e., 100 nanometers. The second band has a width of 900 nm minus 650 nm plus 1100 nm minus 1000 nm, i.e., 350 nm. 100 nm is smaller than 350 nm, that is, the band width of the near infrared light passing through the first filter 111 is smaller than the band width of the near infrared light blocked by the first filter 111. The above relation is merely an example, and the wavelength band range of the near-red light band that can pass through the filter member may be different for different filter members, and the wavelength band range of the near-infrared light that is blocked by the filter member may also be different.

In order to avoid introducing wavelength interference due to too wide band width of the near infrared light passing through the first filter 111 during the non-near infrared supplementary lighting period, in some embodiments, the constraint conditions may include: the half bandwidth of the near infrared light passing through the first filter 111 is less than or equal to 50 nm. The half bandwidth refers to the band width of near infrared light with a passing rate of more than 50%.

In order to avoid introducing wavelength interference due to too wide a band width of the near infrared light passing through the first filter 111, in some embodiments, the above constraint may include: the third band width may be less than the reference band width. The third wavelength band width is a wavelength band width of the near infrared light having a transmittance greater than a set ratio, and as an example, the reference wavelength band width may be any one of wavelength band widths in a wavelength band range of 50 nm to 100 nm. The set proportion may be any proportion of 30% to 50%, and of course, the set proportion may be set to other proportions according to the use requirement, which is not limited in the embodiment of the present application. In other words, the band width of the near infrared light having the passing rate larger than the set ratio may be smaller than the reference band width.

For example, referring to fig. 10, the band of near infrared light incident to the first filter 111 is 650 nm to 1100 nm, the set ratio is 30%, and the reference band width is 100 nm. As can be seen from fig. 10, in the wavelength band of the near-infrared light of 650 nm to 1100 nm, the wavelength band width of the near-infrared light with the transmittance of more than 30% is significantly less than 100 nm.

As a possible implementation manner, since human eyes easily mix the color of the supplementary lighting component 2 for near infrared light with the color of the red light in the traffic light, as shown in fig. 11, the supplementary lighting component 2 may further include a second supplementary lighting device 22, and the second supplementary lighting device 22 is used for supplementary lighting for visible light. Thus, if the second light supplement device 22 provides the supplementary light of the visible light at least in the partial exposure time of the second exposure, that is, there are the near-infrared supplementary light and the supplementary light of the visible light at least in the partial exposure time of the second exposure, and the mixed color of the two lights can be distinguished from the color of the red light in the traffic light, so that the confusion of the color of the near-infrared supplementary light of the supplementary light component 2 and the color of the red light in the traffic light by human eyes is avoided. In addition, if the second light supplement device 22 provides supplementary lighting for the visible light in the exposure time period of the first exposure, since the intensity of the visible light in the exposure time period of the first exposure is not particularly high, the brightness of the visible light in the first image signal can be further improved when the supplementary lighting for the visible light is performed in the exposure time period of the first exposure, and the quality of image acquisition is further ensured.

In some embodiments, the second fill-in device 22 may be used for filling in visible light in a normally bright manner; alternatively, the second light supplement device 22 may be configured to supplement the visible light in a stroboscopic manner, where the supplementary visible light exists at least in a partial exposure time period of the second exposure, and the supplementary visible light does not exist in the entire exposure time period of the first exposure; alternatively, the second light supplement device 22 may be configured to supplement the visible light in a stroboscopic manner, where the supplementary visible light does not exist at least in the whole exposure time period of the second exposure, and the supplementary visible light exists in a part of the exposure time period of the first exposure. When the second light supplement device 22 performs visible light supplement in a normally bright manner, not only can the color of the first light supplement device 21 for near-infrared light supplement be prevented from being confused with the color of the red light in the traffic light by human eyes, but also the brightness of the visible light in the first image signal can be improved, and the quality of image acquisition is further ensured. When the second light supplement device 22 performs visible light supplement in a stroboscopic manner, the color of the near-infrared light supplement performed by the first light supplement device 21 by human eyes can be prevented from being confused with the color of the red light in the traffic light, or the brightness of the visible light in the first image signal can be improved, so that the quality of image acquisition is ensured, and the light supplement times of the second light supplement device 22 can be reduced, thereby prolonging the service life of the second light supplement device 22.

As a possible implementation manner, as shown in fig. 12, the filtering component 11 further includes a second filtering device 112 and a switching component 113 (not shown in the figure), the first filtering device 111 and the second filtering device 112 are both connected to the switching component 113, and the switching component 113 is used for switching the second filtering device 112 to the light-incident side of the image sensor 12; after the second filtering means 112 is switched to the light incident side of the image sensor 12, the second filtering means 112 passes light in the visible light band and blocks light in the near infrared light band, and the image sensor 12 generates and outputs a third image signal by exposure.

Thus, when the intensity of visible light in ambient light is weak, for example, at night, the first fill-in device 21 can be used for flash fill-in, the first filtering device 111 can allow part of near-infrared light and visible light to pass through, so that the image sensor 12 can generate and output a second image signal containing near-infrared brightness information and a first image signal containing visible light brightness information, and since the first image signal and the second image signal are both acquired by the same image sensor 12, the viewpoint of the first image signal is the same as that of the second image signal, and thus, complete information of an external scene can be acquired through the first image signal and the second image signal. When the intensity of the visible light is strong, for example, during the day, the proportion of the near-infrared light during the day is relatively high, the color reproduction degree of the collected image is not good, the near-infrared light can be prevented from passing through the second filtering device 112, and the image sensor 12 can generate and output a third image signal containing the information of the intensity of the visible light.

As one possible implementation, the first image signal is generated and output for the first exposure, the second image signal may be generated and output for the second exposure, and after the second image signal and the first image signal are generated and output, the second image signal and the first image signal may be processed. In some cases, the second image signal and the first image signal may be used differently, so in some embodiments, at least one exposure parameter of the second exposure and the first exposure may be different. As an example, the at least one exposure parameter may include, but is not limited to, one or more of exposure time, analog gain, digital gain, aperture size. Wherein the exposure gain comprises an analog gain and/or a digital gain.

In some embodiments, it is understood that, when the near-infrared compensation light exists, the intensity of the near-infrared light sensed by the image sensor 12 is stronger, and the brightness of the near-infrared light included in the generated and outputted second image signal is higher. But the higher brightness near infrared light is not favorable for the acquisition of external scene information. Also, in some embodiments, the greater the exposure gain, the higher the brightness of the image signal output by the image sensor 12, and the smaller the exposure gain, the lower the brightness of the image signal output by the image sensor 12, and therefore, in order to ensure that the brightness of the near-infrared light contained in the second image signal is within a suitable range, in the case where at least one exposure parameter of the second exposure and the first exposure is different, as an example, the exposure gain of the second exposure may be smaller than the exposure gain of the first exposure. In this way, when the fill-in unit 2 performs near-infrared fill-in, the brightness of near-infrared light included in the second image signal generated and output by the image sensor 12 is not too high due to the near-infrared fill-in unit 2 performing near-infrared fill-in.

In other embodiments, the longer the exposure time, the higher the brightness included in the image signal obtained by the image sensor 12, and the longer the motion smear of the moving object in the external scene in the image signal; the shorter the exposure time, the lower the brightness included in the image signal obtained by the image sensor 12, and the shorter the motion smear of the moving object in the external scene in the image signal. Therefore, in order to ensure that the brightness of the near-infrared light contained in the second image signal is within a proper range, and the motion tail of the moving object in the external scene in the second image signal is short. In the case where at least one exposure parameter of the second exposure and the first exposure is different, as an example, the exposure time of the second exposure may be smaller than the exposure time of the first exposure. In this way, when the fill-in unit 2 performs near-infrared fill-in, the brightness of near-infrared light included in the second image signal generated and output by the image sensor 12 is not too high due to the near-infrared fill-in unit 2 performing near-infrared fill-in. And the shorter exposure time makes the motion smear of the moving object in the external scene appearing in the second image signal shorter, thereby facilitating the identification of the moving object. Illustratively, the exposure time for the second exposure is 40 milliseconds, the exposure time for the first exposure is 60 milliseconds, and so on.

It is noted that, in some embodiments, when the exposure gain of the second exposure is less than the exposure gain of the first exposure, the exposure time of the second exposure may be not only less than the exposure time of the first exposure, but also equal to the exposure time of the first exposure. Similarly, when the exposure time of the second exposure is less than the exposure time of the first exposure, the exposure gain of the second exposure may be less than the exposure gain of the first exposure, or may be equal to the exposure gain of the first exposure.

In other embodiments, the second image signal and the first image signal may be used for the same purpose, for example, when both the second image signal and the first image signal are used for intelligent analysis, at least one exposure parameter of the second exposure and the first exposure may be the same in order to enable the same definition of the human face or the target under intelligent analysis when the human face or the target moves. As an example, the exposure time of the second exposure may be equal to the exposure time of the first exposure, and if the exposure time of the second exposure is different from the exposure time of the first exposure, a motion smear may exist in one path of image signals with a longer exposure time, resulting in different definitions of the two paths of image signals. Likewise, as another example, the exposure gain of the second exposure may be equal to the exposure gain of the first exposure.

It is noted that in some embodiments, when the exposure time of the second exposure is equal to the exposure time of the first exposure, the exposure gain of the second exposure may be smaller than the exposure gain of the first exposure, and may also be equal to the exposure gain of the first exposure. Similarly, when the exposure gain of the second exposure is equal to the exposure gain of the first exposure, the exposure time of the second exposure may be less than or equal to the exposure time of the first exposure.

As a possible implementation, the image capturing device may be a video camera, a snapshot machine, a face recognition camera, a code reading camera, a vehicle-mounted camera, a panoramic detail camera, or the like.

As a possible implementation manner, in the embodiment of the present disclosure, a description of channels of the image sensor 12 is provided:

the image sensor 12 includes a plurality of photosensitive channels including at least one of an R photosensitive channel, a G photosensitive channel, a B photosensitive channel, and a W photosensitive channel, which generate and output a first image signal and a second image signal through multiple rolling shutter exposures. The system comprises a light source, a light sensing channel, an R light sensing channel, a G light sensing channel, a B light sensing channel and a W light sensing channel, wherein the R light sensing channel is used for sensing light of a red light wave band and a near infrared wave band, the G light sensing channel is used for sensing light of a green light wave band and a near infrared wave band, the B light sensing channel is used for sensing light of a blue light wave band and a near infrared wave band, and the W light sensing channel is used for sensing light of a full wave band. Therefore, the R photosensitive channel, the G photosensitive channel, the B photosensitive channel and the W photosensitive channel can sense light in a near infrared waveband, so long as at least one of the photosensitive channels is available.

As a possible implementation, the image sensor 12 is any one of a red, green, blue, white RGBW sensor, a red, white, blue RCCB sensor, a red, green, blue RGB sensor, or a red, yellow, blue RYYB sensor; wherein, R represents an R photosensitive channel, G represents a G photosensitive channel, B represents a B photosensitive channel, and W represents a W photosensitive channel. The image sensor 12 may be an RGBW sensor as shown in fig. 13, or the image sensor 12 may be an RCCB sensor as shown in fig. 14, or the image sensor 12 may be an RGB sensor as shown in fig. 15, or the image sensor 16 may be an RYYB sensor as shown in fig. 16.

Each photosensitive channel in the array of channels may be used to sense light of one color when performing the first exposure or performing the second exposure. The channel array of the RGBW sensor comprises red, green, blue and white four-color photosensitive channels, the R photosensitive channel has higher induction quantum efficiency to red light in a red light wave band, the G photosensitive channel has higher induction quantum efficiency to green light in a green light wave band, the B photosensitive channel has higher induction quantum efficiency to blue light in a blue light wave band, and the W photosensitive channel has higher induction quantum efficiency to white light in a full wave band.

In other embodiments, some of the photosensitive channels may also sense only light in the near infrared band and not in the visible band. As an example, the plurality of photosensitive channels may include at least two of an R photosensitive channel, a G photosensitive channel, a B photosensitive channel, and an IR photosensitive channel. The R light sensing channel is used for sensing light of a red light wave band and a near infrared wave band, the G light sensing channel is used for sensing light of a green light wave band and a near infrared wave band, the B light sensing channel is used for sensing light of a blue light wave band and a near infrared wave band, and the IR light sensing channel is used for sensing light of a near infrared wave band.

Illustratively, the image sensor 12 may be an RGBIR sensor, wherein each IR sensing channel in the RGBIR sensor may sense light in the near infrared band, but not in the visible band.

When the image sensor 12 is an RGB sensor, compared with other image sensors, such as an rgbiir sensor, the RGB information collected by the RGB sensor is more complete, and a part of the photosensitive channels of the rgbiir sensor cannot collect visible light, so that the color details of the image collected by the RGB sensor are more accurate.

It is noted that the image sensor 01 may include a plurality of photosensitive channels corresponding to a plurality of sensing curves. Illustratively, referring to fig. 17, an R curve in fig. 17 represents a sensing curve of the image sensor 12 for light in a red wavelength band, a G curve represents a sensing curve of the image sensor 01 for light in a green wavelength band, a B curve represents a sensing curve of the image sensor 12 for light in a blue wavelength band, a W (or C) curve represents a sensing curve of the image sensor 12 for sensing light in a full wavelength band, and an NIR (Near infrared) curve represents a sensing curve of the image sensor 12 for sensing light in a Near infrared wavelength band.

As a possible implementation manner, the multiple rolling shutter exposures include multiple exposure periods, each exposure period includes at least one first exposure and at least one second exposure, that is, the first exposure and the second exposure may be performed alternately, or the first exposure and the second exposure may be performed consecutively, or the second exposure and the first exposure may be performed consecutively. For example, the exposure period is 1 second, the image sensor 12 performs multiple exposures in each exposure period to generate at least one frame of the first image signal and at least one frame of the second image signal, and the first image signal and the second image signal generated in one exposure period are referred to as a set of image signals, so that 25 sets of image signals are generated in 25 exposure periods.

When each exposure period comprises a first exposure and a second exposure, the arrangement sequence of the first exposure and the second exposure in the multiple rolling shutter exposures at least comprises the following types:

in a first possible implementation manner, each exposure period includes a first exposure and a second exposure, and the first exposure and the second exposure are arranged in sequence. For example, as shown in fig. 18, in the multiple rolling shutter exposures, odd-numbered exposures such as the 1 st exposure, the 3 rd exposure, and the 5 th exposure are all the first exposures, and even-numbered exposures such as the 2 nd exposure, the 4 th exposure, and the 6 th exposure are all the second exposures.

In a second possible implementation manner, each exposure period includes a first exposure and a second exposure, and the arrangement order is the second exposure and the first exposure. For example, as shown in fig. 19, in the multiple rolling shutter exposure, odd-numbered exposures such as the 1 st exposure, the 3 rd exposure, and the 5 th exposure are all the second exposures, and even-numbered exposures such as the 2 nd exposure, the 4 th exposure, and the 6 th exposure are all the first exposures.

As shown in fig. 20, each exposure period includes two first exposures and one second exposure, the arrangement order is the first exposure, and the second exposure, and the arrangement order of the first exposure and the second exposure in the multiple rolling shutter exposures may be the first exposure, the second exposure, the first exposure, and the second exposure.

In addition, each exposure period includes two second exposures and one first exposure, the arrangement sequence is the second exposure, the second exposure and the first exposure, and the arrangement sequence of the first exposure and the second exposure in the multiple rolling shutter exposure can be the second exposure, the first exposure, the second exposure, the first exposure and the like.

It should be noted that, the foregoing provides only a few possible implementations of the first exposure and the second exposure, and in practical applications, the implementations are not limited to the foregoing possible implementations, and the embodiments of the present disclosure do not limit this.

As a possible implementation, the exposure time may be adjusted based on the ambient light, and the corresponding processing may be as follows:

when the brightness of the ambient light is smaller than a first numerical value, if the exposure time in the exposure parameter adopted during the first exposure is not a second numerical value, controlling the exposure time in the exposure parameter adopted during the first exposure to be updated to the second numerical value, and if the exposure time in the exposure parameter adopted during the second exposure is not a third numerical value, controlling the exposure time in the exposure parameter adopted during the second exposure to be updated to the third numerical value; when the brightness of the ambient light is greater than or equal to the first value, if the exposure time in the exposure parameter adopted during the first exposure is not the fourth value, the exposure time in the exposure parameter adopted during the first exposure is controlled to be updated to the fourth value, and if the exposure time in the exposure parameter adopted during the second exposure is not the fifth value, the exposure time in the exposure parameter adopted during the second exposure is controlled to be updated to the fifth value, wherein the second value is greater than the fourth value, and the third value is smaller than the fifth value.

The first numerical value, the second numerical value, the third numerical value, the fourth numerical value and the fifth numerical value can be set in advance and stored in the image acquisition device. The second value is greater than the fourth value and the third value is less than the fifth value.

In an embodiment, the image sensor 12 may determine the brightness of the current ambient light during the exposure, and may determine the brightness of the ambient light and the magnitude of the first value by detecting with a light sensor. When the brightness of the ambient light is smaller than the first value, if the exposure time adopted when the first exposure is performed is not the second value, the exposure time adopted when the first exposure is performed is controlled to be updated to the second value, the exposure time used later is the second value, and if the exposure time adopted when the second exposure is performed is not the third value, the exposure time adopted when the second exposure is performed is controlled to be updated to the third value. For example, the exposure time of the first exposure is generally 10ms, the exposure time of the second exposure is generally 15 ms, when the brightness of the ambient light is less than the first value, the exposure time of the first exposure may be updated to 15 ms, and the exposure time of the second exposure may be updated to 10ms, so that when the brightness of the ambient light is lower, the exposure time of the first exposure may be appropriately prolonged, and the exposure time of the second exposure may be appropriately shortened, so that the color, the definition, and the signal-to-noise ratio of the first image signal may all be increased. The above-described updating to the second value and the third value is generally performed so that the total exposure time of the multiple rolling shutter exposures is not changed.

When the brightness of the ambient light is greater than or equal to the first value, if the exposure time taken when performing the first exposure is not the fourth value, the exposure time taken when performing the first exposure is controlled to be updated to the fourth value, and if the exposure time taken when performing the second exposure is not the fifth value, the exposure time taken when performing the second exposure is controlled to be updated to the fifth value.

Therefore, when the light is dark, the exposure time is properly prolonged, the brightness of the acquired first image signal is higher, and when the light is bright, the exposure time is properly shortened, so that overexposure can be avoided.

As a possible implementation manner, before image capturing is performed, when the first exposure is performed and when the second exposure is performed in the image sensor 12 and the light compensation unit 2, respectively, and processing manners of the light compensation unit 2 and the image sensor 12 are configured for each exposure, so that the image sensor 12 and the light compensation unit 2 can respectively perform their own processing during the first exposure. For example, at 10 milliseconds, the fill-in light part 2 is turned off, and the image sensor 12 acquires the first image signal.

As a possible implementation manner, the image acquisition device in the embodiment of the present disclosure may further include a controller, where the controller is electrically connected to the image acquisition component 1 and the light supplement component 2, that is, the controller may be electrically connected to the image acquisition component 1 and the light supplement component 2. The controller may control when the image sensor 12 starts the first exposure and when the second exposure to achieve the first exposure and the second exposure.

As a possible implementation manner, in the embodiment of the present disclosure, when to turn on the light supplement unit 2 and when to turn off the light supplement unit 2 may be controlled manually.

It should be noted that the above-mentioned sensor is only an example, and all the sensors can be used for sensing at least one of visible light of red, green and blue and for sensing near infrared light, and can be applied to the embodiments of the present disclosure.

It should be noted that, the exposure time mentioned in the implementation of the present disclosure refers to the exposure time of each line in each exposure, that is, when the exposure time of the second exposure is 10 milliseconds, the exposure time of each line of effective image signals of the second image signals generated by the second exposure is 10 milliseconds.

It should be further noted that the stroboscopic light supplement lamp is turned on during the second exposure for near-infrared light supplement, and turned off during the first exposure to form a flash.

It should be noted that the above-mentioned image capturing apparatus may be applied to any scene for determining the first image signal and the second image signal, and the embodiment of the present disclosure is not limited thereto.

In an embodiment of the present disclosure, an image capturing method is further provided, which is applied to the image capturing device, where the image capturing device includes: as shown in fig. 21, the implementation procedure of the method may be as follows:

in step 2101, a first light supplement device is used to perform near-infrared light supplement.

The near-infrared supplementary lighting is not carried out during the first exposure, the near-infrared supplementary lighting is carried out during the second exposure, the starting time of the near-infrared supplementary lighting during the second exposure is determined at least according to the first time, and the ending time of the near-infrared supplementary lighting during the second exposure is determined at least according to the second time.

At step 2102, only visible light and a portion of near infrared light is passed through a first filter device.

Step 2103, performing multiple rolling shutter exposure sensing through the image sensor and outputting a first image signal and a second image signal. The first image signal is an image signal generated according to a first exposure, the second image signal is an image signal generated according to a second exposure, the first image signal comprises a plurality of lines of effective image signals, the second image signal comprises a plurality of lines of effective image signals, when the second exposure is a current exposure, the first time is the starting exposure time of the first line of effective image signals of the second image signal generated by the current exposure, and the second time is the ending exposure time of the last line of effective image signals of the second image signal generated by the current exposure.

As a possible implementation manner, the starting time of performing the near-infrared light supplement during the second exposure is not later than the first time, and the ending time of performing the near-infrared light supplement during the second exposure is not earlier than the second time.

As a possible implementation manner, the starting time of performing the near-infrared supplementary lighting during the second exposure is determined according to the first time and a third time, where the third time is the ending exposure time of the last line of effective image signals of the first image signal generated by the last first exposure before the current exposure, and the first time is no earlier than the third time.

As a possible implementation manner, the starting time of performing the near-infrared light supplement during the second exposure is not earlier than the third time and not later than the first time.

As a possible implementation manner, the ending time of the near-infrared supplementary lighting during the second exposure is not earlier than the second time and not later than the fourth time.

As a possible implementation manner, the duration of the near-infrared supplementary lighting performed by the first supplementary lighting device is not less than the sum of the exposure duration of any line of effective image signals currently exposed and the readout duration of effective image signals of the second image signals generated by the current exposure.

As a possible implementation manner, the multiple rolling shutter exposure includes multiple exposure periods, and each exposure period includes at least one first exposure and at least one second exposure.

As a possible implementation, the image capturing component further includes a lens;

As a possible implementation manner, the image sensor includes a plurality of photosensitive channels, the plurality of photosensitive channels include at least one of an R photosensitive channel, a G photosensitive channel, a B photosensitive channel, and a W photosensitive channel, and the plurality of photosensitive channels generate and output a first image signal and a second image signal through multiple rolling shutter exposures;

As a possible implementation, the first exposure is different from the second exposure in at least one exposure parameter, the at least one exposure parameter being one or more of exposure time, exposure gain, aperture size, the exposure gain comprising an analog gain, and/or a digital gain.

As a possible implementation, at least one exposure parameter of the first exposure and the second exposure is the same, the at least one exposure parameter comprises one or more of exposure time, exposure gain, aperture size, the exposure gain comprises analog gain, and/or digital gain.

As a possible implementation manner, the filtering component further includes a second filtering device and a switching component, and both the first filtering device and the second filtering device are connected with the switching component;

a switching member for switching the second filter device to a light incident side of the image sensor;

after the second filter means is switched to the light incident side of the image sensor, the second filter means passes light in the visible light band and blocks light in the near infrared light band, and the image sensor generates and outputs a third image signal by exposure.

the second light supplement device is used for supplementing visible light in a stroboscopic mode, wherein the visible light supplement does not exist at least in the whole exposure time period of the second exposure, and the visible light supplement exists in part of the exposure time period of the first exposure.

As a possible implementation manner, when the central wavelength of the near-infrared light supplement by the first light supplement device is the set characteristic wavelength or falls within the set characteristic wavelength range, the central wavelength and/or the band width of the near-infrared light passing through the first light filter device reach the constraint condition.

As a possible implementation manner, the first light supplement device performs near-infrared light supplement on any wavelength within a wavelength range of 750 ± 10 nanometers; or

The center wavelength of the near-infrared supplementary lighting performed by the first supplementary lighting device is any wavelength within a wavelength range of 940 +/-10 nanometers.

As a possible implementation, the constraint conditions include: the difference value between the central wavelength of the near-infrared light passing through the first light filtering device and the central wavelength of the near-infrared light supplemented by the first light supplementing device is within a wavelength fluctuation range, and the wavelength fluctuation range is 0-20 nanometers; alternatively, the first and second electrodes may be,

the half bandwidth of the near infrared light passing through the first filter is less than or equal to 50 nanometers; alternatively, the first and second electrodes may be,

the first wave band width is smaller than the second wave band width; the first wave band width refers to the wave band width of near infrared light passing through the first filtering device, and the second wave band width refers to the wave band width of the near infrared light blocked by the first filtering device; alternatively, the first and second electrodes may be,

the third wave band width is smaller than the reference wave band width, the third wave band width is the wave band width of the near infrared light with the passing rate larger than the set proportion, and the reference wave band width is any wave band width in the wave band range of 50 nanometers to 150 nanometers.

It should be noted that the processing in the image capturing method is the same as the processing for performing exposure in the image capturing apparatus shown in fig. 3, and the description thereof is omitted here.

It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

The above description is only exemplary of the present disclosure and is not intended to limit the present disclosure, so that any modification, equivalent replacement, or improvement made within the spirit and principle of the present disclosure should be included in the scope of the present disclosure.

Claims

1. An image capturing apparatus, characterized in that the image capturing apparatus comprises:

the image sensor is used for sensing and outputting a first image signal and a second image signal through multiple rolling shutter exposures, wherein the first image signal is an image signal generated according to a first exposure, the second image signal is an image signal generated according to a second exposure, the first exposure and the second exposure are two exposures of the multiple rolling shutter exposures, the first exposure and the second exposure are two consecutive exposures, and the first exposure and the second exposure are alternately performed;

the light supplementing component comprises a first light supplementing device, wherein the first light supplementing device is used for not performing near-infrared light supplementing during the first exposure and performing near-infrared light supplementing during the second exposure, the starting time of performing near-infrared light supplementing during the second exposure is not earlier than the third time and not later than the first time, and the ending time of performing near-infrared light supplementing during the second exposure is not earlier than the second time and not later than the fourth time;

the first image signal comprises a plurality of effective image signals, the second image signal comprises a plurality of effective image signals, when the second exposure is the current exposure, the first time is the exposure starting time of the first effective image signal of the second image signal generated by the current exposure, the second time is the exposure ending time of the last effective image signal of the second image signal generated by the current exposure, the third time is the exposure ending time of the last effective image signal of the first image signal generated by the last exposure before the current exposure, the fourth time is the exposure starting time of the first effective image signal of the first image signal generated by the last exposure after the current exposure, and the second time is not later than the fourth time.

2. The image capturing device according to claim 1, wherein in the current exposure, a duration of the first fill-in device for performing near-infrared fill-in is not less than a sum of an exposure duration of an effective image signal of any line of the current exposure and a readout duration of an effective image signal of a second image signal generated by the current exposure.

3. The image capturing device as claimed in claim 1, wherein the fill-in duration of each row of the effective image signals of the second image signal generated by the current exposure is the same.

4. The image capturing device as claimed in any one of claims 1 to 3, wherein the multiple rolling shutter exposures comprise multiple exposure periods, and each exposure period comprises at least one of the first exposure and at least one of the second exposure.

5. The image capturing device according to any one of claims 1 to 3, wherein the image capturing means further comprises a lens;

6. The image capturing device according to any one of claims 1 to 3, wherein the image sensor includes a plurality of photosensitive channels, the plurality of photosensitive channels include at least one of an R photosensitive channel, a G photosensitive channel, a B photosensitive channel, and a W photosensitive channel, and the plurality of photosensitive channels generate and output the first image signal and the second image signal through the multiple rolling shutter exposures;

7. The image acquisition device according to claim 6, wherein the image sensor is any one of a red, green, blue, white (RGBW) sensor, a red, white, blue (RCCB) sensor, a red, green, blue (RGB) sensor, or a red, yellow, blue (RYYB) sensor;

8. The image capturing device as claimed in any one of claims 1 to 3, wherein the first exposure and the second exposure differ in at least one exposure parameter, the at least one exposure parameter being one or more of exposure time, exposure gain, aperture size, the exposure gain comprising an analog gain, and/or a digital gain.

9. The image capture device of claim 8, wherein an exposure gain of the second exposure is less than an exposure gain of the first exposure.

10. The image capturing device as claimed in any one of claims 1 to 3, wherein at least one exposure parameter of the first exposure and the second exposure is the same, wherein the at least one exposure parameter comprises one or more of exposure time, exposure gain, aperture size, and wherein the exposure gain comprises analog gain, and/or digital gain.

11. The image capturing device of claim 10, wherein the exposure time of the first exposure is equal to the exposure time of the second exposure.

12. The image capturing device as claimed in any one of claims 1 to 3, wherein the filter member further comprises a second filter device and a switching member, and both the first filter device and the second filter device are connected to the switching member;

13. The image capturing device as claimed in any one of claims 1 to 3, wherein the light supplementing unit further comprises a second light supplementing device;

14. The image capturing device as claimed in any one of claims 1 to 3, wherein when the central wavelength of the first fill-in light device for near infrared fill-in light is a set characteristic wavelength or falls within a set characteristic wavelength range, the central wavelength and/or the band width of the near infrared light passing through the first filter device reaches a constraint condition.

15. The image capturing device as claimed in claim 14, wherein the first fill-in light device performs near-infrared fill-in light at any wavelength within a wavelength range of 750 ± 10 nm; or

16. The image capturing device according to claim 14, wherein the constraint condition includes: the difference value between the central wavelength of the near-infrared light passing through the first light filtering device and the central wavelength of the near-infrared light supplemented by the first light supplementing device is within a wavelength fluctuation range, and the wavelength fluctuation range is 0-20 nanometers; alternatively, the first and second electrodes may be,

and the third wave band width is smaller than the reference wave band width, the third wave band width refers to the wave band width of the near infrared light with the passing rate larger than the set proportion, and the reference wave band width is any wave band width in a wave band range of 50-150 nanometers.

17. An image acquisition method is applied to an image acquisition device, and the image acquisition device comprises: image sensor, light filling part and filtering part, the light filling part includes first light filling device, filtering part includes first light filtering device, its characterized in that, the method includes:

performing near-infrared light supplement through the first light supplement device, wherein the near-infrared light supplement is not performed during first exposure, the near-infrared light supplement is performed during second exposure, the starting time of the near-infrared light supplement during the second exposure is not earlier than the third time and not later than the first time, the ending time of the near-infrared light supplement during the second exposure is not earlier than the second time and not later than the fourth time, the first exposure and the second exposure are continuous twice exposure, and the first exposure and the second exposure are alternately performed;