WO2021159768A1

WO2021159768A1 - Image acquisition apparatus and image acquisition method

Info

Publication number: WO2021159768A1
Application number: PCT/CN2020/126076
Authority: WO
Inventors: 陈晓雷; 胡红旗; 吴志江; 赖昌材; 郑士胜; 李瑞华
Original assignee: 华为技术有限公司
Priority date: 2020-02-11
Filing date: 2020-11-03
Publication date: 2021-08-19
Also published as: CN113259546B; CN113259546A

Abstract

The present application provides an image acquisition apparatus and method. The image acquisition apparatus comprises a filter, an image sensor, a control unit, and an image synthesis unit. In embodiments of the present application, the image sensor performs photoelectric imaging on the visible light passing through the filter in a first image sampling interval to obtain a first image comprising chromaticity information, and performs photoelectric imaging on the infrared light (or the infrared light and the visible light) passing through the filter in a second image sampling interval to obtain a second image comprising the chromaticity information, and then the image synthesis unit synthesizes the first image with the second image to obtain a target image of higher quality, so that there is no need to use a beam splitting prism, and only one sensor is needed, which can reduce material costs and the complexity of the apparatus.

Description

Image acquisition device and image acquisition method

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on February 11, 2020, the application number is 202010087243.9, and the application name is "Image Acquisition Device and Image Acquisition Method", the entire content of which is incorporated into this application by reference .

Technical field

This application relates to the field of image applications, and more specifically, to an image acquisition device and an image acquisition method.

Background technique

With the rapid development of image application technology, the imaging quality requirements of image acquisition equipment under low illumination are getting higher and higher. Compared with scenes with normal illuminance, the image quality of low-illuminance environment will be reduced, mainly manifested by the decrease of image brightness and the increase of noise. The biggest factor leading to the deterioration of imaging quality is the decrease of the light input of the imaging sensor and the decrease of the signal-to-noise ratio of its output signal. Therefore, increasing the amount of light entering the image sensor under low illumination is the key to improving the quality of low illumination images. It is a common method to use supplementary light equipment for supplementary light. According to the wavelength of light emitted by the supplementary light equipment, these devices can be divided into two types: visible light supplementary light and infrared supplementary light. Visible light fill light can easily obtain the true color information of the scene, but it will interfere with people. Infrared supplementary light has no obvious interference to people, but it cannot obtain the true color of the object well.

In order to simultaneously obtain visible light imaging and infrared imaging in the scene, the current industry practice is to use a light splitting prism. As shown in Figure 1, the incident light is divided into visible light and infrared light by the optical coating in the dichroic prism after passing through the dichroic prism. On the visible light exit surface, a color sensor is placed to image the visible light to obtain a visible light image. On the exit surface of the infrared light, an infrared light sensor is placed to image the infrared light to obtain an infrared image. Then, the algorithm is used to synthesize the visible light image and the infrared image, and the final color image is output.

However, the above solution requires the use of a spectroscopic prism and two sensors, and the material cost is relatively high, and the two sensors need to be aligned during assembly, and the production complexity is relatively high. In addition, a beam splitter prism is added to the back of the lens, which is not compatible with the existing standard rear focus lens, and causes the light path part to be bulky, which is not conducive to the miniaturization of the device.

Summary of the invention

This application provides an image acquisition device and an image acquisition method, which can respectively acquire a first image containing chromaticity information and a second image containing brightness information at different image sampling intervals, and synthesize the first image and the second image Obtaining the target image, so that no spectroscopic prism is needed, and only one sensor is needed, which can help reduce material costs and device complexity.

In a first aspect, an image acquisition device is provided, which includes a filter, an image sensor, a control unit, and an image synthesis unit.

The filter is used to gate the incident light.

The image sensor is used for photoelectric imaging of the light passing through the filter in the incident light.

The control unit is connected to the filter and the image sensor, and is used to control the filter to pass the visible light in the incident light and block the infrared light in the incident light from passing through ( That is, the infrared light in the incident light cannot pass through), and the infrared light in the incident light passes through the second image sampling interval. as well as

It is used to control the image sensor to photoelectrically image the light that passes through the filter at the first image sampling interval in the incident light, to obtain a first image, and to perform photoelectric imaging of the incident light in the second image. Photoelectric imaging is performed through the light of the filter at the image sampling interval to obtain a second image.

The image synthesis unit is used to synthesize the first image and the second image to generate a first target image.

Therefore, in the embodiment of the present application, the light passing through the filter is photoelectrically imaged at the first image sampling interval to obtain a first image containing chromaticity information, and the light passing through the filter is performed at the second image sampling interval. Photoelectric imaging, acquiring a second image containing brightness information, and then synthesizing the first image and the second image, can acquire a higher-quality target image. Since the image sensor in the embodiment of the present application can obtain the first image and the second image in different time periods, the embodiment of the present application can use one sensor, which reduces the material cost, and does not require registration of the image sensor, reducing complexity It can help reduce the volume of the device.

Optionally, the controller may also be connected to the image synthesis unit for controlling the image synthesis unit to synthesize the first image and the second image.

With reference to the first aspect, in some implementations of the first aspect, the filter includes a first filter area and a second filter area, and the first filter area is used to pass visible light but not infrared light. , The second filter area is used to pass infrared light. Optionally, the second filter area may pass visible light, or may not pass visible light, which is not limited in the embodiment of the present application.

The control unit is specifically configured to use the first filter area to filter the incident light at the first image sampling interval, and use the second filter area to filter the incident light at the second image sampling interval. The incident light is filtered.

Therefore, in the embodiments of the present application, it can be realized that the first filter area covers the photosensitive area of the image sensor at the first image sampling interval, and the second filter area covers the photosensitive area of the image sensor at the second image sampling interval, thereby enabling It is realized that the image sensor performs photoelectric imaging of the visible light passing through the filter at the first image sampling interval, and performs photoelectric imaging of the infrared light (or infrared light and visible light) passing through the filter at the second image sampling interval.

It should be noted that the above-mentioned "using the first filter area to filter the incident light at the first image sampling interval" includes two cases, which excludes and does not exclude the simultaneous use of the second filter area to filter the incident light. The incident light is filtered, that is, only the first filter area is used to filter the incident light, or the first filter area and the second filter area are used at the same time to filter the incident light. Similarly, the above-mentioned "using the second filter area to filter the incident light at the second image sampling interval" also includes two cases, which excludes and does not exclude the simultaneous use of the first filter area to filter the incident light. The incident light is filtered, that is, only the second filter area is used to filter the incident light, or the first filter area and the second filter area are used at the same time to filter the incident light.

With reference to the first aspect, in some implementations of the first aspect, the filter is a circular filter, and the first filter area is a first fan-shaped area of the circular filter, so The second filter area is a second fan-shaped area of the circular filter;

The control unit includes a motor, and the motor is used to control the circular filter to rotate around the center of the circular filter, so that the first fan-shaped area covers all areas at the first image sampling interval. In the photosensitive area of the image sensor, the second fan-shaped area covers the photosensitive area at the second image sampling interval.

Since infrared light interferes greatly with RGB image imaging, but visible light does not interfere much with infrared imaging, the “covering” in “the first sector area covers the photosensitive area of the image sensor at the first image sampling interval” refers to Cover all photosensitive areas. The "covering" in "the second fan-shaped area covers the photosensitive area at the second image sampling interval" means that it can cover all the photosensitive area or part of the photosensitive area.

Therefore, in the embodiments of the present application, the circular filter is rotated around the center of the circle, so that the spectrum that reaches the image sensor located behind the circular filter can change periodically, so that the image sensor can periodically perform color frame exposure and black and white frames. exposure.

With reference to the first aspect, in some implementations of the first aspect, the photosensitive area is a rectangular area, and when the duration of the first image sampling interval is the same as the duration of the second image sampling interval, The circular filter meets the following conditions:

R≥c+b/(2sin((π–ωt)/2));

Where ω represents the angular velocity of the rotation of the circular filter, t represents the duration of the first image sampling interval or the second image sampling interval, R represents the radius of the circular filter, and b represents the total The first side length of the photosensitive area of the image sensor, c represents the second side length of the photosensitive area of the image sensor, and the first side and the second side are perpendicular to each other. Among them, ωt≤π.

It needs to be said that at this time, no imaging will be performed within the angle of 2β, that is, no photoelectric imaging will be performed when the first sector area and the second sector area cover the photosensitive area at the same time. Wherein, β is the central angle of the circular filter facing the first side length.

With reference to the first aspect, in some implementations of the first aspect, the central angle corresponding to the first sector area is greater than or equal to ωt+2arcsin(b/(2(Rc))), and/or the second The central angle corresponding to the sector area is greater than or equal to ωt+2arcsin(b/(2(Rc))).

R≥c+b/(2sin(π–ωt));

Where ω represents the angular velocity of the rotation of the circular filter, t represents the duration of the first image sampling interval or the second image sampling interval, R represents the radius of the circular filter, and b represents the total The first side length of the photosensitive area of the image sensor, c represents the second side length of the photosensitive area of the image sensor, and the first side and the second side are perpendicular to each other.

With reference to the first aspect, in some implementations of the first aspect, the central angle corresponding to the first sector area is greater than or equal to ωt+2arcsin(b/(2(R-c))).

With reference to the first aspect, in some implementation manners of the first aspect, the photosensitive area is a rectangular area, and the circular filter satisfies the following conditions:

R≥c+b/(2sin(β/2));

Wherein, R represents the radius of the circular filter, b represents the first side length of the photosensitive area of the image sensor, c represents the second side length of the photosensitive area of the image sensor, and β is the first The first side and the second side are perpendicular to each other at the central angle of the circular filter opposite to the side length.

With reference to the first aspect, in some implementations of the first aspect, the rotational angular velocity of the circular filter is ω, and the central angle of the first sector region in the circular filter is (2π-γ) , The central angle of the second fan-shaped area is γ, the exposure time of the first image (ie the first image image sampling interval) is t ₁ , the exposure time of the second image (ie the second image image sampling interval) ) Is t ₂ ,

Then ω, γ, t ₁ , t ₂ satisfy the following formula:

2π-γ-β≥ωt ₁ ;

γ+β≥ωt ₂ or γ≥ωt ₂ or γ-β≥ωt ₂ .

Optionally, t ₁ and t ₂ may be equal or different, which is not limited in the embodiment of the present application. When t ₁ and t ₂ are equal, it can be expressed as t ₁ =t ₂ =t.

With reference to the first aspect, in some implementations of the first aspect, the control unit includes a driver and a lever, a first end of the lever is connected to the filter, and the driver is used to control The filter is moved so that the first filter area covers the photosensitive area of the image sensor at the first image sampling interval, and the second filter area covers all the photosensitive areas at the second image sampling interval.述photosensitive area.

Since infrared has great interference on RGB image imaging, but visible light has little interference on infrared imaging, the "covering" in "the first filter area covers the photosensitive area of the image sensor at the first image sampling interval" refers to It covers all the photosensitive areas. The "covering" in "the second filter area covers the photosensitive area at the second image sampling interval" means that it can cover all the photosensitive area or part of the photosensitive area.

Therefore, in the embodiment of the present application, the filter is connected through a lever, and the filter at the end of the lever is driven to move under the drive of the driver, so that the spectrum reaching the image sensor located behind the filter can change periodically. , To realize that the image sensor periodically performs color frame exposure and black and white frame exposure.

With reference to the first aspect, in some implementations of the first aspect, the filter includes an electrically controlled light-absorbing material,

The control unit is specifically configured to apply a voltage to the filter, and control the magnitude of the voltage to make the filter pass visible light but not infrared light during the first image sampling interval, and The second image sampling interval passes infrared light.

Therefore, in the embodiment of the present application, by setting the electronically controlled light-absorbing material in the filter, and by changing the voltage applied to the filter, the filter can periodically change the absorption peak of the light-absorbing material in the filter, thereby The spectrum that reaches the image sensor located behind the filter can be changed periodically, so that the image sensor can periodically perform color frame exposure and black-and-white frame exposure.

With reference to the first aspect, in some implementations of the first aspect, the electrically controlled light-absorbing material includes an organic color-changing material or a liquid crystal material. In this way, the response time of the electronically controlled light-absorbing material can reach the millimeter level, and can quickly respond to the switch between black and white frame exposure and color frame exposure.

In combination with the first aspect, some implementations of the first aspect further include:

The optical unit includes an optical lens for capturing the incident light and causing the incident light to be imaged on the image sensor, wherein the optical filter is arranged before or after the optical lens, or is arranged on the Between the two lenses in the optical lens.

With reference to the first aspect, in some implementation manners of the first aspect, the control unit is specifically configured to:

In the case where it is determined that the gain of the image sensor is greater than the preset value, the filter is controlled to pass visible light and not infrared light at the first image sampling interval, and to pass infrared light at the second image sampling interval And controlling the image sensor to perform photoelectric imaging of the light passing through the filter in the first image sampling interval, and perform photoelectric imaging of the light passing through the filter in the second image sampling interval.

When it is determined that the gain of the image sensor is greater than the preset value, it can be determined that the photographed environment is a low-illuminance environment. Therefore, the embodiment of the present application can control the filter to pass visible light but not infrared light at the first image sampling interval when the subject environment is perceived as a low-illuminance environment, and pass infrared light (or pass infrared light at the second image sampling interval). Light and visible light).

With reference to the first aspect, in some implementation manners of the first aspect, the control unit is further configured to:

When it is determined that the gain of the image sensor is less than or equal to the preset value, the image sensor is controlled to perform photoelectric imaging of the incident light in the third image sampling interval to obtain a second target image, wherein the third image The sampling interval is equal to the sum of the first image sampling interval and the second image sampling interval.

In this way, the frame rate of the output image of the image acquisition device in a low-illuminance environment can be kept unchanged compared to the frame rate of the output image in a normal environment.

In the second aspect, an image acquisition method is provided. The method may be executed by the image acquisition device in the foregoing first aspect or any one of the possible implementation manners of the first aspect.

In this method, the filter is controlled to pass visible light in the incident light but not infrared light at the first image sampling interval, and the filter is controlled to pass the infrared light in the incident light at the second image sampling interval; The sensor performs photoelectric imaging of the light passing through the filter at the first image sampling interval to obtain a first image; the image sensor performs photoelectric imaging on the light passing through the filter at the second image sampling interval Photoelectric imaging is used to obtain a second image; the first image and the second image are synthesized by an image synthesis unit to generate a first target image.

The steps of the method of the second aspect may refer to the operations of the corresponding modules of the device of the first aspect, and will not be repeated here.

In a third aspect, a computer-readable medium is provided for storing a computer program. The computer program includes instructions for executing the first aspect or the second aspect or any possible implementation of the first aspect or the second aspect .

In a fourth aspect, a computer program product containing instructions is also provided. When the computer program product runs on a computer, the computer can execute the first aspect or the second aspect or any possible implementation of the first or second aspect. Instructions in the way.

Description of the drawings

Figure 1 is a schematic diagram of imaging using a beam splitting prism.

Fig. 2 is a schematic diagram of an image acquisition device provided by an embodiment of the present application.

FIG. 3 is another schematic diagram of the image acquisition device provided by the embodiment of the present application.

Fig. 4 is a specific example of a circular filter provided by an embodiment of the present application.

Fig. 5 is an example of exposure of the circular filter provided in the embodiment of the present application.

FIG. 6 is a specific example of the connection between the lever and the filter provided by the embodiment of the present application.

FIG. 7 is an example of the light filter connected with the lever provided in the embodiment of the present application for exposure.

FIG. 8 is another example of the light filter connected with the lever provided in the embodiment of the present application for exposure.

FIG. 9 is another example of the light filter connected with the lever provided in the embodiment of the present application for exposure.

FIG. 10 is another specific example of the filter provided by the embodiment of the present application.

Fig. 11 is a schematic diagram of another image acquisition device provided by an embodiment of the present application.

FIG. 12 is a schematic diagram of synthesizing the first image and the second image in the YUV color space provided by an embodiment of the present application.

FIG. 13 is a schematic flowchart of an image acquisition method provided by an embodiment of the present application.

Detailed ways

The technical solution in this application will be described below in conjunction with the accompanying drawings.

FIG. 2 shows a schematic diagram of an image acquisition device 200 provided by an embodiment of the present application. As an example, the image acquisition device 200 may be a surveillance camera, a mobile phone or other wearable devices with a shooting function or a camera in a smart home device, etc., which is not limited in the embodiment of the present application.

Exemplarily, in the application scenario of surveillance camera, the surveillance camera needs 7×24 hours (referring to 7 days a week, 24 hours a day) to operate. In other words, most surveillance cameras spend about half of their working time in low-light environments. Therefore, it is necessary for the surveillance camera to have a higher imaging quality (such as high brightness and low noise) in a low-illuminance environment, which can help to obtain important information (such as information such as a human face and a car body color). In addition, in the application scenario of taking pictures, when the camera is used in a low-light environment, the camera is also required to be able to obtain images that satisfy the business application.

In some possible embodiments, the image acquisition device 200 may be a device for acquiring images at night, and related units of the device 200 can perform operations such as light gating and corresponding image synthesis at night, but this community implements Examples are not limited to this.

As shown in FIG. 2, the image acquisition device 200 includes a filter 210, an image sensor 220, a control unit 230 and an image synthesis unit 240.

Among them, the filter 210 is used to gate incident light.

The image sensor 220 is used for photoelectric imaging of the light passing through the filter 210 in the incident light.

The control unit 230, connected to the filter 210 and the image sensor 220, is used to control the filter 210 to pass the visible light in the incident light and block the infrared light in the incident light from passing (that is, not passing through all the light) at the first image sampling interval. The infrared light in the incident light), and the infrared light in the incident light that passes through the second image sampling interval. And, for controlling the image sensor 220 to photoelectrically image the light passing through the filter 210 at the first image sampling interval, to obtain a first image, and to perform photoelectric imaging of the light passing through the filter 210 at the second image sampling interval. Perform photoelectric imaging to obtain a second image.

That is to say, the control unit 230 can control the filter 210 to pass different light types at different image sampling intervals, that is, realize the gating function of the spectrum, and then can control the type of light reaching the surface of the image sensor 220.

It should be noted that in the embodiments of the present application, the first image is obtained by imaging the visible light of the real scene, which can reflect the chromaticity information of the real scene, and the second image is mainly obtained by imaging the infrared light of the real scene. , Can reflect the brightness information of the real scene. In some embodiments, the first image may be referred to as a color frame image, and the second image may be referred to as a black and white frame image.

Since infrared light affects the imaging of visible light, the filter 210 cannot pass infrared light in the first image sampling interval. The visible light does not affect the imaging of infrared light, so the filter 210 may not pass the visible light in the incident light or may pass the visible light in the incident light in the second image sampling interval, which is not limited in the embodiment of the present application. When the filter 210 can pass visible light in the incident light at the second image sampling interval, it can help reduce the complexity of the filter.

Among them, the image sensor 220 is used to receive light signals and convert the light signals into electrical signals. As an example, the image sensor 220 may be a charge coupled device (CCD), a complementary metal oxide semiconductor (complementary metal oxide semiconductor, CMOS), etc., which is not limited in the embodiment of the present application.

That is to say, in the embodiment of the present application, the control unit 230 controls the filter 210 and the image sensor 220 to realize that at the first image sampling interval, the filter 210 allows visible light in the incident light to pass through, and does not allow the incident light to pass. The mid-infrared light passes through, and the light passing through the filter 210 is photoelectrically imaged on the image sensor 220 to obtain the first image; at the second image sampling interval, the filter 210 allows the infrared light in the incident light to pass , Or allow infrared light and visible light in the incident light to pass, and the light passing through the filter 210 is photoelectrically imaged on the image sensor 220 at this time to obtain a second image.

It should be noted that since the image sensor 220 performs photoelectric imaging of visible light at the first image sampling interval, photoelectric imaging of infrared light at the second image sampling interval, or photoelectric imaging of visible light and infrared light at the second image sampling interval. Therefore, one sensor may be used in the embodiment of the present application to obtain the visible light and the infrared light in the photographed scene at different time periods, and perform imaging separately. In other words, in the embodiment of the present application, the image sensor 220 may be time-division multiplexed.

Optionally, the duration of the first image sampling interval and the second image sampling interval may be the same or different, which is not limited in the embodiment of the present application.

In other possible implementation manners, two sensors may also be used to perform visible light imaging and infrared light imaging, respectively, which is not limited in the embodiment of the present application.

The image synthesis unit 240 is configured to synthesize the first image collected by the image sensor 220 and the second image collected by the image sensor 220 to obtain a target image. Among them, the chrominance information in the target image mainly comes from the first image, and the brightness information in the target image mainly comes from the second image.

In some optional embodiments, the control unit 230 may also be connected to the image synthesis unit 240 to control the image synthesis unit 240 to synthesize the first image and the second image.

In the embodiment of the present application, the duration of acquiring one frame of target image may be referred to as the target image acquisition period. In the embodiment of the present application, the first image and the second image need to be acquired in a time-sharing period in a target image acquisition period, that is, the first image is acquired at the first image sampling interval in the target image acquisition period, and the target image is acquired The second image is acquired at the second image sampling interval in the cycle. Then, based on the first image and the second image, a target acquisition image of the target image acquisition period can be generated.

As an example, when the output frame rate of the device 200 is 25 frames per second, one target image acquisition period is 0.04 s. At this time, the first image sampling interval and the second image sampling interval may be 0.02 s, respectively. As a specific example, in the first 0.02s of the target image acquisition period of 0.04s, the first image can be acquired, and the second image can be acquired 0.02s later.

It should be noted that, in the embodiments of the present application, the image synthesis unit may be implemented by electronic hardware, or computer software, or a combination of computer software and electronic hardware, which is not limited in the embodiments of the present application. As an example, the image synthesis unit may be implemented by a central processing unit (CPU), a graphics processing unit (GPU) or a specific image processing chip. In some possible implementation manners, the CPU, GPU, or image processing chip used for image synthesis can read codes to realize the synthesis of the first image and the second image, which is not limited in the embodiment of the present application.

In some possible implementations, the filter 210 may include a first filter area and a second filter area. Among them, the first filter area is used to pass visible light but not infrared light, and the second filter area is used to pass infrared light. Optionally, the second filter area may pass visible light, or may not pass visible light, which is not limited in the embodiment of the present application.

As shown in FIG. 3, the filter 210 may include a first filter area 2101 and a second filter area 2102. In addition, the control unit 220 may be specifically configured to use a first filter area to gate the incident light at the first image sampling interval, and use a second filter area to select the incident light at the image sampling interval. Gating.

As an example, the control unit 230 may control the filter 210 to move so that the first filter area in the filter 210 covers the photosensitive area of the image sensor 220 at the first image sampling interval, and the second filter area is at the second image sampling interval. The image sampling interval covers the photosensitive area of the image sensor 220. That is, in the first image sampling interval, the first filter area may be in the optical path before the image sensor, and the visible light in the optical path is gated so that the image sensor receives visible light; in the second image sampling interval, the second filter area In the optical path before the image sensor, the infrared light in the optical path (or infrared light and visible light in the optical path) is gated, so that the image sensor is controlled to receive infrared light (or infrared light and visible light).

Therefore, in the embodiments of the present application, the first filter area in the filter 210 can be implemented to cover the photosensitive area of the image sensor 220 at the first image sampling interval, and the second filter area to cover the image at the second image sampling interval. The photosensitive area of the sensor 220 can further realize that the image sensor performs photoelectric imaging of the visible light passing through the filter at the first image sampling interval, and performs photoelectric imaging of the infrared light (or infrared light and visible light) passing through the filter at the second image sampling interval. Photoelectric imaging.

It should be noted that FIG. 3 is to illustrate the schematic structure of the filter 210, so only some modules or units in the image acquisition device are shown. That is, in addition to the filter 210 and the control unit 230 shown in FIG. 3, the image acquisition device also includes other modules or units (not shown in FIG. 3), such as image sensors and image synthesis Unit, this embodiment of the application does not limit this.

The following describes specific implementations of two filters including the first filter area and the second filter area provided by the embodiments of the present application.

method one

The filter 210 may be designed as a circular filter. The first filter area 2101 is the first fan-shaped area in the circular filter, and the second filter area is the second fan-shaped area in the circular filter. Correspondingly, the control unit 230 may include a motor, and the motor may be used to control the circular filter to rotate around the center of the circular filter, so that the first fan-shaped area covers the image within the first image sampling interval. The photosensitive area of the sensor is covered by a second fan-shaped area in the second image sampling interval. In some descriptions, the circular filter may also be referred to as a color wheel, which is not limited in the embodiment of the present application.

In some embodiments, the motor can control the circular filter to rotate at a uniform speed around the center of the circle, or to rotate at a variable speed, or control the circular filter to rotate clockwise or counterclockwise around the center of the circle, but the embodiments of the present application are not limited to this.

Optionally, when the control unit 230 includes a motor, the control unit 230 may also include a control chip, and the control chip may send a control instruction to the motor to control the operation of the motor, so as to realize the control of the circular filter by the motor. As an example, the control chip can read instructions or codes in the memory to achieve image acquisition.

Among them, the control chip can be different from the CPU, or it can be the CPU.

Please refer to FIG. 4, which shows a specific example of a circular filter. Among them, the rectangular shaded area is an example of the photosensitive area of the image sensor, and its two side lengths are b and c, respectively. The center of the circular filter is point O, and the radius is R. AO and BO are respectively the boundary lines of the two filters included in the circular filter (that is, the first sector area and the second sector area).

Exemplarily, the fan-shaped area OBCA is an example of the first fan-shaped area, which allows visible light to pass but not infrared light. The fan-shaped area OBDA is an example of the second fan-shaped area, which allows infrared light to pass through, or may allow infrared light and visible light to pass through. For ease of understanding, the following descriptions are given by taking the fan-shaped area OBCA as allowing visible light and not allowing infrared light to pass, and the fan-shaped area OBDA allowing infrared light to pass through, or allowing infrared light and visible light to pass through as examples.

In this way, when the entire area of the photosensitive area of the image sensor (that is, the shaded area in FIG. 4) is within the fan-shaped area OBCA, color frame exposure can be performed. When the area or part of the area is within the fan-shaped area OBDA, black-and-white frame exposure can be performed.

Generally, the radius of the circular filter is larger than the length of the photosensitive area of the image sensor. At this time, the photosensitive area of the image sensor can be set on the side away from the center of the circular filter, and the circular filter can completely cover the photosensitive area of the image sensor, such as the structure shown in FIG. 4.

Please continue to refer to Figure 4, when the length of the photosensitive area of the image sensor is c and the width is b, the central angle β of the circular filter facing the side of the sensor with length b (that is, the length of b is close to the center of the circle). The angle formed by the two end points G and F of the side length of O and the center O, namely ∠FOG) satisfies the following formula:

R≥c+b/(2sin(β/2)) (1)

From formula (1), β=2arcsin(b/(2(R-c)));

Among them, R is the radius of the circular filter.

In some embodiments, the empirical value of c can be represented by the following formula:

c≤R/3;

The following describes the process of the image sensor periodically performing color frame exposure and black-and-white frame exposure along with the rotation of the circular filter in conjunction with FIG. 4 and FIG. 5.

As shown in Fig. 4, assuming that the circular filter rotates clockwise around the center of the circle, the intersection of OA and the vertex F of the shadow area can be taken as the start of a target image acquisition cycle. When OA rotates one circle clockwise and intersects with vertex F again, the target image acquisition cycle ends. In the target image acquisition period, the image sensor can perform one black and white frame exposure and one color frame exposure.

Specifically, when OA intersects the vertex F of the shadow area (as shown in Figure 4), the infrared light can pass through the second filter area to reach the image sensor (that is, the photosensitive area of the image sensor). At this time or before, It is necessary to end the color frame exposure in the previous image acquisition cycle.

Optionally, when OA intersects the vertex F of the shadow area, that is, when the fan-shaped area AOBF begins to overlap with the photosensitive area of the image sensor, black-and-white frame exposure can be started. Or, optionally, as the circular filter rotates, it may be when AO intersects the vertex G or the first time period after that, that is, when the fan-shaped area AOBD and the photosensitive area of the image sensor completely overlap or the first time period thereafter , And then start the black and white frame exposure. Alternatively, it is also possible to start the exposure of the black and white frame at any time between the intersection of OA and vertex F to the intersection of OA and vertex G.

It should be noted that in the period from the intersection of OA and vertex F to the intersection of OA and vertex G, since the sector area AOBD does not completely cover the photosensitive area of the image sensor at this time, the strobed infrared light is relatively less than the sector area AOBD. The infrared light strobed when the photosensitive area of the image sensor is not completely covered, therefore, the quality of the black and white frame image during this time period is worse than that when the fan-shaped area AOBD completely covers the photosensitive area of the image sensor.

As the circular filter continues to rotate, BO will successively intersect vertex F and another vertex G in the shadow area. Optionally, when BO and vertex F intersect or a second time period before the intersection, that is, when the fan-shaped area AOBF starts to not completely cover the photosensitive area of the image sensor or the second time period before, the black-and-white frame exposure may end. Or, optionally, as the circular filter rotates, when BO intersects the vertex G (as shown in FIG. 5), that is, when the fan-shaped area AOBF does not overlap with the image sensor at all, the black-and-white frame exposure ends. Alternatively, it is also possible to end the exposure of the black and white frame at any time between the intersection of BO and vertex F to the intersection of BO and vertex G.

Exemplarily, the frame number of the black and white frame obtained at this time may be denoted as (2n-1). As the circular filter continues to rotate, after BO intersects with vertex G (ie, the situation shown in FIG. 5) to before OA intersects with vertex F again, color frame exposure can be performed. For example, the color frame exposure may be started after BO and vertex G intersect or after the third time period after the intersection, and the color frame exposure may be ended at the fourth time period before AO and vertex F intersect or intersect. Exemplarily, the frame number of the color frame obtained at this time may be denoted as (2n). Wherein, n is an integer greater than or equal to 1.

Three specific examples in which the image sensor periodically performs color frame exposure and black-and-white frame exposure are described below. Wherein, the size of the central angle of the fan-shaped area OBDA can be γ, and the size of the central angle of the fan-shaped area OBCA can be (2π-γ). The fan-shaped area in the filter rotates clockwise around the center O at an angular velocity ω at a uniform speed.

The first example: when the AO first intersects the vertex F of the shadow area (ie the photosensitive area of the image sensor), the black and white frame exposure is turned on. When BO and the G point of the shadow area intersect, the black and white frame exposure ends, and the color frame exposure starts. When AO intersects the vertex F of the shadow area again, the color frame exposure ends, and the black and white frame exposure of the next cycle begins. At this time, the black-and-white frame exposure angle is (γ+β), and the color frame exposure angle is (2π-γ-β), which can be denoted as α (that is, α=2π-γ-β).

Meet at this time:

α=ωt ₁ ;

γ+β=ωt ₂ ;

Wherein, the exposure time of the color frame can be t ₁ , and the exposure time of the black and white frame can be t ₂ . Optionally, t ₁ and t ₂ may be equal or different, which is not limited in the embodiment of the present application.

When the exposure time of the color frame is equal to the exposure time of the black and white frame, that is, t ₁ =t ₂ , then α=γ+β=π.

In some possible implementations, the exposure angle of the color frame may also be less than or equal to α, that is, α≥ωt ₁ , and the exposure angle of the black and white frame may also be less than or equal to γ+β, that is, γ+β≥ωt ₂ . When designing a circular filter, in order to ensure a compact structure, it can usually be designed as α=ωt ₁ and γ+β=ωt ₂ .

The second example: when the AO coincides with the central axis of the shadow area, turn on the black and white frame exposure. When BO coincides with the central axis of the shadow area, the black-and-white frame exposure ends. When BO intersects the G point of the shaded area, the color frame exposure starts. When AO intersects the vertex F of the shadow area again, the color frame exposure ends. When the AO coincides with the central axis of the shadow area again, the black and white frame exposure of the next cycle starts. At this time, the black and white frame exposure angle is γ, and the color frame exposure angle is still α.

Here, in the process from the intersection of AO and vertex F to the coincidence of AO and the central axis of the shadow area (corresponding to the (β/2) angle close to OA of the fan-shaped area OBCA), neither black-and-white frame exposure nor color light is performed. exposure. Similarly, in the process from BO coincides with the central axis of the shadow area to the intersection of BO and vertex F (corresponding to the (β/2) angle close to OB of the fan-shaped area OBCA), neither black-and-white frame exposure nor color is performed. Frame exposure.

Meet at this time:

α=ωt ₁ ;

γ=ωt ₂ ;

When the exposure time of the color frame is equal to the exposure time of the black and white frame, that is, t ₁ =t ₂ =t, then α=γ=π-(β/2).

That is, β=2(π-α)=2(π-ωt). At this time, the above formula (1) can be transformed into the following formula:

R≥c+b/(2sin((π–ωt)));

In some possible implementations, the exposure angle of the color frame may also be less than or equal to α, that is, α≥ωt ₁ , and the exposure angle of the black and white frame may also be less than or equal to γ, that is, γ≥ωt ₂ . When designing a circular filter, in order to ensure a compact structure, it can usually be designed as α=ωt ₁ and γ=ωt ₂ .

That is, when t ₁ =t ₂ =t, the central angle (that is, α+β) corresponding to the first sector area is greater than or equal to ωt+2arcsin(b/(2(Rc))).

In the third example, when AO intersects the vertex G of the shadow area, the exposure of the black and white frame is started. When BO intersects the vertex F of the shadow area, the exposure of the black and white frame ends. When BO intersects the G point of the shaded area, the color frame exposure starts. When AO intersects the vertex F of the shadow area again, the exposure of the color frame ends. When AO intersects the vertex G of the shadow area again, the black and white frame exposure of the next cycle begins. At this time, the exposure angle of the black and white frame is (γ-β), and the exposure angle of the color frame is still α.

Here, in the process from the intersection of AO and vertex F to the intersection of AO and vertex G (corresponding to the β angle of the fan-shaped area OBCA close to OA), neither black-and-white frame exposure nor color light exposure is performed. Similarly, in the process where BO intersects with vertex F and intersects with vertex G (corresponding to the angle β of the fan-shaped area OBCA close to OB), neither black-and-white frame exposure nor color frame exposure is performed.

Meet at this time:

α=ωt ₁ ;

γ-β=ωt ₂ ;

When the exposure time of the color frame is equal to the exposure time of the black and white frame, that is, t ₁ =t ₂ =t, then α=π-β and γ=π.

That is, β=π-α=π-ωt, and the above formula (1) can be transformed into the following formula:

R≥c+b/(2sin((π–ωt)/2));

In some possible implementations, the exposure angle of the color frame may also be less than or equal to α, that is, α≥ωt ₁ , and the exposure angle of the black and white frame may also be less than or equal to γ-β, that is, γ-β≥ωt ₂ . When designing a circular filter, in order to ensure a compact structure, it can usually be designed as α=ωt ₁ and γ-β=ωt ₂ .

That is, when t ₁ =t ₂ =t, the central angle (that is, α+β) corresponding to the first sector area is greater than or equal to ωt+2arcsin(b/(2(Rc))), and/or the second sector The central angle (ie γ) corresponding to the area is greater than or equal to ωt+2arcsin(b/(2(Rc))).

It should be noted that the larger the angular velocity ω of the rotation of the circular filter, the smaller the _{exposure time t 1} or t _2.

In some alternative embodiments, the position of the OA can be detected by a Hall sensor. As an implementation manner, a magnet may be provided at position A on the filter, and a hall sensor may be provided at the first position corresponding to position A on the image sensor in FIG. 4. When the Hall sensor detects the maximum magnetic field intensity, it can be determined that OA intersects the vertex F.

When the filter rotates at a constant speed, a timer can be set to control the black and white frame exposure and the color frame exposure. As a possible implementation, the timing can be started from the moment when OA intersects the vertex F. A specific example, for the above first example, the black and white can be opened at the timing of frame exposure time is 0, t ₂ is the timing when the end of the monochrome frame exposure, and the exposure start color frame. After the _{duration of t 1} , OA intersects with the vertex F again, the timer is cleared and the timing is restarted.

Way two

The control unit 240 may include a driver and a lever. The first end of the lever is connected to the filter 210. The driver is used to control the movement of the filter 210 through the lever so that the The first filter area 2101 covers the photosensitive area of the image sensor, and the second filter area 2102 in the filter 210 covers the photosensitive area of the image sensor in the second time period.

Optionally, the second end of the lever can be fixed and used as a fulcrum. At this time, the driver can move the filter 210 connected to the first end of the lever by controlling the movement of the first position in the lever. Alternatively, the second position between the two end points on the lever can be fixed and used as a fulcrum. At this time, the driver can move the filter 210 connected to the first end of the lever by controlling the movement of the second end of the lever.

As an example, the driver may be a piezoelectric ceramic driver, but the embodiment of the present application is not limited thereto.

Optionally, when the control unit 230 includes a driver and a lever, the control unit 230 may also include a control chip, and the control chip may send a control instruction to the driver to control the operation of the driver, so as to realize the control of the filter by the lever. As an example, the control chip can read instructions or codes in the memory to achieve image acquisition.

Please refer to FIG. 6, which shows a specific example of the connection between the lever and the filter. Exemplarily, JK is the boundary line of the first filter region 2101 and the second filter region 2102. In addition, the first end J of the lever 2402 is connected to the filter 210, and the second end H is fixed. As an example, the second end H may be fixed by a bearing. As a specific implementation, the driver 2401 may include a linear motion part, and the linear motion part may be connected to the lever 2402 at point I through a bearing. As shown in FIG. 6, the distance e that the first end J of the lever 2402 moves up and down and the distance f that the linear movement part moves up and down satisfy the following relationship:

Among them, L1 represents the distance between the end point of the second end H and the point I, and L2 represents the distance between the end point of the first end J and the I motor.

Continuing to refer to FIG. 6, driven by the driver 2401, the filter moves around the fixed end H. Exemplarily, when the filter moves from bottom to top, when JK intersects a vertex M of the shadow area (as shown in Figure 7), infrared light can pass through the second filter area 2102 to reach the image sensor, and the color stops at this time. Frame exposure. Optionally, the black and white frame exposure can be started at this time. As the filter continues to rotate upward, the filter will reach the highest point.

Optionally, as shown in FIG. 8, when the filter reaches the highest point, the second filter area 2102 can still be located in front of the shadow area (ie, the image sensor). In some implementations, parameters such as the length of L1 and L2 and the displacement of the linear motion part can be controlled to realize that when the filter reaches the highest point, the second filter area 2102 can still be located in front of the image sensor.

After the filter reaches the highest point, it starts to swing downward. As the filter swings, after JK will intersect the vertex M again (as shown in FIG. 9), the second filter area 2102 has no intersecting area with the shadow part. Optionally, the black-and-white frame exposure can be ended at this time. The frame number of the black and white frame obtained at this time can be denoted as (2n-1). After the black and white frame exposure is over, as the filter continues to swing downward, the color frame exposure can be started.

Optionally, when the filter reaches the lowest point, the first filter area 2101 can still be located in front of the image sensor. In some implementations, parameters such as the length of L1 and L2 and the displacement of the linear motion part can be controlled to realize that when the filter reaches the lowest point, the first filter area 2101 can still be located in front of the image sensor.

After the filter reaches its lowest point, it starts to move upward. With the movement of the filter, when JK intersects the vertex M again, the color frame exposure ends. Exemplarily, the frame number of the color frame obtained at this time may be denoted as (2n).

It should be understood that the "up", "down", and "rear" described in the embodiments of the present application are only for those skilled in the art to better understand the movement of the lever in the embodiments of the present application, or the image sensor and filter The relative positions of the slices are not used to limit the technical solutions of the embodiments of the present application.

The embodiment of the present application also provides a specific implementation manner of the filter, as shown in the following manner three.

Way three

The filter 210 may include an electrically controlled light-absorbing material. At this time, the control unit 230 may be specifically configured to apply a voltage to the filter 210, and by controlling the magnitude of the voltage, the filter 210 can pass visible light but not infrared light during the first time period. , And passing infrared light during the second time period.

In some optional embodiments, the above-mentioned electrically controlled light-absorbing material includes an organic color-changing material or a liquid crystal material. In this way, the response time of the electronically controlled light-absorbing material can reach the millimeter level, and can quickly respond to the switch between black and white frame exposure and color frame exposure.

Please refer to FIG. 10, which shows a specific example of the filter 210. Wherein, the electrically controlled light-absorbing material can be arranged between two layers of transparent electrodes. Exemplarily, the electrically controlled light-absorbing material can be wrapped between two layers of transparent electrodes. In this way, the control unit 230 can control the absorption peak of the light-absorbing material to switch between the infrared region and the ultraviolet region through the change of the voltage applied between the transparent electrodes, so as to realize the gating function of the spectrum.

Exemplarily, when the control unit 230 controls the light-absorbing material to gate infrared light, the infrared light can pass through the filter 210 to reach the image sensor, and at this time, black-and-white frame exposure can be performed. The frame number of the black and white frame obtained at this time can be denoted as (2n-1). When the control unit 230 controls the light-absorbing material to gate visible light (or visible light and infrared light), color frame exposure can be performed. The frame number of the color frame obtained at this time can be denoted as (2n).

In addition, in some optional embodiments, glass may also be provided on the outside of the transparent electrode (that is, the side away from the electrically controlled light-absorbing material).

FIG. 11 shows a schematic diagram of another image acquisition device 300 provided by an embodiment of the present application. As shown in FIG. 11, the device 300 includes an optical unit 301, an infrared fill light unit 302, a tunable filter 303, an image sensor 304, an image processing unit 305, a filter control unit 306 and an image synthesis unit 307. Among them, the tunable filter 303 may be an example of the above-mentioned filter 210, the image sensor 304 may be an example of the above-mentioned image sensor 220, and the filter control unit 306 may be an example of the above-mentioned control unit 230 The image synthesis unit 307 may be an example of the above image synthesis unit 230.

It should be understood that FIG. 11 shows the modules or units of the image acquisition device, but these modules or units are only examples. The image acquisition device in the embodiment of the present application may also include other modules or units, or include each module or unit in FIG. 11 Deformation of the unit. In addition, the image acquisition device in FIG. 11 may not include all the modules or units in FIG. 11.

Among them, the optical unit 301 is used to capture incident light and make the incident light image on the image sensor 304. Exemplarily, the main component in the optical unit includes an optical lens. Among them, the optical lens can be used for optical imaging. As an example, the optical lens may be a lens in which visible light and infrared light are confocal. Optionally, the light unit 301 may also include a filter, which is mainly used to polarize light and the like.

It should be noted that the light transmission between the optical unit 301 and the image sensor 304 is. In some possible descriptions, it can be considered that the transmission of light belongs to the transmission of data. Or, in some possible descriptions, it can be considered that the transmission of light does not belong to the transmission of data. The embodiment of the application does not limit this.

It should also be noted that the filters described in other parts of the embodiments of this application refer to filters used for visible light gating or infrared light gating, such as the filters in FIG. 2, FIG. 3, and The tunable filter shown in Figure 11, etc.

Wherein, the tunable filter 303 can be arranged in front of the optical lens or behind the lens, which is not limited. In addition, when the optical lens includes at least two optical lenses, the tunable filter 303 may also be disposed between two of the at least two optical lenses (this time may be referred to as the tunable filter 303 setting In the lens)

When the tunable filter 303 is arranged in front of the optical lens, after the incident light is gated by the tunable filter 303, the optical lens makes the visible light or infrared light gated by the tunable filter 303 image on the image sensor 304 . When the tunable filter 303 is arranged behind the optical lens, after the incident light passes through the light lens, the tunable filter 303 gates the visible light or infrared light therein, and then the visible light or infrared light passing through the filter is Image on the image sensor 304. When the tunable filter 303 is set in the optical lens, the tunable filter 303 gates visible light or infrared light during the incident light passing through the optical lens, and the light is in the image after passing through the optical lens. Image on sensor 304.

The infrared supplement light unit 302 is used to supplement light to the photographed environment in a low-illuminance environment. As an example, infrared light emitting diodes (light emitting diodes, LEDs) may be used, or other light emitting devices or devices may be used. The center wavelength of the infrared light supplement unit 302 can be 750 nm, 850 nm, or 950 nm, which is not limited in the embodiment of the present application.

In some possible implementation manners, the turning on and off, the light intensity, or the center wavelength of the infrared supplement light unit 302 can be controlled by a switch or a level signal.

The image processing unit 305 is configured to process the digital image signal acquired by the image sensor 304, such as at least one of demosaicing, automatic exposure, and automatic white balance. As an example, the image processing unit 305 may be implemented by dedicated hardware, such as an image signal processor (image signal processor, ISP), which is not limited in the embodiment of the present application.

The filter control unit 306 is used to control the tunable filter 303 so that the tunable filter 303 gates the visible light in the incident light and cannot pass the infrared light in the incident light at the first image sampling interval. The sampling interval selects infrared light in the incident light, or selects infrared light and visible light in the incident light at the second image sampling interval.

Exemplarily, the filter control unit 306 may transmit a control signal for the tunable filter 303 to the tunable filter 303, so that the tunable filter 303 gates the incident light at the first image sampling interval. Visible light does not pass infrared light in the incident light, and the infrared light in the incident light is gated in the second time period, or the infrared light and the visible light in the incident light are gated in the second image sampling interval.

Optionally, the tunable filter 303 may send a feedback signal about the tunable filter 303 to the filter control unit 306, so that the filter control unit 306 knows the gating status of the tunable filter 303.

In some optional embodiments, the filter control unit 306 may control the start and end of the exposure of the image sensor 304 according to the light gating condition of the tunable filter 303. That is, the filter control unit 306 can control the synchronization of the light gating of the tunable filter 303 and the exposure of the image sensor 304.

For example, when the filter control unit 306 outputs a control signal to control the tunable filter 303 to switch to visible light and not allow infrared light to pass, the control signal can also be used to trigger the image sensor 304 to perform color framing. exposure. When the filter control unit 306 outputs a control signal to control the tunable filter 303 to switch to select infrared light, the control signal can also be used to trigger the image sensor 304 to perform black and white frame exposure. Based on this, the embodiment of the application can realize the synchronization of the switching of the tunable filter 303 and the exposure of the image sensor 304.

Optionally, after the exposure time is met, the filter control unit 306 may output one or more control signals to stop the exposure of the image sensor 304.

In some optional embodiments, the filter control unit 306 may control the image synthesis unit to perform image synthesis according to the light gating condition of the tunable filter 303. For example, when the tunable filter 303 is a color wheel, when the color wheel rotates once, the image synthesis unit is controlled to synthesize the first image and the second image obtained in the period to obtain the target image.

Specifically, as the tunable filter 303 is switched, the image sensor 304 will periodically alternately output a first image containing chromaticity information and a second image containing brightness information. As an example, the image sensor 304 may successively output a frame of a first image and a frame of a second image in one image acquisition period, where the frame number of the first image and the frame number of the second image in one period are adjacent. As a specific example, an image acquisition cycle may include the second image of the (2n-1)th frame (which can be denoted as I(2n-1)) and the 2n frame of the first image (which can be denoted as I(2n)), Wherein n is an integer greater than or equal to 1.

In some optional embodiments, after the first image and the second image in an image acquisition period are processed by the image processing unit 305, the filter control unit 306 controls the image synthesis unit 307 to perform Two frames of images are combined to obtain the target image.

Taking Y (luminance) UV (chrominance) color space as an example, the UV component in the target image can be the UV component of the even-numbered frame image (that is, the first image), and the Y component in the target image can be the odd-numbered frame image (Ie the second image) Y component. Among them, the UV component is used to represent the chrominance information of the image, and the Y component is used to represent the brightness information of the image.

Exemplarily, the target image can be denoted as I', then I'(n) of the nth frame can be expressed as the following formula:

_I'Y (n)=I _Y (2n-1) (2)

_I'UV (n) = I _UV (2n) (3)

Among them, _I'Y (n) represents the Y component of the target image with frame number n, I _Y (2n-1) represents the Y component of the second image with frame number (2n-1), and _I'UV (n) Represents the UV component of the target image with frame number n, and I _UV (2n) represents the UV component of the second image with frame number 2n.

From the above formula (2) and formula (3), it can be seen that after the second image of the (2n-1)th frame and the first image of the 2nth frame are synthesized by the image synthesis unit 307, a target image of frame number n can be obtained .

FIG. 12 shows a schematic diagram of the image synthesis unit 307 synthesizing the first image and the second image in the YUV color space. The second image with a frame number of 1 and the first image with a frame number of 2 are two frames of images of the image sensor 304 in the first image acquisition period. The image synthesis unit 307 may synthesize the two frames of images into a target image with a frame number of 1. Among them, the Y component of the target image with frame number 1 comes from the Y component of the second image with frame number 1, and the UV component comes from the UV component of the first image with frame number 2. Similarly, the image synthesizing unit 307 can synthesize the second image with the frame number of 3 and the first image with the frame number of 4 in the second image acquisition period to obtain the target image with the frame number 2. Among them, the Y component of the target image with frame number 2 comes from the Y component of the second image with frame number 3, and the UV component comes from the Y component of the first image with frame number 4.

It should be understood that FIG. 12 only shows an example of the first image acquisition period and the second image acquisition period. After the second image acquisition period, the image sensor may also acquire the first image and the first image in the subsequent image acquisition period. For the second image, the manner of image synthesis in the subsequent image acquisition period can be referred to the description in the first image acquisition period or the second image acquisition period. For the sake of brevity, it will not be repeated here.

As an example, the filter control unit 306 can be provided with a Hall sensor and a timer to monitor the position information of the tunable filter 303, and according to the position information and the timer information, send a message to the tunable filter 303, image The sensor and the image synthesis unit send control signals.

In some possible implementations, the control unit 230 is further configured to determine the environmental illuminance of the photographed environment according to the gain of the image sensor 220. When the image acquisition device 200 is actually used, as the illuminance of the subject environment decreases, the gain of the image sensor 220 gradually increases. When the gain of the image sensor 220 is greater than the preset value, it can be determined that the subject environment is a low-illuminance environment. At this time, the filter can be controlled to pass visible light but not infrared light at the first image sampling interval, and pass infrared light at the second image sampling interval. As an example, the preset value may be 36dB.

In some optional embodiments, the control unit 230 is further configured to, when determining that the gain of the image sensor 220 is less than or equal to a preset value, control the image sensor 220 to photoelectrically image the visible light at the third image sampling interval to obtain the target image. Here, in one target image acquisition period, the target image can be directly acquired in the third image sampling interval.

Here, the third image sampling interval may be equal to the sum of the first image sampling interval and the second image sampling interval. For example, assuming that the image acquisition device needs 10 milliseconds (ms) to output an image, the first image sampling interval plus the second image sampling interval should be 10ms, and the two together produce an output image, and the third image sampling interval itself is 10ms, it generates an output image by itself.

In other words, the output frame rate of the image sensor in the third image sampling interval is half of the output frame rate of the image sensor in the first image sampling interval or the second image sampling interval, that is, the image sensor's gain is less than or equal to the preset When the value is set (that is, in a normal environment), the output frame rate is half of the output frame rate of the image sensor when its gain is greater than the preset value (that is, in a low-illuminance environment).

Exemplarily, for the device 300, the filter control unit 306 controls the output frame rate of the image sensor 304 at the first image sampling interval or the second image sampling interval to be the output frame rate of the image sensor 304 at the third image sampling interval. The output frame rate is 2 times.

In this way, after the image synthesis unit synthesizes the two frames of images captured by the image sensor into one frame of image, the frame rate of the output image of the image capture device in a low-illuminance environment can be kept unchanged compared to the frame rate of the output in a normal environment.

As an example, under normal ambient illuminance (for example, the gain of the image sensor is lower than 36 dB), the output frame rate of the image sensor is 25 frames per second. When it is detected that the photographed environment is a low-light environment (for example, the gain of the image sensor is 36dB), the output frame rate of the image sensor can be modified to 50 frames per second.

In some optional embodiments, the control unit 230 may move the filter out of the optical path at the third image sampling interval, which is not limited in the embodiment of the present application.

Therefore, in the embodiment of the present application, the light passing through the filter is photoelectrically imaged at the first image sampling interval to obtain a first image containing chromaticity information, and the light passing through the filter is performed at the second image sampling interval. Photoelectric imaging is to obtain a second image containing brightness information, and then synthesize the first image and the second image to obtain a higher-quality color target image. Since the image sensor in the embodiment of the present application can obtain the first image and the second image in different time periods, the embodiment of the present application can use one sensor, which reduces the material cost, and does not require registration of the image sensor, reducing complexity It can help reduce the volume of the device.

FIG. 13 shows a schematic flowchart of an image acquisition method 400 provided by an embodiment of the present application. The method 400 may be executed by the image acquisition device 200 in FIG. 2 above, or executed by the image acquisition device 300 in FIG. 11. As shown in FIG. 13, the method 400 includes steps 410 to 440.

It should be noted that FIG. 13 shows the steps or operations of the image acquisition method, but these steps or operations are only examples, and the embodiment of the present application may also perform other operations or variations of each operation in FIG. 13. In addition, it is possible that not all the operations in Figure 13 need to be performed.

Optionally, 410, estimate the environmental illuminance.

Exemplarily, the environmental illuminance can be estimated based on the gain of the image sensor. Then, it may be determined whether to perform step 420 according to the estimated environmental illuminance.

In the process of image acquisition, as the environmental illuminance decreases, the gain of the image sensor gradually increases. For example, when the gain of the image sensor is lower than 36dB, it can be determined that the subject environment is a normal illuminance environment. When the gain of the image sensor is higher than 36dB, it can be determined that the subject environment is a low-light environment.

When it is determined that the photographed environment is a normal illuminance environment, the filter in FIG. 2 or FIG. 3 can be moved out of the light path, and the image obtained by the photoelectric imaging of the visible light by the image sensor can be output, displayed or saved. As an example, the output frame rate of the image sensor may be 25 frames per second at this time.

When it is determined that the photographed environment is a low-illuminance environment, step 420 may be performed.

As an example, step 410 may be executed by the control unit 230 in FIG. 2, or step 410 may be executed by the filter control unit 306 in FIG. 11.

420. Acquire a first image and a second image.

Specifically, the visible light passing through the filter is acquired at the first image sampling interval, and the infrared light passing through the filter is acquired at the second image sampling interval. Among them, the filter can be referred to the above description, for the sake of brevity, it will not be repeated here.

Then, an image sensor is used to photoelectrically image the light passing through the filter at the first image sampling interval, so that a first image can be obtained. The second image can be obtained by photoelectrically imaging the light passing through the filter at the second image sampling interval by the image sensor.

In some embodiments, after determining that the subject environment is a low-illuminance environment, the output frame rate of the image sensor can be modified from 25 frames/sec to 50 frames/sec, that is, the frame rate of the image capture device is doubled. It can ensure that the output frame rate of the image acquisition device in a low illumination environment is the same as the output frame rate under normal illumination.

In some embodiments, when the filter is a circular filter, the motor may be started before step 420 to drive the circular filter to rotate. When the rotation speed of the circular filter reaches the set rotation speed, the exposure of the black and white frame and the color frame is turned on to obtain the first image and the second image.

As an example, the filter and image sensor may be controlled by the control unit 230 in FIG. 2 to obtain the first image and the second image, or the filter control unit in FIG. 11 may control the tunable filter and the image sensor, Acquire the first image and the second image. Specifically, the process of the first image and the second image can be referred to the above description, and for the sake of brevity, it will not be repeated here.

Optionally, after step 420, basic image processing may be performed on the acquired first image and second image. For example, demosaicing, white balance, color correction, etc., which are not limited in the embodiment of the present application.

430. Synthesize the first image and the second image to generate a target image.

As an example, the control unit 230 may control the image synthesis unit to perform this step 430. Specifically, the synthesis process can be referred to the above description, for the sake of brevity, it will not be repeated here.

Optionally, 440, transmit, display, or store the target image.

Exemplarily, the target image can be output to the relevant business module. The relevant business module performs image transmission or display/storage operations on the synthesized image according to the business application.

It should be noted that the examples in the embodiments of the present application are merely to help those skilled in the art understand and implement the embodiments of the present application, rather than limiting the scope of the embodiments of the present application. Those skilled in the art can make equivalent transformations or modifications according to the examples given herein, and such transformations or modifications should still fall within the scope of the embodiments of the present application.

It should also be noted that in various embodiments of the present invention, the size of the sequence number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not be implemented in the present invention. The implementation process of the example constitutes any limitation.

The embodiments of the present application also provide a computer-readable storage medium, which includes a computer program, which when running on a computer, causes the computer to execute the relevant instructions in the method or device provided in the above-mentioned embodiments.

The embodiments of the present application also provide a computer program product containing instructions. When the computer program product runs on a computer, the computer executes the related instructions in the method or device provided in the above-mentioned embodiments.

A person of ordinary skill in the art may realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and conciseness of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed system, device, and method can be implemented in other ways. For example, the device embodiments described above are merely illustrative, for example, the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components may be combined or It can be integrated into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional units in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.

If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solution of the present application essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program code .

The above are only specific implementations of this application, but the protection scope of this application is not limited to this. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application. Should be covered within the scope of protection of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

An image acquisition device, characterized in that it comprises:

Filter, used to gate the incident light;

An image sensor, used for photoelectric imaging of the light passing through the filter in the incident light;

The control unit is connected to the filter and the image sensor, and is used to control the filter to pass the visible light of the incident light and block the infrared light of the incident light to pass through at the first image sampling interval, And pass the infrared light in the incident light at the second image sampling interval; and

It is used to control the image sensor to photoelectrically image the light that passes through the filter at the first image sampling interval in the incident light, to obtain a first image, and to perform photoelectric imaging of the incident light in the second image. Photoelectric imaging is performed through the light of the filter at the image sampling interval to obtain a second image; and

The image synthesis unit is used to synthesize the first image and the second image to generate a first target image.
The device according to claim 1, wherein the filter comprises a first filter area and a second filter area, the first filter area is used to pass visible light and block infrared light from passing through, and The second filter area is used to pass infrared light;

The control unit is specifically configured to use the first filter area to filter the incident light at the first image sampling interval, and use the second filter area to filter the incident light at the second image sampling interval. The incident light is filtered.
3. The device of claim 2, wherein the filter is a circular filter, the first filter area is a first fan-shaped area of the circular filter, and the second The filter area is the second fan-shaped area of the circular filter;

The control unit includes a motor, and the motor is used to control the circular filter to rotate around the center of the circular filter, so that the first fan-shaped area covers all areas at the first image sampling interval. In the photosensitive area of the image sensor, the second fan-shaped area covers the photosensitive area at the second image sampling interval.
The device according to claim 3, wherein the photosensitive area is a rectangular area, and when the duration of the first image sampling interval is the same as the duration of the second image sampling interval, the circular The filter meets the following conditions:

R≥c+b/(2sin((π–ωt)/2));

Where ω represents the angular velocity of the rotation of the circular filter, t represents the duration of the first image sampling interval or the second image sampling interval, R represents the radius of the circular filter, and b represents the total The first side length of the photosensitive area of the image sensor, c represents the second side length of the photosensitive area of the image sensor, and the first side and the second side are perpendicular to each other.
The device according to claim 4, wherein the central angle corresponding to the first sector area is greater than or equal to ωt+2arcsin(b/(2(Rc))), and/or the second sector area corresponds to The central angle of is greater than or equal to ωt+2arcsin(b/(2(Rc))).
The device according to claim 3, wherein the photosensitive area is a rectangular area, and when the duration of the first image sampling interval is the same as the duration of the second image sampling interval, the circular The filter meets the following conditions:

R≥c+b/(2sin(π–ωt));

Where ω represents the angular velocity of the rotation of the circular filter, t represents the duration of the first image sampling interval or the second image sampling interval, R represents the radius of the circular filter, and b represents the total The first side length of the photosensitive area of the image sensor, c represents the second side length of the photosensitive area of the image sensor, and the first side and the second side are perpendicular to each other.
The device according to claim 6, wherein the central angle corresponding to the first sector area is greater than or equal to ωt+2arcsin(b/(2(R-c))).
7. The device according to any one of claims 3-7, wherein the photosensitive area is a rectangular area, and the circular filter satisfies the following conditions:

R≥c+b/(2sin(β/2));

Wherein, R represents the radius of the circular filter, b represents the first side length of the photosensitive area of the image sensor, c represents the second side length of the photosensitive area of the image sensor, and β is the first The first side and the second side are perpendicular to each other at the central angle of the circular filter opposite to the side length.
8. The device according to claim 8, wherein the rotational angular velocity of the circular filter is ω, the central angle of the first sector area in the circular filter is (2π-γ), and the The central angle of the second sector area is γ, the exposure time of the first image is t 1 , and the exposure time of the second image is t 2 ,

Then ω, γ, t 1 , t 2 satisfy the following formula:

2π-γ-β≥ωt 1 ;

γ+β≥ωt 2 or γ≥ωt 2 or γ-β≥ωt 2 .
The device according to claim 2, wherein the control unit comprises a driver and a lever, the first end of the lever is connected to the filter, and the driver is used to control the filter through the lever. The light sheet moves so that the first filter area covers the photosensitive area of the image sensor at the first image sampling interval, and the second filter area covers the photosensitive area at the second image sampling interval .
The device according to claim 1, wherein the filter includes an electrically controlled light-absorbing material,

The control unit is specifically configured to apply a voltage on the filter, and by controlling the magnitude of the voltage to make the filter pass visible light and block infrared light from passing in the first image sampling interval, and The second image sampling interval passes infrared light.
The device according to claim 11, wherein the electrically controlled light-absorbing material comprises an organic color-changing material or a liquid crystal material.
The device according to any one of claims 1-12, further comprising:

The optical unit includes an optical lens for capturing the incident light and causing the incident light to be imaged on the image sensor, wherein the optical filter is arranged before or after the optical lens, or is arranged on the Between the two lenses in the optical lens.
The device according to any one of claims 1-13, wherein the control unit is specifically configured to:

In the case of determining that the gain of the image sensor is greater than the preset value, control the filter to pass visible light and block infrared light at the first image sampling interval, and pass infrared light at the second image sampling interval And controlling the image sensor to perform photoelectric imaging of the light passing through the filter in the first image sampling interval, and perform photoelectric imaging of the light passing through the filter in the second image sampling interval.
The device according to any one of claims 1-14, wherein the control unit is further configured to:

When it is determined that the gain of the image sensor is less than or equal to the preset value, the image sensor is controlled to perform photoelectric imaging of the incident light in the third image sampling interval to obtain a second target image, wherein the third image The duration of the sampling interval is equal to the sum of the duration of the first image sampling interval and the duration of the second image sampling interval.
An image acquisition method, characterized in that it comprises:

Controlling the filter to pass visible light in the incident light and block infrared light from passing in the first image sampling interval, and controlling the filter to pass the infrared light in the incident light in the second image sampling interval;

Photoelectrically imaging the incident light that passes through the filter at the first image sampling interval by using an image sensor to obtain a first image;

Photoelectrically imaging the incident light that passes through the filter at the second image sampling interval by using the image sensor to obtain a second image;

The first image and the second image are synthesized by an image synthesis unit to generate a first target image.
The method according to claim 16, wherein the filter includes a first filter area and a second filter area, the first filter area is used to pass visible light and block infrared light from passing, and the first filter area The second filter area is used to pass infrared light;

Wherein, controlling the filter to pass visible light in the incident light and block infrared light from passing at the first image sampling interval, and controlling the filter to pass the infrared light in the incident light at the second image sampling interval includes:

The first filter area is controlled to filter incident light at the first image sampling interval, and the second filter area is controlled to filter incident light at the second image sampling interval.
The method according to claim 17, wherein the filter is a circular filter, the first filter area is a first fan-shaped area of the circular filter, and the second The filter area is the second fan-shaped area of the circular filter;

Wherein, controlling the first filter area to filter incident light at the first image sampling interval, and controlling the second filter area to filter incident light at the second image sampling interval includes:

The circular filter is controlled to rotate around the center of the circular filter by a motor, so that the first fan-shaped area covers the photosensitive area of the image sensor at the first image sampling interval. Two fan-shaped areas cover the photosensitive area at the second image sampling interval.
The method according to claim 18, wherein the photosensitive area is a rectangular area, and when the duration of the first image sampling interval is the same as the duration of the second image sampling interval, the circular The filter meets the following conditions:

R≥c+b/(2sin((π–ωt)/2));

Where ω represents the angular velocity of the rotation of the circular filter, t represents the duration of the first image sampling interval or the second image sampling interval, R represents the radius of the circular filter, and b represents the total The first side length of the photosensitive area of the image sensor, c represents the second side length of the photosensitive area of the image sensor, and the first side and the second side are perpendicular to each other.
The method according to claim 19, wherein the central angle corresponding to the first sector area is greater than or equal to ωt+2arcsin(b/(2(Rc))), and/or the second sector area corresponds to The central angle of is greater than or equal to ωt+2arcsin(b/(2(Rc))).
The method according to claim 18, wherein the photosensitive area is a rectangular area, and when the duration of the first image sampling interval is the same as the duration of the second image sampling interval, the circular The filter meets the following conditions:

R≥c+b/(2sin(π–ωt));

Where ω represents the angular velocity of the rotation of the circular filter, t represents the duration of the first image sampling interval or the second image sampling interval, R represents the radius of the circular filter, and b represents the total The first side length of the photosensitive area of the image sensor, c represents the second side length of the photosensitive area of the image sensor, and the first side and the second side are perpendicular to each other.
The device according to claim 21, wherein the central angle corresponding to the first sector area is greater than or equal to ωt+2arcsin(b/(2(R-c))).
The device according to any one of claims 18-22, wherein the photosensitive area is a rectangular area, and the circular filter satisfies the following conditions:

R≥c+b/(2sin(β/2));

Wherein, R represents the radius of the circular filter, b represents the first side length of the photosensitive area of the image sensor, c represents the second side length of the photosensitive area of the image sensor, and β is the first The first side and the second side are perpendicular to each other at the central angle of the circular filter opposite to the side length.
The device according to claim 23, wherein the rotational angular velocity of the circular filter is ω, the central angle of the first sector area in the circular filter is (2π-γ), and the The central angle of the second sector area is γ, the exposure time of the first image is t 1 , and the exposure time of the second image is t 2 ,

Then ω, γ, t 1 , t 2 satisfy the following formula:

2π-γ-β≥ωt 1 ;

γ+β≥ωt 2 or γ≥ωt 2 or γ-β≥ωt 2 .
The method of claim 17, wherein the first filter area is controlled to filter incident light at the first image sampling interval, and the second filter is controlled at the second image sampling interval. The light area filters the incident light, including:

The movement of the filter is controlled by a driver and a lever, so that the first filter area covers the photosensitive area of the image sensor during the first image sampling interval, and the visible light passing through the filter is acquired. The second image sampling interval, the second filter area covers the photosensitive area of the image sensor, and the infrared light passing through the filter is acquired;

Wherein, the first end of the lever is connected with the filter.
The method according to claim 16, wherein the filter includes an electrically controlled light-absorbing material,

Wherein, controlling the filter to pass visible light in the incident light and block infrared light from passing at the first image sampling interval, and controlling the filter to pass the infrared light in the incident light at the second image sampling interval includes:

By applying a voltage to the filter and controlling the magnitude of the voltage so that the filter passes visible light and blocks infrared light from passing through the first image sampling interval, the visible light passing through the filter is obtained , And passing infrared light at the second image sampling interval to obtain the infrared light passing through the filter.
The method according to claim 26, wherein the electrically controlled light-absorbing material comprises an organic color-changing material or a liquid crystal material.
The method according to any one of claims 16-27, further comprising:

In the case of determining that the gain of the image sensor is greater than the preset value, control the filter to pass visible light and block infrared light at the first image sampling interval, and pass infrared light at the second image sampling interval And controlling the image sensor to perform photoelectric imaging of the light passing through the filter in the first image sampling interval, and perform photoelectric imaging of the light passing through the filter in the second image sampling interval.
The method according to any one of claims 16-28, further comprising:

When it is determined that the gain of the image sensor is less than or equal to the preset value, the incident light is photoelectrically imaged at a third image sampling interval by the image sensor to obtain a second target image, wherein the third image sampling interval is The duration is equal to the sum of the duration of the first image sampling interval and the duration of the second image sampling interval.