WO2021175211A1

WO2021175211A1 - Exposure control-based video capture method and apparatus

Info

Publication number: WO2021175211A1
Application number: PCT/CN2021/078677
Authority: WO
Inventors: 胡彬林; 刘俊; 唐娜
Original assignee: 华为技术有限公司
Priority date: 2020-03-03
Filing date: 2021-03-02
Publication date: 2021-09-10
Also published as: CN113347368A; CN113347368B

Abstract

The present application relates to the field of image capture technology and provides an exposure control-based video capture method and an apparatus. The method comprises: using a same camera module, within a same time period, and performing in alternation automatic exposure and manual exposure on image frames being shot, wherein a manual exposure parameter is set according to an automatic exposure parameter, and has the unique quality of periodically changing. In the described method, the automatic exposure parameter slowly changes according to the brightness of an environment, and so automatically exposed video frames can be used for video display; and the manual exposure parameter can be adjusted, thereby allowing for rapid response to a change of brightness in a scenario; the manual exposure parameter has a plurality of values within a single period, and a video frame will be shot and obtained based on every value of the manual exposure parameter, allowing a relatively optimally exposed frame to be found for every region within a scenario having slowly changing brightness, and utilizing the video frames obtained by the manual exposure parameter to perform video analysis thereby improves video analysis accuracy.

Description

Video acquisition method and device based on exposure control

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on March 3, 2020, the application number is 202010140270.8, and the application name is "a method and device for video capture based on exposure control", the entire content of which is incorporated by reference In this application.

Technical field

This application relates to the field of image acquisition technology, and in particular to a video acquisition method and video acquisition device based on exposure control.

Background technique

With the rapid development of science and technology, in recent years, intelligent security has gradually become the key direction of the transformation and upgrading of security enterprises. In the security system, as the most front-end image acquisition equipment, surveillance cameras play a key role in it.

For current surveillance cameras, the light is weak when working at night and it is difficult to collect high-quality color images. Therefore, the method of supplement light is often used to increase the brightness of the image collected by the camera, but the additional light is currently unable to The sunlight can fill light evenly. In this case, when using the automatic exposure method to shoot, because the exposure parameters are calculated by considering the brightness of the overall scene, it is inevitable that some areas of the scene will be overexposed or underexposed. Status, resulting in a decrease in the accuracy of video analysis (also known as: intelligent video analysis, video content analysis, such as: video target detection, recognition and/or analysis). When the manual exposure method is used for shooting, because the exposure parameters change slowly and discontinuously, such video frames cannot be used for video display like auto-exposure video frames.

Therefore, how to collect a video stream through exposure control so that the video stream can not only meet the viewing needs of users, but also improve the accuracy of video analysis is a problem to be solved urgently.

Summary of the invention

This application provides a video capture method and device based on exposure control, which alternately performs automatic exposure and manual exposure during the process of video frame capture, so that the obtained video stream can not only meet the needs of users for viewing, but also ensure subsequent video analysis When you can find a suitable exposure video frame.

The first aspect of the present application provides a video acquisition method based on exposure control. The method includes: determining automatic exposure parameters of a camera module according to environmental data, and then generating manual exposure parameters according to the automatic exposure parameters. The camera module collects video frames based on automatic exposure parameters to obtain a first video stream, and collects video frames based on manual exposure parameters to obtain a second video stream. Among them, the video frames constituting the first video stream and the video frames constituting the second video stream are video streams that are alternately collected by the camera module at the same time period. Exemplarily, the first video stream is composed of odd-numbered frames among all video frames collected by the camera module, and the second video stream is composed of even-numbered frames among all video frames collected by the camera module, and vice versa. It should be noted that, in addition to the alternate acquisition method of one-frame interval such as odd-even frames, two or even three frames may also be interval. For example, the first video stream is composed of 1, 2, 4, 5, 7, and 8 of all the video frames collected by the camera module (take the first 9 video frames as an example), and the second video stream is composed of the camera module. The 3, 6, and 9 frames of all the video frames collected in the group constitute the video frame that constitutes the second video stream at an interval of 2 frames, and the subsequent video frames can be deduced by analogy. The present invention does not specifically limit the alternate collection method. It depends on the actual situation. The above-mentioned video acquisition method based on exposure control can obtain two video streams exposed by different methods. Among them, because the automatic exposure parameters change slowly, the video frames collected by the automatic exposure can be used for video display; the manual exposure parameters are adjustable, so that the video frames collected by the manual exposure can respond to the brightness changes in the scene to be shot in time. Improve the quality of video capture.

In another possible implementation manner, the automatic exposure parameters include: shutter time, aperture size, and gain size, and obtaining manual exposure parameters according to the automatic exposure parameters includes: for each parameter of the first part of the automatic exposure parameters, according to The parameter generates a set of sequences and periodically expands to obtain the parameter expansion sequence, which is used as the corresponding parameter in the first part of the manual exposure parameter; keeps the second part of the automatic exposure parameter unchanged, as the manual The second part of the exposure parameters. In the above-mentioned video acquisition method based on exposure control, a set of exposure parameter sequences is generated for each parameter of the first part of the automatic exposure parameters, and each value in the exposure parameter sequence is determined by a certain area of the scene to be shot. Optimal exposure parameters required. In this exposure mode, based on each value of the manual exposure parameter, a video frame will be captured, so that for each area, at least one manual exposure video frame with more appropriate exposure for the area can be obtained, which improves video capture The device's ability to adapt to areas with uneven brightness in the scene. Optionally, in a specific implementation, the first part of the parameters includes the shutter time and/or the gain size in the automatic exposure parameters, and the second part of the parameters includes the aperture size in the automatic exposure parameters.

In another possible implementation, the first part of the automatic exposure parameter includes the shutter time, and for each parameter of the first part of the automatic exposure parameter, a set of sequences is generated according to the parameter and periodically expanded to obtain the parameter expansion Sequence, the corresponding parameters in the first part of the parameters as the manual exposure parameters include: acquiring the shutter time in the automatic exposure parameters and generating a set of shutter time sequences according to the shutter time, and then repeating the shutter time sequence cyclically to obtain the shutter time extended sequence as The shutter time in the manual exposure parameters. In the above implementation, the two parameters in the manual exposure parameters: the aperture size and the gain size are consistent with the parameters corresponding to the automatic exposure parameters. The shutter time in the manual exposure parameters is based on the shutter time included in the automatic exposure parameters. Perform periodic expansion to obtain the parameter expansion sequence, so that when the video capture device shoots video frames based on manual exposure parameters, it can adapt to scenes with multi-level uneven light and dark, not just scenes with significant contrast of light and dark. Optionally, a set of shutter time sequences included in the manual exposure parameter is composed of at least 3 shutter time values. For example, if the shutter time of the automatic exposure parameter is 1/500 second, based on the light distribution of the scene to be shot, a set of shutter time series included in the manual exposure parameter includes [1/500 second, 1/250 second, 1/125 second , 1/250 second, 1/500 second], according to the parameter expansion sequence of the periodically expanded shutter time, including [1/500 second, 1/250 second, 1/125 second, 1/250 second, 1/500 second , 1/250 second, 1/125 second, 1/250 second, 1/500 second,...].

In another possible implementation, the first part of the automatic exposure parameters includes the shutter time and the gain size. For each parameter of the first part of the automatic exposure parameters, a set of sequences is generated according to the parameters and periodically expanded to obtain the The parameter expansion sequence, as the corresponding parameter in the first part of the manual exposure parameter, also includes: obtaining the gain size in the automatic exposure parameter and generating a set of gain size sequence according to the gain size; loop repeating the gain size sequence to obtain the gain The size expansion sequence is used as the gain size in the manual exposure parameter. Linking the two parameters of manual exposure with shutter time and gain size and automatic exposure, in addition to improving the camera's ability to adapt to scenes with gradual brightness changes, it also increases the signal-to-noise ratio of the second video stream and improves image quality. Similar to the example in the previous possible implementation, the gain size in the manual exposure parameter can also be expanded into a set of gain size sequence according to the gain size in the automatic exposure parameter, and it is periodically expanded to obtain the gain size in the manual exposure parameter. The size of the gain.

In another possible implementation manner, the shutter time in the manual exposure parameter and/or the gain in the manual exposure parameter changes periodically, and includes rising and falling changes in one cycle. The change rule of the manual exposure parameter conforms to the brightness distribution of the scene shot by the video capture device.

In another possible implementation manner, the second part of the automatic exposure parameters includes: aperture size. In order to ensure the stability of the camera and not affect the video stream of the automatic exposure branch, the aperture size of the manual exposure parameter should be consistent with the aperture size of the automatic exposure.

In another possible implementation manner, the first video stream is used for video display, and the second video stream is used for video analysis. The first video stream uses automatic exposure, and its exposure parameters can only change slowly in order to adapt to the visual ability of the human eye, so it can be used for video display. The second video stream uses manual exposure. In order to make each light and dark area in the scene have a better exposure frame, manual exposure parameters need to be greatly changed, which is not suitable for human viewing, but can be sent to the intelligent analysis module for identification Analyze various light and dark areas in the scene.

In another possible implementation manner, the video frames constituting the first video stream and the video frames constituting the second video stream are fused to obtain the third video stream. That is, the third video stream contains video frames obtained by two exposure methods: automatic exposure and manual exposure.

In another possible implementation, the third video stream will be used for video analysis. Compared with only analyzing the video stream under manual exposure, this implementation provides more video frames for video analysis, which greatly improves the accuracy of identification and analysis.

In a second aspect, the present application provides a video capture device, which includes a camera module, a first image signal processing module, and a second image signal processing module. The first image signal processing module is used to determine the automatic exposure parameters of the camera module according to environmental data; the second image signal processing module is used to obtain manual exposure parameters according to the automatic exposure parameters; the camera module includes The lens and the image sensor are used to collect video frames based on the automatic exposure parameters to form a first video stream; and are also used to collect video frames based on the manual exposure parameters to form a second video stream, where the first video is formed The video frames of the stream and the video frames constituting the second video stream are alternately collected by the camera module.

In another possible implementation manner, the automatic exposure parameters include: shutter time, aperture size, and gain, and the second image signal processing module is used to: for each parameter of the first part of the automatic exposure parameter, generate a parameter based on the parameter Group sequences and perform periodic expansion to obtain the parameter expansion sequence as the corresponding parameter in the first part of the manual exposure parameter; keep the second part of the automatic exposure parameter unchanged as the second part of the manual exposure parameter.

In another possible implementation manner, the first part of the automatic exposure parameter includes the shutter time, and the second image signal processing module is used to: obtain the shutter time in the automatic exposure parameter; and generate a set of shutter time series according to the shutter time , The shutter time extended sequence obtained by cyclically repeating the shutter time sequence is used as the shutter time in the manual exposure parameter.

In another possible implementation manner, the first part of the automatic exposure parameters further includes a gain size, and the second image signal processing module is further used to: obtain the gain size in the automatic exposure parameter; and generate a set of gains according to the gain size The size sequence, the gain size expansion sequence obtained by cyclically repeating the gain size sequence is used as the gain size in the manual exposure parameter.

In another possible implementation manner, the shutter time in the manual exposure parameter and the gain in the manual exposure parameter change periodically, and include rising and falling changes in one cycle.

In another possible implementation manner, the second part of the automatic exposure parameters includes: aperture size.

In another possible implementation manner, the first video stream is used for video display, and the second video stream is used for video analysis.

In another possible implementation manner, the device further includes a fusion module, configured to perform fusion processing on the video frames constituting the first video stream and the video frames constituting the second video stream to obtain the third video stream.

In another possible implementation manner, the third video stream is used for video analysis.

The technical effects that can be achieved by the video capture device and possible implementation manners provided by the second aspect are the same as those that can be achieved by the exposure control-based video capture method and possible implementation manners according to the first aspect, and will not be repeated here. .

In a third aspect, the present application provides a video surveillance system, the system includes the video capture device described in the second aspect; a video display module for displaying the first video stream; a video analysis module for analyzing the second and/ Or the third video stream.

In a fourth aspect, this application provides a computer program product that contains computer-readable code instructions. When these computer-readable code instructions are executed by the computer, the computer can configure the video capture device so that the device can execute The method described in the first aspect and any of the various possible implementation manners of the first aspect.

In the fifth aspect, the present application provides a non-transitory computer-readable storage medium, which contains computer program code instructions, which can be executed when the computer-readable code instructions are executed by a computer The configuration of the video capture device by the computer is implemented, so that the device can execute the method described in the first aspect and any of the various possible implementation manners of the first aspect. The non-transitory computer-readable storage medium includes one or more of the following options: Read-Only Memory (ROM), Programmable ROM (Programmable ROM), Erasable PROM (Erasable PROM, EPROM) ), flash memory (Flash Memory), electrical EPROM (Electrically EPROM, EEPROM) and hard drive (Hard Drive).

Description of the drawings

FIG. 1 is a schematic diagram of the application architecture of a video capture method based on exposure control provided by an embodiment of the application.

FIG. 2 is a schematic diagram of a scene with uneven brightness distribution provided by an embodiment of the application.

FIG. 3 is a schematic diagram of shutter time required for different brightness areas in a scene provided by an embodiment of the application.

FIG. 4 is a schematic flowchart of an automatic exposure method provided by an embodiment of the application.

FIG. 5 is a schematic flowchart of steps of a video acquisition method based on exposure control provided by an embodiment of the application.

FIG. 6 is a schematic structural diagram of a video capture method based on exposure control provided by an embodiment of the application.

FIG. 7 is a schematic diagram of a flow of generating manual exposure parameters according to automatic exposure parameters according to an embodiment of the application.

FIG. 8 is a schematic diagram of a shutter cycle period provided by an embodiment of the application.

FIG. 9 is a schematic diagram of a parity frame fusion structure provided by an embodiment of the application.

FIG. 10 is a schematic structural diagram of a video capture device provided by an embodiment of the application.

Detailed ways

In order to facilitate understanding, the technical terms involved in the embodiments of the present application are described first.

1) Exposure parameters: The exposure parameters during shooting determine the brightness of the image. The exposure parameters include one or more of the shutter time, gain size, or aperture size.

Shutter Time (ST), also known as exposure time, shutter is a valve that controls the light-in time. The longer the shutter time, the more light entering, and the brighter the captured image; the shorter the shutter time, the less light entering, and the darker the captured image.

Gain refers to the process of amplifying or reducing the electrical signal or digital signal after photoelectric conversion. Increase the gain, and the captured image will be brighter than the actual scene; reduce the gain, the captured image will be darker than the actual scene.

The aperture is used to control the amount of light entering the fuselage through the lens. Increasing the aperture means increasing the amount of light entering, and the captured image is brighter; reducing the aperture means reducing the amount of light entering, and the captured image is darker.

2) Automatic Exposure: It means that the camera automatically adjusts the exposure parameters (shutter time, gain, aperture) according to the ambient brightness data through the algorithm for exposure, so that the brightness of the subject is normal.

3) Manual Exposure: It is an exposure method that manually sets the camera's exposure parameters (shutter time, gain, and aperture).

4) Image Signal Processor (ISP): The main function is to process the signal output by the front-end image sensor. The main functions are white balance adjustment, automatic exposure control, etc.

The following will introduce a schematic diagram of the system application architecture of the exposure control method provided by the present application in conjunction with FIG. 1. As shown in FIG. 1, the camera system 12 includes a camera module 13, an image signal processor (ISP) 14, and an encoder 15.

The camera module 13 includes a lens module 131, an image sensor 132, and a gain control circuit 133 integrated on the image sensor. The lens module 131 includes an aperture in front of the lens and an optical lens combination, and is mainly used to collect light from the subject 11. The optical lens combination is usually a lens group composed of one or more pieces of optical glass (or plastic), and can be composed of a lens such as a concave lens, a convex lens, an M-type lens, or a combination of lenses. The image sensor 132 may be a CCD image sensor composed of a charge-coupled device (CCD), a CMOS image sensor composed of a complementary metal oxide semiconductor (CMOS), or a contact type A CIS image sensor composed of an image sensor (contact image sensor, CIS), etc. The image sensor is mainly used to receive the light signal transmitted from the camera module, convert the light signal into an electrical signal, and perform photoelectric conversion. The gain control circuit 133 is generally integrated in the image sensor, and is mainly used to ensure that the standard video signal that can be output in the captured image under different scene illumination conditions.

The image signal processor 15 (ISP) is a special digital signal processor (Digital Signal Processor, DSP), and its main function is to perform post-processing on the signal output by the front-end image sensor. Different ISPs are used to match image sensors from different manufacturers. The excellence of ISP is very important in the entire camera product, and it can directly affect the quality of the image presented to the user. The ISP has a special circuit to connect with the previous camera module, which can control the camera module 13 to adopt different camera parameters, which is used to realize the 2A control (Automatic white balance/Automatic exposure) that we often mentioned. Exposure) or 3A control (Automatic white balance/Automatic exposure/Automatic Focus, automatic white balance/automatic exposure/automatic focus).

The encoder 16 is mainly used to compress and encode signal data according to a standard format to facilitate the transmission of video signals.

In the field of video surveillance such as warehouse management, unattended, and safe cities, it is often necessary to shoot images/videos in low-light scenes. Due to insufficient ambient light, it is difficult for the camera to obtain high-quality color images. In this case, it is often necessary to improve the imaging quality of the camera by means of supplementary light. However, the light of the fill light is different from natural light, it is a non-parallel light, so its fill light effect is not as uniform as natural light during the day. As shown in Figure 2, the central part of the fill light area has sufficient light, and the edge of the fill light area has weak light. In order to obtain a properly exposed image, for example, as shown in Figure 3, the innermost circle area has sufficient fill light, so it needs the shortest exposure time, assuming it is ST1; the amount of light in the second inner circle area is only Next to the innermost circle, the required exposure time is assumed to be ST2; and so on, the exposure times required for the other two arc areas are ST3 and ST4 respectively, and further, ST1<ST2<ST3<ST4.

In the existing automatic exposure technology, the same set of exposure parameters are used globally, so the face is mostly in an over-exposed or under-exposed state, which will lead to poor image quality of the face and difficult to recognize. The overall realization process of the existing automatic exposure technology is shown in Figure 4:

In step S41, the camera module collects ambient light, and generally uses some filters to improve the image quality.

Step S42, the image signal processor ISP calculates the brightness of the image, and outputs a statistical value M. In the automatic exposure technology, the common methods of statistical brightness include the average brightness method, the weight average method, and the brightness histogram. The most common method is the average brightness method. The average brightness method is to average the brightness of all pixels of the image, and finally achieve the target brightness by continuously adjusting the exposure parameters. The weighted average method is to set different weights to different areas of the image to calculate the image brightness. The brightness histogram method calculates the image brightness by assigning different weights to the peaks in the histogram.

Step S43: Calculate the absolute value of the brightness statistical value M and the preset target brightness difference, and determine whether it is less than the threshold. If the difference between the two is less than the threshold value, proceed to step S44, otherwise, proceed to step S45.

Step S44, output exposure parameters. The exposure parameters are applied to the image captured by the camera module.

Step S45, recalculate the exposure parameters and apply the new exposure parameters to the camera module, repeat steps S41-S43, until the absolute value of the statistical brightness M and the preset target brightness difference is less than the threshold, and output the final exposure parameters.

The above-mentioned automatic exposure method uses only one exposure parameter globally when shooting objects or faces, and it is difficult to adapt to scenes with uneven environmental brightness. In addition, in order to ensure the smooth change of the picture brightness, the exposure parameter adjustment of the automatic exposure method is slow. When the pedestrian enters the central area of the fill light from the dark area, the face will quickly change from the under-exposed state to the over-exposed state, making it difficult to obtain the normal state. The image in the exposed state greatly reduces the accuracy of face recognition.

Therefore, in order to solve the above-mentioned problem, the embodiment of the present application provides an exposure method in which the exposure parameter periodically changes to improve the image quality of the face in the different brightness areas in the picture, thereby improving the face recognition rate. Use the same camera module to perform automatic exposure and manual exposure at the same time interval, where the parameters of manual exposure are set according to the parameters of automatic exposure, and have the characteristics of periodic changes; then the video stream obtained by the automatic exposure method is used for the video Show that the video stream obtained by the manual exposure method is used for target recognition/analysis. In this application, by alternately performing manual exposure and automatic exposure when capturing images, it can quickly respond to changes in scene brightness and improve the imaging quality of different brightness areas in the picture. Moreover, there are more than two types of manual exposure parameters that change periodically, which can adapt to scenes with uneven brightness distribution at multiple levels, rather than limited to scenes with significant contrast between bright and dark. Therefore, the camera's ability to adapt to scenes with uneven brightness is greatly improved. .

In order to make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the accompanying drawings in the embodiments of the present application.

Figure 5 is the overall solution process of the present invention. Combined with the schematic structural diagram of Figure 6, the method of the present invention includes the following basic steps:

Step S51: The first image signal processing device (ISP1) obtains brightness data of the current shooting scene, and generates automatic exposure parameters based on the brightness data.

As a specific implementation, when step S51 is implemented, the brightness data of the current shooting scene is collected by the camera through the lens module of the environment brightness of the scene to be shot and determined by the image sensor.

Step S52: The second image signal processing device (ISP2) generates manual exposure parameters according to the automatic exposure parameters generated by ISP1. ISP2 can directly retrieve the value of automatic exposure parameters from ISP1, and calculate the corresponding manual exposure parameters accordingly.

Step S53: ISP1 applies the automatic exposure parameter to the camera module, and obtains the odd-numbered video stream (the first video stream) among all the video frames collected by the camera. That is, when the camera module shoots an odd-numbered frame video stream, the exposure parameters (shutter time, aperture size, gain size) adopt the automatic exposure parameter values calculated by ISP1.

Step S54: ISP2 applies the manual exposure parameter to the camera module, and obtains the even-numbered video stream (second video stream) among all the video frames collected by the camera. That is, when the camera module is shooting an even-numbered frame video stream, the exposure parameters (shutter time, aperture size, gain size) should be switched to the manual exposure parameter values calculated by ISP2.

Step S55: The odd-numbered frame video stream is used for video display, and the even-numbered frame video stream is transmitted to the target detection and recognition module for target analysis.

Because the odd-numbered frame video stream adopts the automatic exposure method, its parameters are slowly and continuously changed, so it can be sent to the back end for video display. However, the manual exposure parameters used in the even-numbered frame video stream are not continuously changing, which cannot be adapted by the human eye, and is mainly used for delivery to the analysis module for face recognition or target detection.

It is worth noting that the above-mentioned odd and even frames are only to distinguish between two video streams. In fact, it is also possible that odd frames are collected based on manual exposure parameters, and even frames are collected based on automatic exposure parameters, even every three frames of the video stream. The first two frames are collected based on automatic exposure parameters, and the third frame is collected based on manual exposure parameters, which is not specifically limited in this application. In order to facilitate the introduction of the technical solution of the present invention, all the following embodiments take as an example that the odd-numbered frames of all the video frames captured by the camera are captured with automatic exposure parameters, and the even-numbered frames are captured with manual exposure parameters as examples.

Exemplarily, on the basis of the foregoing embodiment, in step S52, ISP2 generates a manual exposure parameter according to the automatic exposure parameter generated by ISP1, including: for each parameter of the first part of the automatic exposure parameter, a parameter is generated according to the parameter. Group sequences and perform periodic expansion to obtain the parameter expansion sequence as the corresponding parameter in the first part of the manual exposure parameter; keep the second part of the automatic exposure parameter unchanged as the second part of the manual exposure parameter. The above method includes two solutions: one is that the first part of parameters includes shutter time, and the second part of parameters includes aperture size and gain size; the other is that the first part of parameters includes shutter time and gain size, and the second part of parameters includes aperture size. Taking the first solution as an example, as shown in FIG. 7, step S52 specifically includes:

Step S71: ISP2 obtains the automatic exposure parameters obtained by the current ISP1: shutter time ST _AE , aperture size, gain size. ISP1 generates automatic exposure parameters based on the ambient brightness data, and ISP2 can directly obtain the shutter time ST _AE from ISP1.

Step S72: ISP2 generates a set of shutter sequences that change up and down: [K1*ST _AE , K2*ST _AE , K3*ST _AE ,..., KN*ST _AE ]. Among them, K1, K2,...KN are the values manually set by the user according to the situation, such as 1, 3, 7, 10. The generated shutter sequence is [ST _AE , 3*ST _AE , 7*ST _AE , 10*ST _AE ].

Step S73: According to the shutter sequence, automatically generate a shutter period that changes up and down. As shown in FIG. 8, the shutter time of the even-numbered frames is _{periodically changed up and down according to the automatic exposure shutter time ST AE} 1 time, 3 times, 7 times, 10 times, 7 times, 3 times, and 1 time. It is worth noting that sometimes the target analysis module cannot receive all data frames due to the limitation of computing power, and can only process every frame. Therefore, sometimes the parameters of every two even-numbered frames need to be kept the same. For example, the shutter time of the even-numbered frame is 1 times, 1 times, 3 times, 3 times, 7 times, 7 times, 10 times, 10 times _{of the automatic exposure shutter time ST AE.} Times, 7 times, 7 times, 3 times, 3 times, 1 times, 1 times.

Step S74: Apply the generated shutter period to the camera module cyclically to obtain an even-numbered frame video stream. In addition, the aperture size and gain size of the manual exposure parameters are consistent with the aperture size and gain size of the automatic exposure parameters.

Among them, steps S72 and S73 can also be directly combined into one step. That is, when generating a shutter sequence, you can directly generate a shutter period, for example, the shutter sequence is [ST _AE , 3*ST _AE , 7*ST _AE , 10*ST _AE , 7*ST _AE , 3*ST _AE , ST _AE ], Then step S74 is executed to cyclically apply the shutter sequence to the camera module.

It is worth noting that the above steps only use the shutter time in the manual exposure parameter to link with the shutter time in the automatic exposure parameter. In order to obtain a more suitable exposure image, on the basis of linking the shutter time, you can also The exposure parameter of gain size is linked. Exemplarily, the gain of manual exposure can be periodically changed based on the gain of automatic exposure by -2dB, -4dB, -6dB, -8dB, -6dB, -4dB, and -2dB. Assuming that the gain of automatic exposure is A, the sequence of gains is [A-2,A-4,A-6,A-8,A-6,A-4,A-2], and then this extended sequence is looped Applied to camera modules. Furthermore, sometimes the parameters in a period do not necessarily need to be completely symmetrical, and it depends on the specific situation. In addition, because the manual exposure parameters are generated based on the automatic exposure parameters, when the automatic exposure parameters change, the manual exposure parameter sequence at this time is immediately stopped looping, and the manual exposure parameter sequence is recalculated according to the new automatic exposure parameters, and then the The new manual exposure parameter sequence is cyclically applied to the camera module to obtain even-numbered frame data. By alternately performing manual exposure and automatic exposure, the technical solution of the present application can quickly respond to changes in brightness when objects move in the scene, and improve the camera's ability to adapt to scenes containing multiple levels of brightness.

For ease of understanding, the embodiments of the present application apply the technical solution to a scene of mixed traffic between people and vehicles, and show the implementation details of the solution in detail. Assuming that under the lighting conditions of the current shooting scene of the camera, the shutter time required for the license plate is 5ms, the shutter time for light-colored vehicles is 5-10ms, the shutter time for dark-colored vehicles is 10-20ms, and the exposure time for nearby faces 5-10ms, the shutter time required for a distant face is 10-20ms. Similarly, take automatic exposure for odd frames, manual exposure for even frames, and linkage with only the exposure parameter of shutter time as an example. The specific steps are as follows:

Step1: In order to ensure that the license plate and nearby faces are not exposed, assume that the shutter time ST _AE of the odd-numbered frame automatic exposure road is 5ms, that is, ST _AE = 5ms;

Step2: ISP2 obtains the iris, gain and shutter time ST _{AE of the} automatic exposure path, the iris and gain of the manual exposure are consistent with the iris and gain of the automatic exposure, and the shutter time sequence can be designed as [K1*ST _AE ,K2*ST _AE , K3*ST _AE ,K4*ST _AE ], where K1, K2, K3, K4 are 1, 2, 3, 4 respectively;

Step3: The shutter sequence, generates a shutter cycle period of even frames, is able to satisfy both the needs of the exposure plate, the body and the face distance, the shutter time even frame may _{_{_{ST AE, 2ST AE, 3ST AE}}} , 4ST AE, 3ST _AE , 2ST _AE , ST _AE, the lifting and lowering modes change cyclically;

Step4: If the shutter time of odd-numbered frames changes due to changes in ambient lighting, assuming ST _{AE is} updated to 4ms, the shutter time of even-numbered frames in the next cycle should also be updated synchronously, that is, recalculate the even number of the next cycle according to Step2 and Step3 Frame shutter time. Then, ISP2 applies the new manual exposure parameters to the camera module and reacquires even-numbered frame data.

The method provided by the present invention can take into account scenes containing multiple levels of brightness, and is not limited to scenes with significant contrast between bright and dark. This method greatly improves the probability of obtaining images with appropriate exposure, and significantly improves target analysis such as face analysis or license plate recognition. The accuracy of this ensures the stability of the social environment.

In another embodiment, in order to further improve the imaging effect of manual exposure, an odd-even frame fusion module can be added before ISP2. The even-numbered frame video streams are fused, and then transmitted to ISP2 for subsequent target analysis, further improving the accuracy of the analysis.

Suppose the image depth taken by the camera (image depth refers to the number of bits used to store each pixel, and is also used to measure the color resolution of the image. Image depth determines the number of colors that each pixel of a color image may have, or determines the grayscale The number of gray levels that each pixel of the image may have. It determines the maximum number of colors that can appear in a color image, or the maximum gray level in a grayscale image) is 8bit, then the range of the pixel value of each pixel It is 0-255. Exemplarily, taking an image of an odd frame as a reference, each pixel in the captured image has a pixel value in the range of 0-255. Then the dots with pixel value lower than 50 adopt odd frame data, the dots with pixel value higher than 150 adopt even frame data, and the dots with pixel value between 50-150 adopt weighted average data of odd and even frames. So far, each pair of odd and even frames The fusion is completed according to the shooting sequence, and then transmitted to ISP2 for subsequent video analysis. It is worth noting that the above fusion scheme is only a schematic example. When other collection methods are used, the fusion method needs to be adjusted. For example: when the alternate method of capturing images is automatic exposure for the first two frames of every three frames and manual exposure for the third frame, then every three frames in the shooting sequence are grouped together, and the first automatic exposure image frame in each group is taken as the reference Image, according to the above method, the last two frames of images are merged into the reference image. Then, the multiple sets of video frames are sequentially merged to obtain a merged video stream. In short, the video fusion method needs to match the alternate acquisition method, and the specific fusion rules can be determined based on user experience or actual conditions.

The video capture method based on exposure control provided in this application divides the video stream into manual exposure and automatic exposure, which can not only meet the needs of users to watch videos in real time, but also overcome the disadvantage of slow response to scene brightness changes in existing exposure methods. Moreover, the manual exposure parameters cover a wider and more layered range, which can adapt to multi-level scenes with gradual light and dark, ensuring that each area has a better exposure frame, thereby greatly improving the accuracy of video analysis.

The foregoing describes in detail the video capture method based on exposure control provided by the embodiments of the present application with reference to FIGS. 1 to 9. The following describes the video capture device of the present application with reference to FIG. 10. For details, please refer to the method embodiment of this application.

FIG. 10 is a video capture device provided by an embodiment of the application. The device 100 may include a camera module 101, a first image signal processing module 102, and a second image signal processing module 103.

The first image signal processing module 102 is configured to determine the automatic exposure parameters of the camera module according to the environmental data.

The second image signal processing module 103 is configured to obtain manual exposure parameters according to the automatic exposure parameters.

The camera module 101 includes a lens 1011 and an image sensor 1012 for collecting video frames based on the automatic exposure parameters to form a first video stream; and also for collecting video frames based on the manual exposure parameters to form a second video stream.

Wherein, the video frames constituting the first video stream and the video frames constituting the second video stream are alternately collected by the camera module. Optionally, the automatic exposure parameters include: shutter time, aperture size, and gain size. The second image signal processing module 103 is specifically configured to generate a set of sequences for each parameter of the first part of the automatic exposure parameters. And perform periodic expansion to obtain the parameter expansion sequence as the corresponding parameter in the first part of the manual exposure parameter; keep the second part of the automatic exposure parameter unchanged as the second part of the manual exposure parameter.

Optionally, the first part of the automatic exposure parameters includes the shutter time, and the second image signal processing module 103 is specifically configured to: obtain the shutter time in the automatic exposure parameters; generate a set of shutter time series according to the shutter time, and repeat all The extended shutter time sequence obtained from the shutter time sequence is used as the shutter time in the manual exposure parameter.

Optionally, the first part of the auto-exposure parameter further includes a gain size, and the second image signal processing module 103 is further used to: obtain the gain size in the auto-exposure parameter; generate a set of gain size sequence according to the gain size, and repeat the loop The gain size expansion sequence obtained from the gain size sequence is used as the gain size in the manual exposure parameter.

Optionally, the shutter time in the manual exposure parameter and the gain in the manual exposure parameter change periodically, and include rising and falling changes in one cycle.

Optionally, the second part of the automatic exposure parameters includes: aperture size.

Optionally, the first video stream is used for video display, and the second video stream is used for video target detection, identification and/or analysis.

Optionally, the device further includes a fusion module, configured to perform fusion processing on the video frames constituting the first video stream and the video frames constituting the second video stream to obtain a third video stream.

As another possible embodiment, the present application also provides a video surveillance system, including: the video capture device provided in the foregoing device embodiment, which is used to obtain the first video stream according to the automatic exposure method, and to obtain the first video stream according to the manual exposure method. The second video stream, the manual exposure parameters and the automatic exposure parameters are partially linked; the video display module is used to display the first video stream; the video analysis module is used to analyze the second video stream and/or the third video stream Video streaming.

It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions, and may be fully or partially integrated into a physical entity during actual implementation, or may be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be implemented in the form of hardware; some modules can be implemented in the form of calling software by processing elements, and some of the modules can be implemented in the form of hardware.

The present invention also provides an embodiment of a computer program product. The computer program product contains computer-readable code instructions. When the computer-readable code instructions are executed by the computer, the computer can configure the video capture device so that the device can execute the method. Any of the methods described in the various possible implementation manners of the embodiment.

The present invention also provides an embodiment of a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium contains computer program code instructions, which can be implemented when the computer-readable code instructions are executed by a computer. The configuration of the video capture device by the computer enables the device to execute any of the methods described in the various possible implementation manners of the method embodiments. The non-transitory computer-readable storage medium includes one or more of the following options: Read-Only Memory (ROM), Programmable ROM (Programmable ROM), Erasable PROM (Erasable PROM, EPROM) ), flash memory (Flash Memory), electrical EPROM (Electrically EPROM, EEPROM) and hard drive (Hard Drive).

The above-mentioned embodiments are only used to illustrate the technical solutions of the present invention and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, it is still possible to modify the technical solutions described in the foregoing embodiments. , Or equivalently replace some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

A video acquisition method based on exposure control, characterized in that the method includes:

Determine the automatic exposure parameters of the camera module according to the environmental data;

Obtaining manual exposure parameters according to the automatic exposure parameters;

Based on the automatic exposure parameters, the video frames collected by the camera module constitute a first video stream;

Based on the manual exposure parameters, the video frames collected by the camera module constitute a second video stream;

The video frames constituting the first video stream and the video frames constituting the second video stream are alternately collected by the camera module.
The method according to claim 1, wherein the automatic exposure parameters include: shutter time, aperture size, and gain, and the obtaining manual exposure parameters according to the automatic exposure parameters includes:

For each parameter of the first partial parameter of the automatic exposure parameter, a set of sequences is generated according to the parameter and periodically expanded to obtain the parameter expansion sequence, which is used as a corresponding parameter in the first partial parameter of the manual exposure parameter;

The second part of the parameter of the automatic exposure parameter is kept unchanged as the second part of the parameter of the manual exposure parameter.
The method according to claim 2, wherein the first part of the automatic exposure parameter includes a shutter time, and for each parameter of the first part of the automatic exposure parameter, a set of sequences is generated according to the parameter And perform periodic expansion to obtain the parameter expansion sequence, and the corresponding parameters in the first part of the parameters as the manual exposure parameters include:

Acquiring the shutter time in the automatic exposure parameter;

Generating a set of shutter time sequences according to the shutter time;

The shutter time expansion sequence obtained by cyclically repeating the shutter time sequence is used as the shutter time in the manual exposure parameter.
The method according to claim 3, wherein the first part of the parameters of the automatic exposure parameters further includes a gain size, and for each parameter of the first part of the parameters of the automatic exposure parameters, a set of parameters is generated according to the parameters. Sequence and perform periodic expansion to obtain the parameter expansion sequence, and the corresponding parameter in the first part of the parameter as the manual exposure parameter further includes:

Acquiring the gain size in the automatic exposure parameter;

Generating a set of gain size sequences according to the gain size;

The gain size expansion sequence obtained by cyclically repeating the gain size sequence is used as the gain size in the manual exposure parameter.
The method according to claim 3 or 4, wherein the shutter time in the manual exposure parameter and/or the gain in the manual exposure parameter changes periodically, and includes rising and falling within one cycle Variety.
The method according to any one of claims 2-5, wherein the second part of the automatic exposure parameters includes: aperture size.
The method according to any one of claims 1-6, wherein the first video stream is used for video display, and the second video stream is used for video analysis.
The method according to any one of claims 1-7, wherein the method further comprises:

Perform fusion processing on the video frames constituting the first video stream and the video frames constituting the second video stream to obtain a third video stream.
A video capture device, characterized in that the device includes a camera module, a first image signal processing module, and a second image signal processing module;

The first image signal processing module is configured to determine the automatic exposure parameters of the camera module according to environmental data;

The second image signal processing module is configured to obtain manual exposure parameters according to the automatic exposure parameters;

The camera module includes a lens and an image sensor, and is configured to collect video frames to form a first video stream based on the automatic exposure parameters, and to collect video frames to form a second video stream based on the manual exposure parameters.

Wherein, the video frames constituting the first video stream and the video frames constituting the second video stream are alternately collected by the camera module.
The device according to claim 9, wherein the automatic exposure parameters include: shutter time, aperture size, and gain size, and the second image signal processing module is used for:

For each parameter of the first part of the parameters of the automatic exposure parameters, generate a set of sequences according to the parameters and perform periodic expansion to obtain the parameter expansion sequence, which is used as the corresponding parameter in the first part of the parameters of the manual exposure parameters;

Keep the second part of the automatic exposure parameter unchanged, and serve as the second part of the manual exposure parameter.
The device according to claim 10, wherein the first part of the automatic exposure parameter includes a shutter time, and the second image signal processing module is used for:

Acquiring the shutter time in the automatic exposure parameter;

Generating a set of shutter time sequences according to the shutter time;

The shutter time expansion sequence obtained by cyclically repeating the shutter time sequence is used as the shutter time in the manual exposure parameter.
The apparatus according to claim 11, wherein the first part of the automatic exposure parameter further includes a gain size, and the second image signal processing module is further configured to:

Acquiring the gain size in the automatic exposure parameter;

Generating a set of gain size sequences according to the gain size;

The gain size expansion sequence obtained by cyclically repeating the gain size sequence is used as the gain size in the manual exposure parameter.
The device according to claim 11 or 12, wherein the shutter time in the manual exposure parameter and the gain in the manual exposure parameter change periodically, and include rising and falling changes in one cycle.
The device according to any one of claims 10-13, wherein the second part of the automatic exposure parameters includes: aperture size.
The device according to any one of claims 9-14, wherein the first video stream is used for video display, and the second video stream is used for video analysis.
The device according to any one of claims 9-15, wherein the device further comprises a fusion module, configured to perform processing between the video frames constituting the first video stream and the video frames constituting the second video stream. Fusion process to obtain the third video stream.
A video surveillance system, characterized in that it comprises:

The video capture device according to any one of claims 9-16;

A video display module for displaying the first video stream;

The video analysis module is used to analyze the second video stream and/or the third video stream.