CN118175433A - ISP automatic tuning method based on different scenes in same video picture - Google Patents

Abstract

The invention belongs to the technical field of ISP tuning of cameras, and particularly relates to a method for automatically tuning ISPs based on different scenes in the same video picture, which comprises the steps of firstly creating a database in a camera, setting a plurality of scenes in the database, wherein each scene comprises a plurality of characteristics, and configuring one set of ISP parameters for each scene; then capturing a video picture by a camera, and extracting features of the current scene of the video picture; calling the database, comparing the extracted features with features in different scenes in the database, and judging the type of the current scene through the feature comparison result; then, according to the current scene type, ISP parameters of the corresponding scene type in the database are called; finally, the adjustment of the camera image and the processing effect is realized by calling ISP parameters matched with the current scene, and the adjusted image and video picture are output. The automatic tuning of ISP parameters of the camera under different application scenes is realized, and the adaptability of the camera is effectively enhanced.

Description

ISP automatic tuning method based on different scenes in same video picture

Technical Field

The invention belongs to the technical field of ISP tuning of cameras, and particularly relates to a method for automatically tuning ISPs of different scenes in the same video picture.

Background

With the development of the age and the progress of scientific technology, the security monitoring industry is deeper and deeper into the field of view of people, the requirements for video monitoring and video application are continuously increasing, and the application scene used for a camera is diversified and complicated. In the field of video surveillance, the image quality of the camera is critical. Whereas ISP (IMAGE SIGNAL Processing) parameters of the camera have an important influence on the image quality. The use scene of the camera comprises different scenes such as face snapshot, license plate snapshot, human shape detection, vehicle detection, non-motor vehicle, flame detection and the like. ISP is the same set of parameters under different use scenes, and when facing objects with different sizes, depth of field, different colors and other characteristics, the traditional security camera often presents unsatisfactory effects, and even under some scenes, overexposure, defocusing and other conditions can occur.

How to make the camera show the best image effect under the different application scenes, the current technical method mainly comprises the following two steps:

one is to adjust the general ISP parameters according to the standardized scene using ISP parameters of the fixed standardized scene. This method cannot be adjusted according to the specific use of the scene. This results in less than optimal image quality of the camera in complex and variable surveillance scenarios.

Yet another is to manually adjust ISP parameters, i.e. by manual intervention, to specific scenarios. The method can achieve better image effect, but is labor-wasting and low in efficiency.

Therefore, how to solve the technical problem that the camera in the prior art cannot automatically adjust the ISP parameters in different application scenes is a technical problem that needs to be solved at present.

Disclosure of Invention

The invention aims to provide a method for automatically adjusting and optimizing ISPs based on different scenes in the same video picture, which is used for solving the technical problem.

In order to solve the technical problems, the invention adopts the following technical scheme:

A method for automatically adjusting and optimizing ISP based on different scenes in the same video picture comprises the following steps:

s1: creating a database in a camera, and setting a plurality of scenes in the database, wherein each scene comprises a plurality of characteristics, and configuring one set of ISP parameters for each scene;

s2: capturing a video picture by a camera, and extracting features of the current scene of the video picture;

s3: invoking the database, comparing the extracted features with features in different scenes in the database, and judging the type of the current scene through the feature comparison result;

S4: invoking ISP parameters of corresponding scene types in the database according to the current scene types;

S5: and (3) adjusting the image and the processing effect of the camera by calling ISP parameters matched with the current scene, and outputting the adjusted image and video picture.

Preferably, in step S1, the following specific processes are included:

s11: creating a plurality of test scenes, extracting the characteristics of each test scene, and storing each scene and the corresponding characteristics thereof in the database;

s12: in each test scene, each parameter in ISP parameters is adjusted, and a set of ISP parameters which are adjusted to the optimal picture in the corresponding test scene is obtained;

s13: and respectively storing the multiple sets of ISP parameters in the adjusted different test scenes in the database, and adding corresponding scene labels.

Preferably, each set of ISP parameters includes exposure time, gain, color correction, sharpening, defogging, dynamic contrast, gamma, digital wide dynamic, dead pixel removal, purple fringing, 2D/3D noise reduction, saturation, demosaicing, lens shading, black level, and digital anti-shake.

Preferably, the plurality of test scenarios includes: a portrait scene, a face scene, a vehicle scene, and a flame scene.

Preferably, in step S4, the following specific procedures are included:

S41: the video camera acquires video stream information acquired by the image sensor, transmits the video stream information through an MIPI bus, and dynamically loads an IQ file and a corresponding algorithm model;

S42: generating YUV data by automatically matching ISP parameters of corresponding algorithm scenes;

s43: the YUV data generates video stream information through a VENC channel, and at the same time generates AI algorithm structured information through processing of a corresponding AI algorithm model;

s44: the video stream information and the AI algorithm structured information are pushed and transmitted through a network.

Preferably, the algorithm model in step S41 includes a person AI algorithm, a face AI algorithm, a vehicle AI algorithm, and a flame AI algorithm.

Preferably, the manner of dynamically loading the IQ file and the corresponding algorithm model in step S41 includes a timing switching manner, an algorithm identification adaptive switching manner, and a manual switching manner.

Preferably, the timing switching mode is as follows: and presetting a time period in the camera, switching the camera to an algorithm model corresponding to the current time period in the corresponding time period, and loading ISP parameters of the corresponding algorithm model.

Preferably, the algorithm identifies the self-adaptive switching mode to automatically extract the characteristic value of the picture by the camera, matches the characteristic value in the database according to different characteristic values when the extracted characteristic value is the figure, the face, the vehicle or the flame, so as to determine the current scene, and automatically invokes the ISP parameters of the corresponding scene according to the current scene.

Preferably, the algorithm identifying the adaptive switching mode comprises the following specific steps:

s411: designating an ROI area on a video picture;

s412: inputting video stream frame data;

s413: positioning the position of a target in a video picture and corresponding confidence coefficient, using rectangular frame reality, and outputting detection results of a human face, a vehicle and flame;

S414: loading a corresponding algorithm model according to the detection result;

If only one type of target exists in the detection result display screen in S414, loading a corresponding algorithm model according to the type of the target scene; if multiple types of targets exist in the detection result display picture, calculating the duty ratio of the number of each type of targets in the ROI area, loading the algorithm model corresponding to the type of target scene with the largest number, and calling ISP parameters of the type of target scene.

The beneficial effects of the invention include:

The invention provides a method for automatically adjusting and optimizing ISPs based on different scenes in the same video picture, which comprises the steps of firstly creating a database in a camera, setting a plurality of scenes in the database, wherein each scene comprises a plurality of characteristics, and configuring one set of ISP parameters for each scene; then capturing a video picture by a camera, and extracting features of the current scene of the video picture; calling the database, comparing the extracted features with features in different scenes in the database, and judging the type of the current scene through the feature comparison result; then, according to the current scene type, ISP parameters of the corresponding scene type in the database are called; finally, the adjustment of the camera image and the processing effect is realized by calling ISP parameters matched with the current scene, and the adjusted image and video picture are output. The automatic tuning of ISP parameters of the camera under different application scenes is realized, and the adaptability of the camera is effectively enhanced.

Drawings

Fig. 1 is a flow chart of the method for automatically tuning ISP based on different scenes in the same video frame according to the present invention.

Fig. 2 is a flow chart of the invention for loading corresponding ISP parameters to realize camera tuning.

FIG. 3 is a flow chart of three ways of dynamically loading IQ files and algorithm models according to the present invention.

Detailed Description

The invention is further described in detail below with reference to fig. 1 to 3:

referring to fig. 1, the method for automatically tuning ISP based on different scenes in the same video picture comprises the following steps:

s1: creating a database in a camera, and setting a plurality of scenes in the database, wherein each scene comprises a plurality of characteristics, and configuring a set of characteristics and a set of ISP parameters for each scene;

s2: capturing a video picture by a camera, and extracting features of the current scene of the video picture;

s3: invoking the database, comparing the extracted features with features in different scenes in the database, and judging the type of the current scene through the feature comparison result;

S4: invoking ISP parameters of corresponding scene types in the database according to the current scene types;

S5: and (3) adjusting the image and the processing effect of the camera by calling ISP parameters matched with the current scene, and outputting the adjusted image and video picture.

The files stored in the database are IQ files, the IQ files in the database are loaded to carry out data calling, each scene in the database and the characteristics of each scene are extracted from the real scenes in different time periods for multiple times, and the characteristic data of each scene are stored in the database. The ISP parameters corresponding to each scene are adjusted for a plurality of times in different time periods, so that the video picture output by the camera in the scene is relatively optimal, and a plurality of sets of ISP parameters are added into the scene label and stored in the database.

When the video camera works, the video picture of the current scene is captured first, then the characteristics in the video picture of the current scene are extracted, and then the characteristic data of each scene in the database is called to compare with the characteristics of the picture of the current scene, and the category of the current scene is judged. And calling a set of ISP parameters corresponding to the current scene category in the database again based on the current scene judgment result, using the set of ISP parameters for adjusting the camera image and the processing effect, and outputting the adjusted video picture.

Through the automatic ISP parameter tuning process, the automatic tuning of the ISP parameters of the camera under different application scenes or the automatic switching of the ISP parameters according to scene transformation is realized, and the adaptability of the camera is effectively enhanced.

In step S1, the following specific procedures are included:

s11: creating a plurality of test scenes, extracting the characteristics of each test scene, and storing each scene and the corresponding characteristics thereof in the database;

s12: in each test scene, each parameter in ISP parameters is adjusted, and a set of ISP parameters which are adjusted to the optimal picture in the corresponding test scene is obtained;

s13: and respectively storing the multiple sets of ISP parameters in the adjusted different test scenes in the database, and adding corresponding scene labels.

Wherein, each set of ISP parameters can comprise parameters such as exposure time, gain, color correction, sharpening, defogging, dynamic contrast, gamma, digital wide dynamic, dead pixel removal, purple fringing, 2D/3D noise reduction, saturation, brightness, demosaicing, lens shading, black level, digital anti-shake and the like. The plurality of test scenes can comprise a portrait scene, a face scene, a vehicle scene, a flame scene and the like, wherein the flame scene is a scene which is strictly monitored for fire.

ISP parameters in the database are combined into files by different formats such as. Ini or. Bin. Corresponding scene labels, such as face scene labels, license plate scene labels and flame scene labels, are added on the file. The field Jing Biao signs the bin file, loads the file through the MI_ISP_ ApiCmdLoadBinFile function interface, loads the file to the main chip of the camera, and the main chip sets a sensor register through loading ISP parameters, processes various ISP parameters on the received image data and then outputs an image stream meeting the requirements.

For example, when a camera acquires a video picture, the license plate on the vehicle is expected to be clearly and correctly grabbed, and the license plate is relatively close to the vehicle lamp, so that strong light of the vehicle lamp can interfere with imaging of the license plate position, strong light irradiation and image ghosting of the vehicle lamp need to be solved when the camera shoots, and therefore the light receiving parameters need to be enhanced, and the exposure shutter is reduced.

In the face scene, the face needs to be captured, the complete and clear face image is required to be captured, and the face imaging is required to solve the problem that the face is overexposed and the face details are highlighted, so that the imaging picture in the face scene needs to be adjusted by adjusting ISP parameters such as target brightness, exposure shutter, brightness, sharpening and the like.

In the above scheme, the data stored in the database includes two aspects of data, namely, the picture characteristic data under each scene; another aspect is corresponding ISP parameter data for each scenario. Judging the type of the current scene by calling the picture feature data of the scene, and calling corresponding ISP parameters based on the current scene type to realize the automatic tuning of the ISP parameters of the camera picture.

Referring to fig. 2, in step S4, the following specific process is included:

S41: the video camera acquires video stream information acquired by the image sensor, transmits the video stream information through an MIPI bus, and dynamically loads an IQ file and a corresponding algorithm model;

S42: generating YUV data by automatically matching ISP parameters of corresponding algorithm scenes;

s43: the YUV data generates video stream information through a VENC channel, and at the same time generates AI algorithm structured information through processing of a corresponding AI algorithm model;

s44: the video stream information and the AI algorithm structured information are pushed and transmitted through a network.

The algorithm model in step S41 includes a person AI algorithm, a face AI algorithm, a vehicle AI algorithm, and a flame AI algorithm.

Referring to fig. 3, the manners of dynamically loading IQ files and corresponding algorithm models in step S41 include a timing switching manner, an algorithm identification adaptive switching manner, and a manual switching manner.

For example, when the scene is judged to be a face scene according to the characteristics of the current scene picture, video stream information input by the camera from the image sensor is transmitted through the MIPI bus. And generating YUV data by automatically matching ISP parameters of the human face AI algorithm scene. The video stream information and the AI algorithm structured information are generated through the processing of the VENC channel and the AI algorithm of the human face respectively, and the streaming pushing and the transmission are carried out through a network.

The timing switching mode is as follows: and presetting a time period in the camera, switching the camera to an algorithm model corresponding to the current time period in the corresponding time period, and loading ISP parameters of the corresponding algorithm model. For example, under the condition of large people flow in daytime, the human face scene can be set, and a human face AI algorithm model is selected. The vehicle scene or the flame scene can be switched to at night, and accurate monitoring of the monitoring picture is realized.

The algorithm identifies the self-adaptive switching mode to automatically extract the characteristic value of the picture for the camera, matches the characteristic value in the database according to different characteristic values when the extracted characteristic value is the figure, the face, the vehicle or the flame, so as to determine the current scene, and automatically invokes ISP parameters of the corresponding scene according to the current scene.

The algorithm identification self-adaptive switching mode comprises the following specific steps:

s411: designating an ROI area on a video picture;

s412: inputting video stream frame data;

s413: positioning the position of a target in a video picture and corresponding confidence coefficient, using rectangular frame reality, and outputting detection results of a human face, a vehicle and flame;

S414: loading a corresponding algorithm model according to the detection result;

The target characteristics in the picture, such as face characteristics, human shape characteristics, vehicle characteristics, flame characteristics and the like, are automatically detected, the category of the scene is automatically judged, the corresponding algorithm with the scene is automatically switched to call the ISP parameters of the scene, a special set of ISP parameters are automatically called according to different detection targets, and the adaptability of the camera is effectively improved.

If only one type of object exists in the detection result display screen in S414, a corresponding algorithm model is loaded according to the object scene type. For example, when only a face target appears in a video picture, the video picture is switched to a face scene, ISP parameters in the face scene are called, and clear imaging of the face is realized. When only a vehicle target appears in the video picture, switching to a vehicle scene, and calling ISP parameters in the vehicle scene to realize clear imaging of the vehicle and the license plate.

Many times, however, there may be more than one type of object in the video captured by the camera, such as the simultaneous presence of a face and a vehicle. If multiple types of targets exist in the detection result display picture, calculating the duty ratio of the number of each type of targets in the ROI area, loading the algorithm model corresponding to the type of target scene with the largest number, and calling ISP parameters of the type of target scene.

In summary, according to the method for automatically tuning ISP based on different scenes in the same video picture provided by the invention, firstly, a database is created in a camera, and a plurality of scenes are set in the database, each scene comprises a plurality of characteristics, and a set of ISP parameters is configured for each scene; then capturing a video picture by a camera, and extracting features of the current scene of the video picture; calling the database, comparing the extracted features with features in different scenes in the database, and judging the type of the current scene through the feature comparison result; then, according to the current scene type, ISP parameters of the corresponding scene type in the database are called; finally, the adjustment of the camera image and the processing effect is realized by calling ISP parameters matched with the current scene, and the adjusted image and video picture are output. The automatic tuning of ISP parameters of the camera under different application scenes is realized, and the adaptability of the camera is effectively enhanced.

Claims (10)

1. The method for automatically adjusting and optimizing ISP based on different scenes in the same video picture is characterized by comprising the following steps:

S1: creating a database in a camera, setting a plurality of scenes in the database, and configuring one set of ISP parameters for each scene;

s2: capturing a video picture by a camera, and extracting features of the current scene of the video picture;

s3: invoking the database, comparing the extracted features with features in different scenes in the database, and judging the type of the current scene through the feature comparison result;

S4: invoking ISP parameters of corresponding scene types in the database according to the current scene types;

S5: and (3) adjusting the image and the processing effect of the camera by calling ISP parameters matched with the current scene, and outputting the adjusted image and video picture.

2. The method of automatic tuning based on different scenes ISP in the same video frame according to claim 1, wherein in step S1, the following specific procedures are included:

s11: creating a plurality of test scenes, extracting the characteristics of each test scene, and storing each scene and the corresponding characteristics thereof in the database;

s12: in each test scene, each parameter in ISP parameters is adjusted, and a set of ISP parameters which are adjusted to the optimal picture in the corresponding test scene is obtained;

s13: and respectively storing the multiple sets of ISP parameters in the adjusted different test scenes in the database, and adding corresponding scene labels.

3. The method of claim 2, wherein each set of ISP parameters includes exposure time, gain, color correction, sharpening, defogging, dynamic contrast, gamma, digital wide dynamics, dead point removal, purple fringing, 2D/3D noise reduction, saturation, demosaicing, lens shading, black level, and digital anti-shake.

4. The method of automatic tuning based on different scenes ISP in the same video frame according to claim 2, wherein the plurality of test scenes comprises: a portrait scene, a face scene, a vehicle scene, and a flame scene.

5. The method of automatic tuning based on different scenes ISP in the same video frame according to claim 1, wherein in step S4, the following specific procedures are included:

S41: the video camera acquires video stream information acquired by the image sensor, transmits the video stream information through an MIPI bus, and dynamically loads an IQ file and a corresponding algorithm model;

S42: generating YUV data by automatically matching ISP parameters of corresponding algorithm scenes;

s43: the YUV data generates video stream information through a VENC channel, and at the same time generates AI algorithm structured information through processing of a corresponding AI algorithm model;

s44: the video stream information and the AI algorithm structured information are pushed and transmitted through a network.

6. The method according to claim 5, wherein the algorithm model in step S41 includes a portrait AI algorithm, a face AI algorithm, a vehicle AI algorithm, and a flame AI algorithm.

7. The method according to claim 6, wherein the means for dynamically loading IQ files and corresponding algorithm models in step S41 comprises a timing switch mode, an algorithm identification adaptive switch mode, and a manual switch mode.

8. The method for automatically tuning ISP based on different scenes in the same video frame according to claim 7, wherein the timing switching mode is: and presetting a time period in the camera, switching the camera to an algorithm model corresponding to the current time period in the corresponding time period, and loading ISP parameters of the corresponding algorithm model.

9. The method for automatically tuning ISP based on different scenes in the same video picture according to claim 7, wherein the algorithm recognizes the adaptive switching mode to automatically extract the characteristic value of the picture for the camera, matches the extracted characteristic value with the characteristic value in the database according to different characteristic values when the extracted characteristic value is portrait, face, vehicle or flame, so as to determine the current scene, and automatically invokes the ISP parameters of the corresponding scene according to the current scene.

10. The method for automatically tuning ISP based on different scenes in the same video frame according to claim 7, wherein the algorithm identifying the adaptive switching mode comprises the following specific steps:

s411: designating an ROI area on a video picture;

s412: inputting video stream frame data;

s413: positioning the position of a target in a video picture and corresponding confidence coefficient, using rectangular frame reality, and outputting detection results of a human face, a vehicle and flame;

S414: loading a corresponding algorithm model according to the detection result;

If only one type of target exists in the detection result display screen in S414, loading a corresponding algorithm model according to the type of the target scene; if multiple types of targets exist in the detection result display picture, calculating the duty ratio of the number of each type of targets in the ROI area, loading the algorithm model corresponding to the type of target scene with the largest number, and calling ISP parameters of the type of target scene.