WO2021102893A1 - Method and apparatus for video anti-shaking optimization and electronic device - Google Patents

Abstract

A method for video anti-shaking optimization, comprising: acquiring a current image frame, current frame gyroscope information, and a processed image frame, the current image frame and the processed image frame being adjacent in the time domain; using the current frame gyroscope information to perform anti-shaking processing on the current image frame to generate a stabilized current image frame; and performing fusion filtering on the stabilized current image frame and the processed image frame to obtain an optimized video image corresponding to the current image frame.

Description

Video anti-shake optimization processing method and device, and electronic equipment Technical field

This application relates to the field of computer technology, and in particular to a video anti-shake optimization processing method and device, electronic equipment, and computer-readable storage media.

Background technique

With the development of computer technology, a variety of electronic devices have emerged. The shooting function of electronic equipment provides convenience for people to shoot anytime and anywhere. The camera module of electronic equipment usually adopts a smaller size sensor and lens group. With the increase in resolution, the pixel size of the sensor is greatly reduced than before, but this will reduce the signal-to-noise ratio of the output video.

In the related art, in order to ensure the image quality of the output video, a time-domain filtering process is usually first considered. However, the time-domain filtering processing requires a large amount of calculation and a longer calculation time, which leads to a decrease in the stability of the output video.

Summary of the invention

According to various embodiments disclosed in the present application, a video anti-shake optimization processing method, electronic equipment, and computer-readable storage medium are provided.

A video anti-shake optimization processing method, including:

Acquiring a current frame image, current frame gyroscope information, and processed frame image; the current frame image and the processed frame image are adjacent in the time domain;

Using the current frame gyroscope information to perform anti-shake processing on the current frame image to generate a stabilized current frame image;

Performing fusion filtering processing on the stabilized current frame image and the processed frame image to obtain an optimized processed video image corresponding to the current frame image.

A video anti-shake optimization processing device, including:

The acquisition module is used to acquire the current frame image, the current frame gyro information, and the processed frame image; the current frame image and the processed frame image are adjacent in the time domain;

An anti-shake module, configured to perform anti-shake processing on the current frame image by using the current frame gyro information to generate a stabilized current frame image; and

The filtering module is configured to perform fusion filtering processing on the stabilized current frame image and the processed frame image to obtain an optimized processed video image corresponding to the current frame image.

An electronic device, including a memory and one or more processors. The memory stores computer-readable instructions. When the computer-readable instructions are executed by the one or more processors, the one or more Each processor performs the following steps:

Acquiring a current frame image, current frame gyroscope information, and processed frame image; the current frame image and the processed frame image are adjacent in the time domain;

Using the current frame gyroscope information to perform anti-shake processing on the current frame image to generate a stabilized current frame image;

Performing fusion filtering processing on the stabilized current frame image and the processed frame image to obtain an optimized processed video image corresponding to the current frame image.

One or more non-volatile computer-readable storage media containing computer-executable instructions, when the computer-executable instructions are executed by one or more processors, cause the processors to perform the following steps:

Acquiring a current frame image, current frame gyroscope information, and processed frame image; the current frame image and the processed frame image are adjacent in the time domain;

Using the current frame gyroscope information to perform anti-shake processing on the current frame image to generate a stabilized current frame image;

Performing fusion filtering processing on the stabilized current frame image and the processed frame image to obtain an optimized processed video image corresponding to the current frame image.

The details of one or more embodiments of the present application are set forth in the following drawings and description. Other features and advantages of this application will become apparent from the description, drawings and claims.

Description of the drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

FIG. 1 is an application environment diagram of a video anti-shake optimization processing method in an embodiment;

Figure 2 is a flowchart of a video anti-shake optimization processing method in an embodiment;

Fig. 3 is a flowchart of a video anti-shake optimization processing method in another embodiment;

4 is a flow chart of performing fusion filtering processing on the stabilized current frame image and the processed frame image by using the initialization registration information in an embodiment;

Fig. 5 is a flow chart of performing fusion filtering processing on the current frame image after the global alignment and the processed frame image after the global alignment in an embodiment;

FIG. 6 is a structural block diagram of a video anti-shake optimization processing device in an embodiment;

FIG. 7 is a structural block diagram of a video anti-shake optimization processing device in another embodiment;

FIG. 8 is a schematic diagram of the internal structure of an electronic device in an embodiment;

Fig. 9 is a schematic diagram of an image processing circuit in an embodiment.

Detailed ways

In order to make the purpose, technical solutions, and advantages of this application clearer and clearer, the following further describes the application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, and are not used to limit the present application.

Fig. 1 is a schematic diagram of an application environment of a video anti-shake optimization processing method in an embodiment. As shown in FIG. 1, the application environment includes an electronic device 100. The electronic device 100 includes a camera 102, a sensor 104, and a processor 106. The electronic device 100 can perform video shooting through the camera 102. The sensor 104 may be a gyro sensor. During the shooting process, the gyroscope sensor collects the gyroscope information corresponding to each frame of video image. The processor 106 may optimize the video image in the manner provided in this application. The optimized video image can be called a processed frame image. The processor 106 may use adjacent single frame or multiple frames of processed frame images and current frame gyroscope information in the time domain to optimize the current frame image. The processor 106 uses the current frame gyroscope information to perform anti-shake processing on the current frame image to generate a stabilized current frame image. The processor 106 then performs fusion filtering processing on the stabilized current frame image and the processed frame image to obtain an optimized processed video image corresponding to the current frame image. Since the anti-shake processing is pre-processed during the optimization process, the jitter in the current frame image can be eliminated in advance. By performing fusion filtering processing on the stabilized current frame image and processed frame image, the filtering processing efficiency can be effectively improved, and the calculation time of fusion filtering can be saved, thereby effectively improving the stability of the output video.

In one embodiment, as shown in FIG. 2, a video anti-shake optimization processing method is provided. The method is described by taking the electronic device in FIG. 1 as an example, and specifically includes the following steps:

Step 202: Obtain a current frame image, current frame gyroscope information, and processed frame image, where the current frame image and the processed frame image are adjacent in the time domain.

The electronic device receives the shooting instruction, and performs video shooting according to the shooting instruction. During the shooting process, the electronic device collects the gyroscope information corresponding to each frame of the image through the gyroscope sensor. The processed frame image may be a single frame image adjacent to the current frame image in the time domain, or a multi-frame image. When the processed frame image is a single frame image, it can be the previous frame processed image adjacent to the current frame image in time domain, or it can be called the previous frame processed image. When the processed frame image is a multi-frame image, it may be the previous frame processed image adjacent to the current frame image in the time domain, the previous frame processed image, and the like. The specific number of processed frame images can be selected according to preset requirements. The processed frame image corresponds to the processed frame gyro information, and the processed frame image of each frame has corresponding gyro information (for easy distinction, it is referred to as processed frame gyro information in this application).

Step 204: Perform anti-shake processing on the current frame image by using the current frame gyro information to generate a stabilized current frame image.

The electronic device performs single-frame denoising processing on the current frame image to obtain the current frame image after denoising. After the electronic device performs posture estimation and filtering processing on the gyroscope information of the current frame, the current posture and the target posture corresponding to the current frame image are obtained. The filtering process can adopt low-pass filtering. In order to facilitate the distinction, the current pose and the target pose corresponding to the current frame image are referred to as the current frame current pose and the current frame target pose hereinafter, respectively. The gyroscope information of the current frame includes the rotation angle information of the camera. The electronic device uses the denoised current frame image, current frame current posture, and current frame target posture for image warp operation (image warp), and rotates the current frame image from the current frame current posture to the current frame target posture according to the rotation angle information , In order to eliminate the jitter in the current frame image, and generate a stable current frame image.

Specifically, the electronic device can also adjust the sequence of single-frame denoising and image reprojection operations. When performing single-frame denoising and performing image reprojection operations, the pixel values of the current frame image will be interpolated. If the interpolation precision of single frame denoising is high, the current frame image is first subjected to single frame denoising processing, and then combined with the current frame gyro information for image reprojection operation, so as to obtain a stabilized current frame image. If the interpolation accuracy of the image re-projection operation is high, the image re-projection operation is first combined with the gyroscope information of the current frame, and then the single-frame denoising process is performed to obtain a stabilized current frame image. Since the interpolation accuracy will affect the image quality of the output video, adjusting the operation with high interpolation accuracy to the front for processing can effectively reduce the noise in the output video and improve the picture quality.

Step 206: Perform fusion filtering processing on the stabilized current frame image and the processed frame image to obtain an optimized processed video image corresponding to the current frame image.

The electronic device performs global alignment processing on the stabilized current frame image and the processed frame image to obtain global alignment information between the stabilized current frame image and the processed frame image. The electronic device performs a weighted average operation based on the global alignment information, and performs multi-frame fusion noise reduction processing on the stabilized current frame image and the processed frame image, thereby obtaining an optimized processed video image corresponding to the current frame image. The electronic device sends the optimized video image to the encoder for video encoding, and after video encoding, it is sent to the monitor for display.

When the electronic device optimizes the next frame of image, it can use the optimized processed video image corresponding to the current frame of image as the processed frame image, and process with reference to the above method to obtain the optimized processed image corresponding to the next frame of image. Video image. Repeat the execution to optimize the entire video.

In the related art, after filtering the image, the image stabilization processing is considered. Since the filtering process needs to perform a global image alignment operation based on the image content, if there are moving objects, the stability of the video will be greatly disturbed. In this embodiment, the current posture and target posture of the current frame image estimated based on the gyroscope information of the current frame have nothing to do with the graphic content by putting the anti-shake processing in front, so that the global motion of the current frame image can be estimated independently. It is not disturbed by moving objects and effectively improves the stability of the video. In addition, in the related art, when the filtering process performs a global alignment operation, it takes a long time to traverse the global alignment of the current frame image and the processed frame image based on the image content, which will also reduce the stability. In this embodiment, after the anti-shake processing, the stabilized current frame image and the processed frame image are basically aligned, and accurate image alignment only needs to traverse a small amount of image range, thereby effectively reducing the amount of calculation for the global alignment operation , Improve the efficiency of fusion filtering processing, and then improve the stability of the output video.

In this embodiment, by acquiring the current frame image, current frame gyro information, and processed frame image from the video, the current frame gyro information can be used to perform anti-shake processing on the current frame image to generate a stabilized current frame image. By putting the anti-shake processing in front, it is possible to eliminate the jitter in the current frame image in advance. By performing fusion filtering processing on the stabilized current frame image and processed frame image, the filtering processing efficiency can be effectively improved, and the calculation time of fusion filtering can be saved, thereby effectively improving the stability of the output video.

In one embodiment, performing fusion filtering processing on the stabilized current frame image and processed frame image includes: performing global alignment processing on the stabilized current frame image and processed frame image; and globally aligning the current frame image with The processed frame image after the global alignment is subjected to fusion filtering processing to obtain an optimized processed frame image.

When the electronic device performs fusion filtering processing on the stabilized current frame image and the processed frame image, it may use Motion Compensation Temporal Filtering (MCTF). When performing global alignment processing, the electronic device can use the stabilized current frame image to match the image characteristics of the processed frame image. Specifically, the electronic device extracts the image feature points of the stabilized current frame image and the processed frame image respectively. The image feature points corresponding to the stabilized current frame image may be referred to as the current frame feature points. The image feature points of the processed frame image can be called the processed frame feature points. The electronic device finds a matching pair between the feature points of the current frame and the feature points of the processed frame, calculates the stable conversion matrix between the current frame image and the processed frame image based on the matching pair, and establishes the correspondence between all pixels, Thus, the global alignment information between the stabilized current frame image and the processed frame image is obtained.

In related technologies, when the current frame feature points are matched with the processed frame feature points, the stabilized current frame image and the processed frame image need to be traversed based on the image content, that is, it is necessary to search for all matches on the entire image Click to find the right matching pair. The entire matching process has a large amount of calculations, which will reduce the matching stability, thereby affecting the stability of the output video.

In this embodiment, since the stabilized current frame image is an image after the current frame image is stabilized according to the gyroscope information of the current frame image, the stabilized current frame image and the processed frame image are already basically aligned. When the electronic device performs global alignment processing on the stabilized current frame image and the processed frame image, it only needs to traverse a small amount of image range, which can greatly improve the speed of calculation and the accuracy of matching, thereby effectively improving the stability of the video image .

Further, the processed frame image may be a multi-frame image. The electronic device can perform global alignment on the processed frame image of each frame and the stabilized current frame image in the manner provided in the above-mentioned embodiment, and then perform weighted average processing based on multiple items of global alignment information, thereby completing multiple Frame fusion filter processing. Since the multi-frame fusion filtering process is to calculate the pixel average between adjacent multi-frame images in the time domain, the global alignment of multi-frame processed frame images can make the output compared with the global alignment of single-frame processed frame images. The video image is more balanced and the noise is lower, which can further improve the stability of the output video.

In an embodiment, as shown in FIG. 3, a video anti-shake optimization processing method is provided, which specifically includes the following steps:

Step 302: Acquire the current frame image, the current frame gyro information, the processed frame image, and the processed frame gyro information.

Step 304: Perform anti-shake processing on the current frame image by using the current frame gyro information to generate a stabilized current frame image.

Step 306: Use the current frame gyro information and the processed frame gyro information to generate initial registration information.

Step 308: Perform fusion filtering processing on the stabilized current frame image and the processed frame image by using the initial registration information to obtain an optimized processed video image corresponding to the current frame image.

The electronic device can perform anti-shake processing on the current frame image with reference to the method provided in the foregoing embodiment to generate a stabilized current frame image. The electronic device obtains the corresponding processed frame gyroscope information according to the processed frame image. After the electronic device performs posture estimation and filtering processing on the gyroscope information of the current frame, the current posture and the target posture corresponding to the current frame image are obtained. In order to facilitate the distinction, the current pose and the target pose corresponding to the current frame image are referred to as the current frame current pose and the current frame target pose hereinafter, respectively. After the electronic device performs posture estimation and filtering processing on the processed frame gyro information, the current posture and the target posture corresponding to the processed frame image are obtained respectively. In order to facilitate the distinction, the current posture and the target posture corresponding to the processed frame image are referred to as the current posture of the processed frame and the target posture of the processed frame, respectively. The electronic device uses the current frame target posture and the processed frame target posture to perform operations to generate initial registration information between the current frame image and the processed frame image.

The electronic device obtains the reference point of the global alignment operation according to the initial registration information. Based on this reference point, you can refer to the method in the above-mentioned embodiment. The electronic device performs a global alignment operation on the stabilized current frame image and the processed frame image to obtain the stabilized current frame image and the processed frame image. Global alignment information. The electronic device performs a weighted average operation based on the global alignment information, and performs multi-frame fusion noise reduction processing on the stabilized current frame image and the processed frame image, thereby obtaining an optimized processed video image corresponding to the current frame image.

In this embodiment, since the stabilized current frame image and the processed frame image have been basically aligned. When the electronic device performs global alignment processing on the stabilized current frame image and the processed frame image, it only needs to traverse a small amount of image range. During the traversal, the registration information is initialized as a reference point, and the matching feature points are searched near the reference point. There is no need to brute force search for all matching points on the entire image, thereby further improving the matching accuracy and matching speed. Thereby, the calculation amount of the fusion filtering process is further reduced, the speed of the fusion filtering is improved, and the stability of the output video is improved.

In one embodiment, as shown in FIG. 4, the step of using the current frame gyroscope information and the processed frame gyroscope information to generate initial registration information includes:

Step 402: Perform attitude estimation on the gyroscope information of the current frame to obtain the target attitude of the current frame.

Step 404: Perform attitude estimation on the processed frame gyro information to obtain the processed frame target attitude.

Step 406: Use the current frame target pose and the processed frame target pose to generate initial registration information.

In this embodiment, the anti-shake processing may use EIS (Electronic image stabilization, electronic image stabilization). The gyro information of the current frame of the electronic device performs EIS attitude estimation and filtering processing to obtain the current frame current attitude and the current frame target attitude. The electronic device performs EIS attitude estimation and filtering processing on the processed frame gyro information, the current attitude of the processed frame and the processed frame target attitude. The filtering process can be low-pass filtering, which removes the high-frequency components (that is, the amount of jitter) in the current frame of gyroscope information or the processed frame of gyroscope information, and retains the low-frequency components to remove jitter. The electronic device calculates the relative posture relationship between the current frame image and the processed frame image by calculating the current frame target posture and the processed frame target posture, and this relative posture relationship can be used as the initial registration between the current frame image and the processed frame image information.

In one of the embodiments, using the current frame target pose and the processed frame target pose to generate the initial registration information includes: obtaining the current frame target pose matrix corresponding to the current frame target pose; obtaining the processed frame target corresponding to the processed frame target pose Pose matrix: use the current frame target pose matrix and the processed frame target pose matrix to calculate to obtain the registration matrix corresponding to the initial registration information.

The target pose of the current frame can be represented by the target pose matrix of the current frame. The processed frame target pose can be represented by the processed frame target pose matrix. The target attitude matrix of the current frame may be a conversion matrix obtained after attitude estimation and filtering according to the gyroscope information of the current frame. The processed frame target attitude matrix may be a conversion matrix obtained after attitude estimation and filtering of the processed frame gyro information. The electronic device calculates the target pose matrix of the current frame and the target pose matrix of the processed frame according to the preset rule to obtain the configuration matrix. For example, the configuration matrix can be obtained by multiplying the target pose matrix of the current frame by the inverse of the target pose matrix of the processed frame. The configuration matrix is used to characterize the initial registration information for performing a global alignment operation on the current frame image and the processed frame image.

In this embodiment, by calculating the initial registration information between the current frame image and the processed frame image, the reference point for the global alignment operation of the current frame image and the processed frame image can be obtained, and the processed frame image can be The update is accurately aligned to the current frame image, which effectively improves the speed of global alignment, thereby saving the time-consuming calculation of fusion filtering and promoting the improvement of the stability of the output video.

In one of the embodiments, when the processed frame images are multiple frames, the electronic device estimates the posture of the processed frame gyroscope information of each frame to obtain the processed frame target posture of each frame. With reference to the above method, the electronic device calculates the current frame target posture and the processed frame target posture of each frame respectively to obtain the configuration matrix corresponding to the current frame image and the processed frame image of each frame. That is, when the processed frame image is multiple frames, the electronic device will generate the same configuration matrix as the processed frame image to ensure that the current frame image can be globally aligned with the processed frame image of each frame. The corresponding initial registration information is adopted to ensure the accuracy of the global alignment of the current frame and the processed frame image of each frame.

In one embodiment, as shown in FIG. 4, the steps of performing fusion filtering processing on the stabilized current frame image and the processed frame image by using the initial registration information include:

Step 402: Perform global alignment processing on the stabilized current frame image and the processed frame image of each frame respectively using corresponding initial registration information.

Step 404: Perform fusion noise reduction processing on the globally aligned current frame image and the multi-frame globally aligned processed frame image to obtain an optimized processed frame image.

The processed frame image includes multiple frames. The electronic device may calculate the initial registration information between the processed image of each frame and the stabilized current frame image according to the method provided in the foregoing embodiment. The electronic device uses the initialized registration information to perform global alignment processing on the stabilized current frame image and the processed frames of each frame in the manner provided in the foregoing embodiment, to obtain multiple pieces of global alignment information. The electronic device performs multi-frame fusion filtering processing based on a number of global alignment information.

Since the video has more time domain information than the static image, in this embodiment, by combining the processed image of the previous frame or the processed image of the previous few frames for fusion and noise reduction, it is possible that there may be moving objects between adjacent frame images. Down, to ensure the effect of noise reduction, effectively improve the stability of the output video.

In one embodiment, performing filtering processing on the stabilized current frame image and processed frame image includes: performing global alignment processing on the stabilized current frame image and processed frame image; using the globally aligned processed frame image and The global aligned current frame image is locally aligned; the locally aligned processed frame image and the locally aligned current frame image are subjected to fusion filtering processing to obtain an optimized processed frame image corresponding to the current frame image.

After the electronic device performs global alignment processing on the stabilized current frame image and processed frame image according to the method provided in the above embodiment, it can further detect whether there is local motion, that is, detect the pixels in the processed frame image after global alignment. The point moves to the position in the current frame image after the global alignment. If no object motion occurs, the movement relationship of the pixels in the processed frame image after the global alignment to the current frame image after the global alignment is consistent, which belongs to the motion of the camera itself. There is no need to continue the local alignment processing, and the image obtained after fusion filtering based on the global alignment information can be used as the optimized processing result of the current frame image.

If object motion occurs, local motion detection needs to be performed to find the position of the moving object in the processed frame image after the global alignment, and the position in the current frame image after the global alignment. Specifically, the electronic device may obtain an image block with a preset pixel size, and based on the image block, search for different image blocks between the globally aligned processed frame image and the globally aligned current frame image, and then combine the different images. Block as a local area. After the global alignment process, the pixel motion relationship between the processed frame image after the global alignment and the current frame image after the global alignment is basically the same, the corresponding relationship between the pixels in the local area can be established to complete the global alignment. The local alignment of the processed frame image and the current frame image after the global alignment. The electronic device uses the partially aligned processed frame image and the partially aligned current frame image to perform multi-frame fusion filtering processing in the manner provided in the foregoing embodiment, thereby obtaining an optimized processing result corresponding to the current frame image.

In this embodiment, by performing local alignment processing on the processed frame image after the global alignment process and the current frame image after the global alignment process, the local jitter caused by the moving object can be eliminated, thereby further improving the stability of the output video.

Further, the processed frame image may be multiple frames. The electronic device may perform global alignment processing on the processed frame image of each frame and the stabilized current frame image in the manner provided in the above-mentioned embodiment, to obtain the processed frame image and the global alignment after the global alignment of each frame. The current frame image after. The electronic device can also perform local alignment processing on the globally aligned processed frame image of each frame and the globally aligned current frame image in the manner provided in the above-mentioned embodiment, and the partially aligned processed frame of multiple frames The image and the partially aligned current frame image are subjected to multi-frame fusion filtering processing, and the optimized processing result corresponding to the current frame image is obtained.

When there is a moving object, by combining the processed image of the previous frame or the processed image of the previous few frames for global alignment and local alignment, and then fusion noise reduction, it can effectively eliminate the local of the moving object between adjacent frames. Jump to further improve the stability of the output video.

In one embodiment, as shown in FIG. 5, the step of performing fusion filtering processing on the globally aligned current frame image and the globally aligned processed frame image includes:

Step 502: Obtain the current pixel value of the current frame image after the global alignment.

Step 504: Obtain the accumulated values of pixels of multiple processed frame images.

Step 506: Perform an averaging operation on the current pixel value and the acquired pixel accumulated value to obtain an average pixel value, and use the average pixel value as the pixel value of the optimized video image.

When the processed frame image is a multi-frame, the global alignment of the stabilized current frame image and the processed frame image of each frame will increase the amount of calculations compared to when the processed frame image is a single frame. In order to effectively balance the contradiction between the computational efficiency and the stability of the output video brought about by the increase in computational complexity. The electronic device can predict and calculate the pixel accumulated value of multiple processed frame images. The electronic device performs anti-shake processing on the current frame image to generate a stabilized current frame image, and the electronic device calculates the pixel value of the stabilized current frame image as the current pixel value. In order to improve the calculation speed of the multi-frame fusion filtering, the electronic device averages the current pixel value and the pixel accumulated value to obtain the average pixel value, and uses the average pixel value as the pixel value of the optimized video image.

For example, the number of processed frame images is N-1, and the current frame image is the Nth frame. The electronic device may accumulate the pixel values of the processed frame image of the previous N-1 frames to obtain the accumulated pixel value. After the image of the Nth frame is subjected to anti-shake processing, it is calculated as the pixel value of the stabilized current frame of image, and it can also become the current pixel value of the image of the Nth frame. The electronic device performs an average calculation on the current pixel value and the pixel accumulated value of the N-1 frame processed frame image, and uses the average pixel as the optimized pixel value of the Nth frame image. The electronic device subtracts the pixel value of the processed frame image of the first frame from the cumulative sum, adds the average pixel value, and updates the pixel cumulative value. The updated pixel accumulated value is used as the pixel accumulated value required for multi-frame fusion filtering of the next frame of image. Repeat this until all frame images in the video are optimized.

In this process, because the multi-frame fusion filtering process adopts the method of taking the average value of adjacent image pixels in the time domain, it is only necessary to calculate the pixel value of the current frame image after the noise reduction process in each calculation. The pixel accumulation value of the processed frame image only needs to update the pixels of the first frame and the last frame, without repeating the calculation of all the pixel accumulations, thus effectively reducing the corresponding calculation amount in the process of multi-frame fusion filtering , Achieving an effective balance between computing efficiency and output video stability.

Further, when the local alignment processing needs to be performed after the local alignment processing, the electronic device may also perform the multi-frame fusion filtering processing in the above manner after the local alignment processing. As a result, while eliminating local jitter, the efficiency of multi-frame fusion filtering processing can be improved, and the stability of the output video can be ensured.

It should be understood that although the various steps in the flowcharts of FIGS. 2 to 5 are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless specifically stated in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least some of the steps in FIGS. 2 to 5 may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. These sub-steps or stages The execution order of is not necessarily performed sequentially, but may be performed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

Fig. 6 is a structural block diagram of a video anti-shake optimization processing device according to an embodiment. The device includes: an acquisition module 602, an anti-shake module 604, and a filtering module 606, wherein:

The acquisition module 602 is used to acquire the current frame image, the current frame gyro information, and the processed frame image. The current frame image and the processed frame image are adjacent in the time domain.

The anti-shake module 604 is configured to perform anti-shake processing on the current frame image by using the current frame gyro information to generate a stabilized current frame image.

The filtering module 606 is configured to perform fusion filtering processing on the stabilized current frame image and the processed frame image to obtain an optimized processed video image corresponding to the current frame image.

In one embodiment, the filtering module 606 is also used to perform global alignment processing on the stabilized current frame image and processed frame image; perform fusion filtering processing on the globally aligned current frame image and the processed frame image after global alignment , Get the optimized frame image.

In one embodiment, the acquisition module 602 is also used to acquire processed frame gyroscope information. As shown in FIG. 7, the device further includes: an initialization registration module 608, which is used to use current frame gyroscope information and processed frame gyroscope information. The instrument information generates initial registration information; the filtering module 606 is also used to perform fusion filtering processing on the stabilized current frame image and the processed frame image using the initial registration information to obtain an optimized processed video image corresponding to the current frame image.

In one embodiment, the initialization registration module 608 is also used to perform pose estimation on the current frame of gyroscope information to obtain the current frame target pose; perform pose estimation on the processed frame gyroscope information to obtain the processed frame target pose; use the current frame The frame target pose and the processed frame target pose generate initial registration information.

In one embodiment, the initialization registration module 608 is also used to obtain the target pose matrix of the current frame corresponding to the target pose of the current frame; obtain the target pose matrix of the processed frame corresponding to the target pose of the processed frame; The posture matrix of the frame target is processed for calculation, and the registration matrix corresponding to the initial registration information is obtained.

In one embodiment, the initialization registration module 608 is also used for the processed frame image including multiple frames, and the processed frame gyroscope information of each frame is estimated to obtain the processed frame target attitude of each frame; The frame target pose and the processed frame target pose of each frame respectively generate multiple pieces of initial registration information.

In one embodiment, the filtering module 606 is further configured to perform global alignment processing on the stabilized current frame image and the processed frame image of each frame by using the corresponding initial registration information; The processed frame images after multi-frame global alignment are processed for fusion and noise reduction to obtain optimized processed frame images.

In one embodiment, the filtering module 606 is further configured to perform global alignment processing on the stabilized current frame image and processed frame image; perform local alignment using the processed frame image after the global alignment and the current frame image after the global alignment; The locally aligned processed frame image and the partially aligned current frame image are subjected to fusion filtering processing to obtain an optimized processed frame image corresponding to the current frame image.

In one embodiment, the filtering module 606 is also used to obtain the current pixel value of the current frame image after the global alignment; obtain the pixel accumulated value of multiple processed frame images; average the current pixel value and the obtained pixel accumulated value Operate to get the average pixel value; use the average pixel value as the pixel value of the optimized video image.

The division of the various modules in the video anti-shake optimization processing device is only used for illustration. In other embodiments, the video anti-shake optimization processing device can be divided into different modules as needed to complete the above-mentioned video anti-shake optimization processing device. All or part of the function.

The implementation of each module in the video anti-shake optimization processing device provided in the embodiments of the present application may be in the form of computer-readable instructions. The computer readable instructions can be executed on the electronic device. The program module formed by the computer readable instructions can be stored in the memory of the electronic device. When the computer-readable instructions are executed by the processor, the steps of the method described in the embodiments of the present application are implemented.

Fig. 8 is a schematic diagram of the internal structure of an electronic device in an embodiment. As shown in FIG. 8, the electronic device includes a processor and a memory connected through a system bus. Among them, the processor is used to provide calculation and control capabilities to support the operation of the entire electronic device. The memory may include a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer readable instructions. The computer-readable instruction may be executed by the processor to implement a video anti-shake optimization processing method provided in the following embodiments. The internal memory provides a cached operating environment for the operating system computer readable instructions in the non-volatile storage medium. The electronic device can be a mobile phone, a tablet computer, or a personal digital assistant or a wearable device.

Those skilled in the art can understand that the structure shown in FIG. 8 is only a block diagram of part of the structure related to the solution of the present application, and does not constitute a limitation on the server to which the solution of the present application is applied. The specific server may include More or fewer components are shown in the figure, or some components are combined, or have different component arrangements.

The embodiment of the present application also provides an electronic device. The above-mentioned electronic equipment includes an image processing circuit. The image processing circuit may be implemented by hardware and/or software components, and may include various processing units that define an ISP (Image Signal Processing, image signal processing) pipeline. Fig. 9 is a schematic diagram of an image processing circuit in an embodiment. As shown in FIG. 9, for ease of description, only various aspects of the image processing technology related to the embodiments of the present application are shown.

As shown in FIG. 9, the image processing circuit includes an ISP processor 940 and a control logic 950. The image data captured by the imaging device 910 is first processed by the ISP processor 940, and the ISP processor 940 analyzes the image data to capture image statistics that can be used to determine and/or one or more control parameters of the imaging device 910. The imaging device 910 may include a camera having one or more lenses 912 and an image sensor 914. The image sensor 914 may include a color filter array (such as a Bayer filter). The image sensor 914 may obtain the light intensity and wavelength information captured by each imaging pixel of the image sensor 914, and provide a set of raw materials that can be processed by the ISP processor 940. Image data. The sensor 920 (such as a gyroscope) can provide the collected image processing parameters (such as anti-shake parameters) to the ISP processor 940 based on the sensor 920 interface type. The sensor 920 interface can use SMIA (Standard Mobile Imaging Architecture) interface, other serial or parallel camera interfaces, or a combination of the above interfaces.

In addition, the image sensor 914 may also send raw image data to the sensor 920, the sensor 920 may provide the raw image data to the ISP processor 940 based on the sensor 920 interface type, or the sensor 920 may store the raw image data in the image memory 930.

The ISP processor 940 processes the original image data pixel by pixel in a variety of formats. For example, each image pixel may have a bit depth of 8, 10, 12, or 14 bits, and the ISP processor 940 may perform one or more image processing operations on the original image data, and collect statistical information about the image data. Among them, the image processing operations can be performed with the same or different bit depth accuracy.

The ISP processor 940 may also receive image data from the image memory 930. For example, the sensor 920 interface sends the original image data to the image memory 930, and the original image data in the image memory 930 is then provided to the ISP processor 940 for processing. The image memory 930 may be a part of a memory device, a storage device, or an independent dedicated memory in an electronic device, and may include DMA (Direct Memory Access) features.

When receiving raw image data from the image sensor 914 interface or from the sensor 920 interface or from the image memory 930, the ISP processor 940 may perform one or more image processing operations, such as temporal filtering. The processed image data can be sent to the image memory 930 for additional processing before being displayed. The image data processed by the ISP processor 940 may be output to the display 970 for viewing by the user and/or further processed by a graphics engine or a GPU (Graphics Processing Unit, graphics processor). In addition, the output of the ISP processor 940 can also be sent to the image memory 930, and the display 970 can read image data from the image memory 930. In one embodiment, the image memory 930 may be configured to implement one or more frame buffers. In addition, the output of the ISP processor 940 may be sent to the encoder/decoder 960 in order to encode/decode image data. The encoded image data can be saved and decompressed before being displayed on the display 970 device. The encoder/decoder 960 may be implemented by a CPU or GPU or a co-processor.

The statistical data determined by the ISP processor 940 may be sent to the control logic 950 unit. For example, the statistical data may include image sensor 914 statistical information such as automatic exposure, automatic white balance, automatic focus, flicker detection, black level compensation, and lens 912 shading correction. The control logic 950 may include a processor and/or a microcontroller that executes one or more routines (such as firmware). The one or more routines can determine the control parameters and ISP processing of the imaging device 910 based on the received statistical data. 940 control parameters. For example, the control parameters of the imaging device 910 may include sensor 920 control parameters (such as gain, integration time of exposure control, anti-shake parameters, etc.), camera flash control parameters, lens 912 control parameters (such as focus or zoom focal length), or these The combination of parameters. The ISP control parameters may include gain levels and color correction matrices for automatic white balance and color adjustment (e.g., during RGB processing), and lens 912 shading correction parameters.

The steps included in the foregoing method embodiments can be realized by using the image processing technology in FIG. 9. In one of the embodiments, the electronic device may use the imaging device 910 to collect video images, and the image sensor 914 to collect sensor information corresponding to each frame of the video image. The image sensor 920 may be a gyroscope sensor, and the corresponding sensor information is gyroscope information. The gyroscope sensor provides the collected gyroscope information and the corresponding video image to the ISP processor 940. The ISP processor 940 performs optimization processing on the current frame image. The ISP processor 940 uses the current frame gyroscope information to perform anti-shake processing on the current frame image to generate a stabilized current frame image. The ISP processor 940 then performs fusion filtering processing on the stabilized current frame image and the processed frame image to obtain an optimized processed video image corresponding to the current frame image. In order to effectively increase the speed of fusion filtering, the ISP processor 940 obtains the processed frame gyroscope information. The ISP processor 940 uses the current frame gyroscope information and the processed frame gyroscope information to generate initial registration information. The ISP processor 940 obtains the reference point of the global alignment operation according to the initial registration information. Based on this reference point, the ISP processor 940 performs a global alignment operation on the stabilized current frame image and the processed frame image, and performs multi-frame fusion filtering processing based on the global alignment information, thereby obtaining optimization processing corresponding to the current frame image After the video image. The ISP processor 940 outputs the optimized processed video image and sends it to the encoder/decoder 960 to encode/decode image data. The encoded image data can be saved and displayed on the display 970 device. Since the anti-shake processing of the ISP processor 940 is pre-processed during the optimization process, the jitter in the current frame image can be eliminated in advance. By performing fusion filtering processing on the stabilized current frame image and processed frame image, the filtering processing efficiency can be effectively improved, and the calculation time of fusion filtering can be saved, thereby effectively improving the stability of the output video.

The embodiment of the present application also provides an electronic device, including a memory and one or more processors. The memory stores computer-readable instructions. When the computer-readable instructions are executed by one or more processors, one or more The processor executes the steps provided in the methods of the above embodiments.

The embodiment of the present application also provides a computer-readable storage medium. One or more non-volatile computer-readable storage media containing computer-executable instructions, when the computer-executable instructions are executed by one or more processors, cause the processors to perform the steps provided in the above-mentioned various embodiment methods.

A computer program product containing computer-readable instructions, when it runs on a computer, causes the computer to execute the video anti-shake optimization processing method provided in the foregoing embodiments.

Any reference to memory, storage, database, or other media used in the embodiments of the present application may include non-volatile and/or volatile memory. Suitable non-volatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which acts as external cache memory. As an illustration and not a limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous Link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

The above-mentioned embodiments only express several implementation manners of the present application, and their description is relatively specific and detailed, but they should not be understood as a limitation to the patent scope of the present application. It should be pointed out that for those of ordinary skill in the art, without departing from the concept of this application, several modifications and improvements can be made, and these all fall within the protection scope of this application. Therefore, the scope of protection of the patent of this application shall be subject to the appended claims.

Claims (20)

A video anti-shake optimization processing method, including: Acquiring a current frame image, current frame gyroscope information, and processed frame image; the current frame image and the processed frame image are adjacent in the time domain; Using the current frame gyroscope information to perform anti-shake processing on the current frame image to generate a stabilized current frame image; and Performing fusion filtering processing on the stabilized current frame image and the processed frame image to obtain an optimized processed video image corresponding to the current frame image. The method according to claim 1, wherein the performing fusion filtering processing on the stabilized current frame image and the processed frame image comprises: Perform global alignment processing on the stabilized current frame image and the processed frame image; and perform fusion filtering processing on the globally aligned current frame image and the globally aligned processed frame image to obtain an optimized processed frame image. The method according to claim 1, wherein the method further comprises: Acquiring processed frame gyroscope information, and using the current frame gyroscope information and the processed frame gyroscope information to generate initial registration information; and The performing fusion filtering processing on the stabilized current frame image and the processed frame image to obtain an optimized processed video image corresponding to the current frame image includes: Using the initialization registration information to perform fusion filtering processing on the stabilized current frame image and the processed frame image to obtain an optimized processed video image corresponding to the current frame image. The method according to claim 3, wherein said generating initialization registration information using said current frame gyroscope information and said processed frame gyroscope information comprises: Performing pose estimation on the gyro information of the current frame to obtain the target pose of the current frame; Performing attitude estimation on the processed frame gyro information to obtain the processed frame target attitude; and The initial registration information is generated using the current frame target pose and the processed frame target pose. The method according to claim 4, wherein said generating initial registration information using said current frame target posture and said processed frame target posture comprises: Acquiring a current frame target pose matrix corresponding to the current frame target pose; Acquiring the processed frame target pose matrix corresponding to the processed frame target pose; and The current frame target posture matrix and the processed frame target posture matrix are used for calculation to obtain the registration matrix corresponding to the initial registration information. The method according to claim 4, wherein the processed frame image includes multiple frames, The performing pose estimation on the processed frame gyro information includes: performing pose estimation on the processed frame gyro information of each frame to obtain the processed frame target pose of each frame; and The generating initial registration information using the current frame target posture and the processed frame target posture includes: using the current frame target posture and the processed frame target posture of each frame to respectively generate multiple pieces of initial registration information. The method according to claim 6, wherein the using the initial registration information to perform fusion filtering processing on the stabilized current frame image and the processed frame image comprises: Perform global alignment processing on the stabilized current frame image and the processed frame image of each frame respectively by using corresponding initial registration information; and Perform a fusion noise reduction process on the current frame image after the global alignment and the processed frame image after the multi-frame global alignment, to obtain an optimized processed frame image. The method according to claim 1, wherein the filtering process on the stabilized current frame image and the processed frame image comprises: Performing global alignment processing on the stabilized current frame image and the processed frame image; Locally align the processed frame image after the global alignment with the current frame image after the global alignment; and Perform fusion filtering processing on the locally aligned processed frame image and the partially aligned current frame image to obtain an optimized processed frame image corresponding to the current frame image. The method according to claim 2, wherein the fusion filtering processing of the current frame image after the global alignment and the processed frame image after the global alignment comprises: Acquiring the current pixel value of the current frame image after the global alignment; Obtaining the accumulated value of pixels of multiple processed frame images; Averaging the current pixel value and the acquired pixel accumulated value to obtain an average pixel value; and The average pixel value is used as the pixel value of the optimized video image. A video anti-shake optimization processing device, including: The acquisition module is used to acquire the current frame image, the current frame gyro information, and the processed frame image; An anti-shake module, configured to perform anti-shake processing on the current frame image and the current frame gyroscope information to generate a stabilized current frame image; and The filtering module is configured to perform fusion filtering processing on the stabilized current frame image and the processed frame image to obtain an optimized processed video image corresponding to the current frame image. An electronic device, including a memory and one or more processors. The memory stores computer-readable instructions. When the computer-readable instructions are executed by the one or more processors, the one or more Each processor performs the following steps: Obtain the current frame image, the current frame gyroscope information, and the processed frame image; Performing anti-shake processing on the current frame image and the current frame gyroscope information to generate a stabilized current frame image; and Performing fusion filtering processing on the stabilized current frame image and the processed frame image to obtain an optimized processed video image corresponding to the current frame image. The electronic device according to claim 11, wherein the processor further executes the following steps: Perform global alignment processing on the stabilized current frame image and the processed frame image; and perform fusion filtering processing on the globally aligned current frame image and the globally aligned processed frame image to obtain an optimized processed frame image. The electronic device according to claim 11, wherein the processor further executes the following steps: Acquiring processed frame gyroscope information, and using the current frame gyroscope information and the processed frame gyroscope information to generate initial registration information; and Using the initialization registration information to perform fusion filtering processing on the stabilized current frame image and the processed frame image to obtain an optimized processed video image corresponding to the current frame image. The electronic device according to claim 13, wherein the processor further executes the following steps: Performing pose estimation on the gyro information of the current frame to obtain the target pose of the current frame; Performing attitude estimation on the processed frame gyro information to obtain the processed frame target attitude; and The initial registration information is generated using the current frame target pose and the processed frame target pose. The electronic device according to claim 13, wherein the processor further executes the following steps: Perform global alignment processing on the stabilized current frame image and the processed frame image of each frame respectively by using corresponding initial registration information; and Perform a fusion noise reduction process on the current frame image after the global alignment and the processed frame image after the multi-frame global alignment, to obtain an optimized processed frame image. The electronic device according to claim 11, wherein the processor further executes the following steps: Performing global alignment processing on the stabilized current frame image and the processed frame image; Locally align the processed frame image after the global alignment with the current frame image after the global alignment; and Perform fusion filtering processing on the locally aligned processed frame image and the partially aligned current frame image to obtain an optimized processed frame image corresponding to the current frame image. The electronic device according to claim 12, wherein the processor further executes the following steps: Acquiring the current pixel value of the current frame image after the global alignment; Obtaining the accumulated value of pixels of multiple processed frame images; Averaging the current pixel value and the acquired pixel accumulated value to obtain an average pixel value; and The average pixel value is used as the pixel value of the optimized video image. One or more non-volatile computer-readable storage media containing computer-executable instructions, when the computer-executable instructions are executed by one or more processors, cause the processors to perform the following steps: Obtain the current frame image, the current frame gyroscope information, and the processed frame image; Performing anti-shake processing on the current frame image and the current frame gyroscope information to generate a stabilized current frame image; and Performing fusion filtering processing on the stabilized current frame image and the processed frame image to obtain an optimized processed video image corresponding to the current frame image. The computer-readable storage medium of claim 18, wherein the processor further executes the following steps: Acquiring processed frame gyroscope information, and using the current frame gyroscope information and the processed frame gyroscope information to generate initial registration information; and Using the initialization registration information to perform fusion filtering processing on the stabilized current frame image and the processed frame image to obtain an optimized processed video image corresponding to the current frame image. The computer-readable storage medium of claim 19, wherein the processor further executes the following steps: Performing pose estimation on the gyro information of the current frame to obtain the target pose of the current frame; Performing attitude estimation on the processed frame gyro information to obtain the processed frame target attitude; and The initial registration information is generated using the current frame target pose and the processed frame target pose.

Images