WO2021163928A1

WO2021163928A1 - Optical flow obtaining method and apparatus

Info

Publication number: WO2021163928A1
Application number: PCT/CN2020/075890
Authority: WO
Inventors: 王瀛; 王林召; 占云龙; 朱衍欢; 林天鹏
Original assignee: 华为技术有限公司
Priority date: 2020-02-19
Filing date: 2020-02-19
Publication date: 2021-08-26
Also published as: CN115104125A

Abstract

The present application relates to the field of video image processing. Disclosed are an optical flow obtaining method and apparatus, for reducing time consuming of iterative calculation in the process of calculating optical flow. The optical flow obtaining method comprises: determining first similarity between a first pixel block in an (i-1)-th frame image and a second pixel block in an i-th frame image; performing at least one of the following two processes: determining second similarity between the first pixel block and a third pixel block in the i-th frame image, the coordinate of the third pixel block being obtained from historical optical flow of the first pixel block from an (i-2)-th frame image to the (i-1)-th frame; determining third similarity between the first pixel block and a fourth pixel block in the i-th frame image, the coordinate of the fourth pixel block being obtained from optical flow of a pixel block adjacent to the first pixel block; and according to at least one of the second similarity and the third similarity, the first similarity, and gradient information of the first pixel block, obtaining target optical flow of the first pixel block from the (i-1)-th frame image to the i-th frame image.

Description

Optical flow acquisition method and device

Technical field

This application relates to the field of video image processing, and in particular to an optical flow acquisition method and device.

Background technique

Optical flow (optical flow) is a concept in the detection of object motion in the field of view, which is used to describe the motion of the observation target, surface or edge caused by the motion of the observer. Optical flow algorithm is a method to infer the moving speed and direction of objects by detecting the changes in the intensity of image pixels over time. It is widely used in image processing and can be used for motion detection, motion tracking, etc.

The essence of optical flow algorithm is to solve the optimization problem iteratively. Taking brightness consistency hypothesis and motion consistency hypothesis as the core ideas, the corresponding energy function equation is constructed. Then through a large number of iterative calculations, gradually approach the optimal solution of the energy function equation. In the iterative calculation process, the initial value used is not controlled, making iterative calculation time-consuming.

Summary of the invention

The embodiments of the present application provide a method and device for obtaining optical flow, which are used to reduce the time consumption of iterative calculation in the process of calculating optical flow.

In order to achieve the foregoing objectives, the following technical solutions are adopted in the embodiments of the present application:

In a first aspect, an optical flow acquisition method is provided, including: determining a first similarity between a first pixel block in the i-1th frame image and a second pixel block in the i-th frame image; wherein, the second pixel The coordinates of the block are the same as the coordinates of the first pixel block, i is a positive integer; perform at least one of the following two processes: determine the second similarity between the first pixel block and the third pixel block in the i-th frame of image, where , The coordinates of the third pixel block are obtained from the historical optical flow of the first pixel block from the i-2th frame image to the i-1th frame; determine the third pixel block between the first pixel block and the fourth pixel block in the i-th frame image Similarity, where the coordinates of the fourth pixel block are obtained from the optical flow of adjacent pixel blocks of the first pixel block; according to at least one of the second similarity and the third similarity, the first similarity and the first pixel block Obtain the target optical flow of the first pixel block from the i-1 frame image to the i frame image.

The optical flow acquisition method provided by this application determines the first similarity between the first pixel block in the i-1th frame image and the second pixel block in the i-th frame image, and the coordinates of the second pixel block are the same as the first pixel block The coordinates of are the same, which reflects the consistency of the first pixel block to remain stationary. By determining the second similarity between the first pixel block in the i-1 frame image and the third pixel block in the i frame image, the coordinates of the third pixel block are from the first pixel block from the i-2 frame image to The historical optical flow of the i-1th frame is obtained, which reflects the optical flow of the first pixel block from the i-2th image to the i-1th frame and the optical flow from the i-1th image to the i-th image Time-domain consistency. By determining the third similarity between the first pixel block and the fourth pixel block in the i-th frame image, the coordinates of the fourth pixel block are obtained from the optical flow of the adjacent pixel blocks of the first pixel block, reflecting the first pixel block The spatial consistency between the optical flow and the optical flow of adjacent pixel blocks. The initial value of the iterative calculation of optical flow is optimized by the above similarity, so as to reduce the number of iterations. A lightweight optical flow algorithm can be implemented, and it can converge quickly at a low number of iterations.

In a possible implementation manner, the coordinates of the third pixel block are equal to the coordinates of the first pixel block plus the historical optical flow of the first pixel block from the i-2th frame of image to the i-1th frame. The size of the third pixel block is the same as the size of the first pixel block.

In a possible implementation manner, the coordinates of the fourth pixel block are equal to the coordinates of the first pixel block plus the optical flow of adjacent pixel blocks of the first pixel block. The size of the fourth pixel block is the same as the size of the first pixel block.

In a possible implementation manner, according to at least one of the second similarity and the third similarity, the first similarity, and the gradient information of the first pixel block, the first pixel block is obtained from the i-1th frame image to The target optical flow of the image of the i-th frame includes: determining the highest similarity according to at least one of the second similarity and the third similarity and the first similarity, and selecting the optical flow corresponding to the highest similarity as the initial value of the optical flow; Combining the gradient information of the first pixel block, the initial value of the optical flow is approximated by Gauss-Newton gradient descent iterative solution, and the iterative result obtained when the exit condition is met is used as the target optical flow. In other words, optimizing the initial value of optical flow based on the highest similarity can reduce the number of iterations.

In a possible implementation manner, according to at least one of the second similarity and the third similarity, the first similarity, and the gradient information of the first pixel block, the first pixel block is obtained from the i-1th frame image to The target optical flow of the i-th frame image includes: determining at least two highest similarities according to at least one of the second similarity and the third similarity and the first similarity, and selecting the ones corresponding to the at least two highest similarities respectively Optical flow is used as at least two initial values of optical flow; combined with the gradient information of the first pixel block, at least two initial values of optical flow are respectively solved by approximate Gauss-Newton gradient descent, and the exit conditions are set to be the same. When exiting from the exit condition, the iterative result with the smallest energy function is selected as the target optical flow. In other words, multiple initial values of the optical flow corresponding to the highest similarity are preferred, so that the optical flow is further optimized according to the iteration result.

In a possible implementation manner, the gradient information includes the sum of gradient values in the X direction, the sum of squares of gradient values in the X direction, the sum of gradient values in the Y direction, the sum of squares of gradient values in the Y direction, and the sum of products of gradient values in the X direction and Y direction. The method also includes: calculating convolution of each pixel of the first pixel block with the Sobel operator in the X direction to obtain a gradient matrix in the X direction, and calculating the convolution of each pixel of the first pixel block with the Sobel operator in the Y direction Product to get the gradient matrix in the Y direction; accumulate and sum all the gradient values of the gradient matrix in the X direction to get the sum of the gradient values in the X direction, square each gradient value of the gradient matrix in the X direction, and then accumulate and sum the gradient values in the X direction Sum of squares; Cumulatively sum all gradient values of the gradient matrix in the Y direction to obtain the sum of gradient values in the Y direction. After squaring each gradient value of the gradient matrix in the Y direction, the sum of squares of the gradient values in the Y direction is obtained by the cumulative sum; for X The gradient value of the same position of the direction gradient matrix and the Y direction gradient matrix are multiplied and then accumulated and summed to obtain the product sum of the gradient values in the X direction and the Y direction.

In a possible implementation manner, the method further includes: determining a fourth similarity between the first pixel block and the fifth pixel block in the i-th frame of image, wherein the coordinates of the fifth pixel block are obtained from the target optical flow; According to the first similarity and the fourth similarity, the confidence of the target optical flow is determined. Limited by the principle of the gradient-based optical flow algorithm, it is easy to cause large errors in the gradient flat area. This application also adds a confidence mechanism. The optical flow with a larger error has a lower confidence, and the optical flow with a smaller error has a higher confidence. Users can choose optical flow with high confidence.

In a possible implementation manner, determining the confidence of the target optical flow according to the first similarity and the fourth similarity includes: if the ratio of the value of the fourth similarity to the value of the first similarity is greater than the first threshold , The target optical flow is determined to be of low confidence; otherwise, if the value of the fourth similarity is greater than the second threshold, the target optical flow is determined to be of low confidence; otherwise, the target optical flow is determined to be of high confidence.

In a possible implementation manner, the method further includes: inserting an intermediate image between the i-1th frame image and the i-th frame image according to the target optical flow. Due to the limitation of the computing power of the processor, there is a gap between the image rendering rate and the display frame rate, and the time-domain super-resolution frame interpolation function can be realized by using the result of the optical flow acquisition method of the present application.

In a second aspect, an optical flow acquisition device is provided, including: a determining module, configured to determine a first similarity between a first pixel block in the i-1th frame image and a second pixel block in the i-th frame image; Wherein, the coordinates of the second pixel block are the same as the coordinates of the first pixel block, and i is a positive integer; the determining module is also used to perform at least one of the following two processes: determining the first pixel block and the i-th frame of image The second similarity of the third pixel block, where the coordinates of the third pixel block are obtained from the historical optical flow of the first pixel block from the i-2th frame image to the i-1th frame; determine the first pixel block and the i-th frame The third similarity of the fourth pixel block in the frame image, where the coordinates of the fourth pixel block are obtained from the optical flow of adjacent pixel blocks of the first pixel block; the acquisition module is used to obtain the second similarity and the third similarity according to the At least one of the similarity, the first similarity, and the gradient information of the first pixel block, to obtain the target optical flow of the first pixel block from the i-1 frame image to the i frame image.

In a possible implementation manner, the coordinates of the third pixel block are equal to the coordinates of the first pixel block plus the historical optical flow of the first pixel block from the i-2th frame of image to the i-1th frame.

In a possible implementation manner, the coordinates of the fourth pixel block are equal to the coordinates of the first pixel block plus the optical flow of adjacent pixel blocks of the first pixel block.

In a possible implementation manner, the acquiring module is specifically configured to: determine the highest similarity according to at least one of the second similarity and the third similarity and the first similarity, and select the optical flow corresponding to the highest similarity as the optical flow. The initial value of the flow; combined with the gradient information of the first pixel block, the initial value of the optical flow is approximated by Gauss-Newton gradient descent iterative solution, and the iterative result obtained when the exit condition is met is used as the target optical flow.

In a possible implementation manner, the acquiring module is specifically configured to: determine at least two highest similarities according to at least one of the second similarity and the third similarity and the first similarity, and select the at least two highest similarities The optical flow corresponding to each degree is used as at least two initial values of optical flow; combined with the gradient information of the first pixel block, the at least two initial values of optical flow are respectively solved by approximate Gauss-Newton gradient descent, and the exit conditions are set to be the same , When exiting due to meeting exit conditions, the iterative result with the smallest energy function is selected as the target optical flow.

In a possible implementation, the gradient information includes the sum of gradient values in the X direction, the sum of square gradient values in the X direction, the sum of gradient values in the Y direction, the sum of square gradient values in the Y direction, and the product sum of gradient values in the X direction and Y direction, to obtain The module is also used to calculate the convolution of each pixel of the first pixel block with the Sobel operator in the X direction to obtain a gradient matrix in the X direction, and calculate each pixel of the first pixel block with the Sobel operator in the Y direction Convolution to obtain the gradient matrix in the Y direction; accumulatively sum all the gradient values of the gradient matrix in the X direction to obtain the sum of the gradient values in the X direction, and square each gradient value of the gradient matrix in the X direction to obtain the gradient in the X direction. The sum of squares of the values; the cumulative sum of all the gradient values of the gradient matrix in the Y direction to obtain the sum of the gradient values in the Y direction, the squaring of each gradient value of the gradient matrix in the Y direction, and the cumulative sum to obtain the sum of squares of the gradient values in the Y direction; The gradient values of the same position of the X-direction gradient matrix and the Y-direction gradient matrix are multiplied and then accumulated and summed to obtain the product sum of the X-direction and Y-direction gradient values.

In a possible implementation, the determining module is further configured to: determine the fourth similarity between the first pixel block and the fifth pixel block in the i-th frame image, wherein the coordinates of the fifth pixel block are obtained from the target optical flow ; According to the first similarity and the fourth similarity, the confidence of the target optical flow is determined.

In a possible implementation manner, the determining module is specifically configured to: if the ratio of the value of the fourth degree of similarity to the value of the first degree of similarity is greater than the first threshold, determine that the target optical flow is of low confidence; otherwise, if the first If the value of the four similarities is greater than the second threshold, the target optical flow is determined to be a low confidence level; otherwise, the target optical flow is determined to be a high confidence level.

In a possible implementation manner, the optical flow acquisition device further includes a frame insertion module, configured to insert an intermediate image between the i-1th frame image and the i-th frame image according to the target optical flow.

In a third aspect, an optical flow acquisition device is provided, including a processor and a memory, where computer instructions are stored in the memory, and the processor executes the computer instructions to implement the methods of the first aspect and its possible implementation manners.

In a fourth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores computer instructions. When the computer instructions in the computer-readable storage medium run on a computer or a processor, the computer or the processor executes the first On the one hand and its possible implementation methods.

In a fifth aspect, a computer program product containing instructions is provided. When the instructions run on a computer or a processor, the computer or the processor executes the above-mentioned first aspect and its possible implementation methods.

For the technical effects of the second aspect to the fifth aspect, reference may be made to the content of the various possible implementation manners of the first aspect.

Description of the drawings

FIG. 1 is a schematic flowchart of an optical flow acquisition method provided by an embodiment of this application;

FIG. 2 is a schematic diagram of a pixel block provided by an embodiment of the application;

FIG. 3 is a schematic diagram of an optical flow provided by an embodiment of the application;

FIG. 4 is a schematic diagram of a second pixel block provided by an embodiment of the application;

FIG. 5 is a schematic diagram of a third pixel block provided by an embodiment of the application;

FIG. 6 is a schematic diagram of adjacent pixel blocks of a first pixel block according to an embodiment of the application;

FIG. 7 is a schematic diagram of a fourth pixel block provided by an embodiment of the application;

FIG. 8 is a schematic diagram of another fourth pixel block provided by an embodiment of the application;

FIG. 9 is a schematic diagram of a Sobel operator provided by an embodiment of the application;

FIG. 10 is a schematic diagram of an approximate Gauss-Newton gradient descent iterative solution provided by an embodiment of the application;

FIG. 11 is a schematic flowchart of another method for obtaining optical flow according to an embodiment of the application;

FIG. 12 is a schematic flowchart of yet another optical flow acquisition method provided by an embodiment of this application;

FIG. 13 is a schematic flowchart of yet another optical flow acquisition method provided by an embodiment of this application;

FIG. 14 is a schematic flowchart of yet another optical flow acquisition method provided by an embodiment of this application;

15 is a schematic structural diagram of an optical flow acquisition device provided by an embodiment of the application;

FIG. 16 is a schematic structural diagram of another optical flow acquisition device provided by an embodiment of the application.

Detailed ways

In the field of video image processing, especially for terminal products, high requirements are placed on the performance and effects of optical flow algorithms, and it is usually necessary to obtain high-accuracy and stable full-image optical flow information at a high speed. Therefore, implementing a lightweight optical flow algorithm through an integrated circuit is a better solution. However, the traditional optical flow algorithm requires a large number of iterative calculations to limit the increase in the frame rate, and generally, the lightweight optical flow algorithm also has the problems of large gradient flat area error and poor stability.

For example, for a large number of iterative calculations, each image block is calculated independently, and the initial value used is not controlled. If a poor initial value is used, a large number of iterative calculations are required to gradually approach a better solution. For poor stability, the optical flow of two adjacent frames of images is calculated independently from the two frames of images, and the correlation between the two frames of images is not fully utilized. For the larger error in the gradient flat area, due to the principle of the optical flow algorithm, the gradient flat area is not easy to converge, and the error is usually large.

This application provides an optical flow acquisition method and device, which determine the first similarity between the first pixel block in the i-1th frame image and the second pixel block in the i-th frame image, and the coordinates of the second pixel block are The coordinates of the first pixel block are the same, which reflects the consistency of the first pixel block to remain stationary. By determining the second similarity between the first pixel block in the i-1 frame image and the third pixel block in the i frame image, the coordinates of the third pixel block are from the first pixel block from the i-2 frame image to The historical optical flow of the i-1th frame is obtained, which reflects the optical flow of the first pixel block from the i-2th image to the i-1th frame and the optical flow from the i-1th image to the i-th image Time-domain consistency. By determining the third similarity between the first pixel block and the fourth pixel block in the i-th frame image, the coordinates of the fourth pixel block are obtained from the optical flow of the adjacent pixel blocks of the first pixel block, reflecting the first pixel block The spatial consistency between the optical flow and the optical flow of adjacent pixel blocks. The initial value of the iterative calculation of optical flow is optimized by the above similarity, so as to reduce the number of iterations. A lightweight optical flow algorithm can be implemented, and it can converge quickly at a low number of iterations.

As shown in FIG. 1, an embodiment of the present application provides an optical flow acquisition method, including steps S101-S104, wherein at least one of steps S102 and S103 is performed.

S101. Determine a first similarity between a first pixel block in the i-1th frame of image and a second pixel block in the i-th frame of image.

In the embodiments of this application, the i-th frame image is the current frame image in the video stream, the i-1th frame image refers to the previous frame image (also can be referred to as the reference frame image), and the i-2th frame refers to the previous two frames image. , And so on. i is a positive integer.

Grid division of the image to obtain pixel blocks (also called unit blocks), each pixel block includes at least four pixels (ie 2 pixels * 2 pixels), the pixel block can be rectangular or square, and the size of the pixel block is Configurable, such as 6 pixels * 6 pixels, 8 pixels * 8 pixels, 10 pixels * 10 pixels, 12 pixels * 12 pixels, 14 pixels * 14 pixels, etc.

The adjacent pixel block includes at least one common pixel point. Exemplarily, as shown in Figure 2, taking the size of each pixel block of 3 pixels * 3 pixels as an example, pixel blocks A1 and A4, A1 and A2, A1 and A3, A2 and A3, A2 and A4, A3 and A4 includes common pixels. The significance of adjacent pixel blocks including common pixels is that if the common pixels belong to multiple pixel blocks, it can be considered that the calculation result of the pixel area constituted by the common pixel points can be weighted and averaged according to the calculation results of the pixel block to which it belongs. Better guarantee robustness. If there is no overlap between the pixel blocks, the calculation result of any pixel area is only independently determined by the calculation result of a single pixel block, so there is a risk of large individual deviations.

As shown in Figure 2, if a plane coordinate system is established for an image, generally the pixel in the upper left corner of the image is the origin, the right is the X direction, and the downward is the Y direction. When an object is moving, the brightness pattern of the corresponding point on the adjacent image is also moving, and this apparent motion that characterizes the image brightness pattern is the optical flow. Specifically, optical flow represents the movement speed and direction of each pixel in two adjacent frames of images. Exemplarily, as shown in Fig. 3, suppose that the optical flow of a certain target (for example, the first pixel block) in the i-1th frame image is (2, 3), which means that the target in the i-1th frame image (For example, the first pixel block) In the i-th frame of image, 2 pixels are moved to the right and 3 pixels are moved down. Therefore, the optical flow (2,3) reflects the same target in two adjacent frames of image The displacement (displacement).

In the embodiment of the present application, the coordinates of the pixel point at the upper left corner of the pixel block are used as the coordinates of the pixel block. Of course, the coordinates of the pixel point at the center of the pixel block can also be used as the coordinates of the pixel block, which is not limited in this application.

When calculating the optical flow in the present application, for example, starting from the pixel block in the upper left corner, one row of pixel blocks is calculated sequentially to the right, and then starting from the leftmost pixel block of the next row and continuing to the right. For example, for the example in FIG. 2, the calculation order of the pixel block is A1, A2, A3, and A4.

Among them, the coordinates of the second pixel block in the image of the i-th frame (for example, the coordinates of the pixel point in the upper left corner) and the coordinates of the first pixel block in the image of the i-1th frame (for example, the coordinate of the pixel point in the upper left corner) ) Is the same, the size of the second pixel block is the same as the size of the first pixel block. That is to say, the coordinates of the first pixel block (for example, the coordinates of the pixel point in the upper left corner) are taken as the coordinates of the second pixel block in the image of the i-th frame (for example, the coordinates of the pixel point in the upper left corner). An area of the same size as the first pixel block is selected as the second pixel block in the image of the i-th frame.

The significance of this step is that the target may not move in the two frames of images, that is, the pixel blocks at the same position in the two frames of images may be the same, and the corresponding optical flow is (0,0).

Exemplarily, as shown in Fig. 4, taking the size of the first pixel block of 3 pixels*3 pixels as an example, the coordinates of the first pixel block in the i-1th frame image are (1,2), then The coordinates of the second pixel block in the image of the i-th frame are (1,2), and the size of the second pixel block is the same as the size of the first pixel block, both of which are 3 pixels*3 pixels.

In image processing, the indicators for evaluating the similarity between images include but are not limited to sum of absolute difference (SAD), sum of square difference (SSD), and normalized cross-correlation (normalized). cross correlation, NCC), etc., this application is not limited. For SAD and SSD, the larger the similarity value, the more dissimilar, and the smaller the similarity value, the more similar. For NCC, the value of similarity is a decimal between 0-1, the larger the value (closer to 1), the more similar, and the smaller its value (closer to 0), the more dissimilar.

Exemplarily, taking SSD as an example, the similarity of two pixel blocks is shown in formula 1:

Among them, x is the coordinate in the image of the i-1th frame; p is the optical flow (the initial value of the optical flow at the first iteration); W(x; p) represents the coordinate x plus the optical flow p; I(W(x;p)) is the pixel value of the pixel block at the coordinate of W(x;p) in the i-th frame image, T(x) is the pixel block at the coordinate x in the i-1th frame image The pixel value of I(W(x;p))-T(x) is the difference result of the above two pixel values.

Specifically, the similarity of brightness between pixel blocks can be calculated, or the similarity of pixel values between pixel blocks can be calculated, which is not limited in this application.

S102. Determine a second degree of similarity between the first pixel block and the third pixel block in the i-th frame of image.

Wherein, the coordinates of the third pixel block are obtained from the historical optical flow of the first pixel block from the image of the i-2th frame to the i-1th frame. Each historical optical flow corresponds to a third pixel block. The coordinates of the third pixel block (for example, the coordinates of the pixel in the upper left corner) are equal to the coordinates of the first pixel block (for example, the coordinates of the pixel in the upper left corner) plus the first pixel For the historical optical flow of the block, the size of the third pixel block is the same as the size of the first pixel block. That is to say, the historical optical flow of the first pixel block plus the coordinates of the first pixel block (for example, the coordinates of the pixel point in the upper left corner) are used as the coordinates of the third pixel block in the i-th frame image (for example, the pixel in the upper left corner). Point coordinates), and based on the coordinates, an area with the same size as the first pixel block is selected as the third pixel block in the i-th frame of image.

Because the motion of objects in the real world is consistent. In most cases, the current optical flow information can be inferred based on the historical optical flow information, or the historical optical flow information can be used to provide a reference for the calculation of the current optical flow information. This can also be referred to as temporal consistency.

The historical optical flow of the first pixel block can be obtained when the optical flow is calculated iteratively on the i-1th frame image, or it can be obtained from the optical flow in the i-2th frame image and the optical flow in the i-1th frame image. Perform filtering estimation (for example, linear filtering) to obtain that when the two historical optical flows are used at the same time, the obtained third pixel block and the second similarity are both two values.

Exemplarily, as shown in FIG. 5, the coordinates of the first pixel block in the image of the i-1th frame are (1,2), and the historical optical flow of the first pixel block is (2,3), then the The coordinates of the third pixel block in the i-frame image are (3, 5), and the size of the third pixel block is the same as the size of the first pixel block, both of which are 3 pixels*3 pixels.

For the similarity between two pixel blocks, see step S101, which will not be repeated here.

S103. Determine a third degree of similarity between the first pixel block and the fourth pixel block in the i-th frame of image.

Wherein, the coordinates of the fourth pixel block are obtained from the optical flow of adjacent pixel blocks of the first pixel block. Each current optical flow corresponds to a fourth pixel block, the coordinates of the fourth pixel block are equal to the coordinates of the first pixel block plus the current optical flow, and the size of the fourth pixel block is the same as the size of the first pixel block. That is to say, the current optical flow of the adjacent pixel block of the first pixel block plus the coordinates of the first pixel block (for example, the coordinates of the pixel point in the upper left corner) are taken as the coordinates of the third pixel block in the i-th frame image ( For example, the coordinates of the pixel point in the upper left corner), and based on the coordinates, an area of the same size as the first pixel block is selected as the third pixel block in the i-th frame of image.

Due to the consistency of rigid body motion, the current optical flow information of adjacent pixel blocks can be used as a reference to optimize the initial value of the iteration. This can also be called spatial consistency. The current optical flow refers to the optical flow obtained when the optical flow is calculated iteratively on the i-th frame image.

As mentioned above, when calculating the optical flow of the pixel block in the embodiment of the present application, it is performed in the order from left to right and top to bottom. Therefore, for the first pixel block, the The optical flow of the pixel block is known. If the optical flow of the pixel block is calculated in the reverse order, the optical flow of the pixel block below and to the right of the first pixel block is known.

Exemplarily, as shown in FIG. 6, each box represents a pixel block, and for display clarity, the overlapping part between adjacent pixel blocks is not shown. The adjacent pixel blocks of the first pixel block include first-order adjacent pixel blocks and second-order adjacent pixel blocks. The first-order adjacent pixel block refers to the pixel block that has received optical flow immediately adjacent to the first pixel block, and the second-order adjacent pixel block The pixel block is a pixel block that is separated from the first pixel block by one pixel block and has obtained optical flow. Adopting the first-order adjacent pixel block to participate in the calculation can reduce the amount of calculation, and using the second-order adjacent pixel block to participate in the calculation can improve the accuracy.

Exemplarily, as shown in FIG. 7, the coordinates of the first pixel block in the image of the i-1th frame are (1,2), and there are two first-order adjacent pixel blocks of the first pixel block (the coordinates are respectively (1,1), (2,1)), and the current optical flow of the first-order adjacent pixel block is (2,3), then there is a fourth pixel block in the i-th frame image, and the fourth pixel The coordinates of the block are (3, 5), and the size of the fourth pixel block is the same as the size of the first pixel block, both of which are 3 pixels*3 pixels. Exemplarily, as shown in FIG. 8, if the current optical flows of two first-order adjacent pixel blocks are respectively (1, 4) and (2, 3), there are two fourth pixel blocks in the i-th frame image , And the coordinates of the fourth pixel block are (2,6) and (3,5) respectively.

S104. According to at least one of the second similarity and the third similarity, the first similarity and the gradient information of the first pixel block, obtain the target light of the first pixel block from the i-1th frame image to the i-th frame image. flow.

If the image is regarded as a two-dimensional discrete function, the gradient information is the derivative of this two-dimensional discrete function, which represents the difference between the edge of the target and the background in the image.

The gradient information includes the sum of gradient values in the X direction, the sum of square gradient values in the X direction, the sum of gradient values in the Y direction, the sum of square gradient values in the Y direction, and the product sum of gradient values in the X direction and Y direction.

The following describes how to obtain the gradient information of the pixel block.

First, calculate the convolution of each pixel of the first pixel block with the X-direction Sobel operator to obtain the X-direction gradient matrix, and calculate the convolution of each pixel of the first pixel block with the Y-direction Sobel operator. Get the Y-direction gradient matrix. The size of the X-direction gradient matrix and the Y-direction gradient matrix are the same as the size of the pixel block, that is, each pixel corresponds to an X-direction gradient value in the X-direction gradient matrix, and also corresponds to a Y-direction gradient value in the Y-direction gradient matrix.

Exemplarily, as shown in FIG. 9, the size of each pixel block A is 4 pixels*4 pixels, and the size of the Sobel operator is 3 pixels*3 pixels as an example. Among them, C0 and C1 are preset convolution kernel coefficients. The 4*4=16 pixel points of the pixel block are traversed sequentially, and the convolution of each pixel point and the Sobel operator is calculated. If the Sobel operator is an X-direction Sobel operator, then an X-direction gradient matrix is obtained; if the Sobel operator is an X-direction Sobel operator, an X-direction gradient matrix is obtained.

Then, accumulate and sum all the gradient values of the gradient matrix in the X direction to obtain the sum of the gradient values in the X direction. After squaring each gradient value of the gradient matrix in the X direction, the cumulative sum is obtained to obtain the sum of squares of the gradient values in the X direction; for the Y direction All gradient values of the gradient matrix are accumulated and summed to obtain the sum of gradient values in the Y direction. After each gradient value of the gradient matrix in the Y direction is squared, the sum of the gradient values in the Y direction is accumulated and summed to obtain the square sum of the gradient values in the X direction and the Y direction. The gradient values of the same position of the gradient matrix are multiplied and then accumulated and summed to obtain the product sum of the gradient values in the X direction and the Y direction. These cumulative sum values are collectively referred to as the gradient information of the pixel block.

In a possible implementation, the highest similarity (that is, the lowest similarity value) can be determined based on at least one of the second similarity and the third similarity and the first similarity, and the light corresponding to the highest similarity can be selected. Flow as the initial value of optical flow. Combining the gradient information of the first pixel block, the initial value of the optical flow is approximated by Gauss-Newton gradient descent iterative solution, and the iterative result obtained when the exit condition is met is used as the target optical flow.

In another possible implementation manner, the at least two highest similarities may be determined according to at least one of the second similarity and the third similarity and the first similarity, and the at least two highest similarities can be selected respectively corresponding to the at least two highest similarities. Optical flow is used as at least two initial values of optical flow. Combining the gradient information of the first pixel block, perform approximate Gauss-Newton gradient descent iterative solution to at least two initial values of optical flow, and set the exit conditions to be the same. When exiting because the exit conditions are met, the one with the smallest energy function is selected The result of the iteration is used as the target optical flow.

Regarding the above-mentioned exit conditions being set to the same implementation manner, of course, other methods of determining the initial value of optical flow can also be used, which is not limited in this application:

For example, the highest similarity can be determined from the second similarity, and then the highest similarity can be determined from the third similarity, and the optical flow corresponding to the two highest similarities and the optical flow corresponding to the first similarity can be taken as three An initial value of optical flow.

The highest similarity can be determined from the second similarity and the first similarity, and then the highest similarity can be determined from the third similarity, and the optical flow corresponding to the highest similarity is used as the initial value of the two optical flows.

The highest similarity can be determined from the second similarity, the highest similarity can be determined from the third similarity and the first similarity, and the optical flow corresponding to the highest similarity can be used as the initial values of the two optical flows. and many more.

In the above process of determining the highest similarity, the threshold can also be combined to retain the similarity less than the threshold.

This application also does not limit the number of the highest similarities. For example, optical flows corresponding to multiple better similarities can be selected as the initial values of optical flows.

The following describes the process of iteratively solving the approximate Gauss-Newton gradient descent of the initial value of the optical flow in combination with the gradient information of the first pixel block.

As shown in Figure 10, the initial value of the optical flow and the gradient information of the first pixel block can be substituted into formula 2 (or formula 3), and the approximate Gauss-Newton gradient descent is iteratively solved. When the exit condition is not met, formula 2 (or The calculated result Δp of formula 3) is used as the negative feedback of optical flow F (ie F=F-Δp), and then substituted into formula 2 (or formula 3) for approximate Gauss-Newton gradient descent iterative solution, until the exit condition is met, then exit the era The optical flow entered into Equation 2 (or Equation 3) is the result of the iteration.

Among them, H is the Hessian matrix, and the Hessian matrix is obtained from the gradient information of the first pixel block. The size of the Hessian matrix is 2*2, H(0,0)=sum of square gradient values in the X direction, H(0,1)=H (1,0)=Sum of products of gradient values in X direction and Y direction, H(1,1)=Sum of squares of gradient values in Y direction;

Is the first gradient vector of the first pixel block in the i-1th frame image, including the X-direction gradient value and the Y-direction gradient value, that is, (X-direction gradient value, Y-direction gradient value);

Is the Jacobian matrix; x is the coordinate in the image of the i-1th frame; p is the optical flow (the initial value of the optical flow at the first iteration); W(x; p) represents the coordinate x plus the optical flow p Coordinates; I(W(x;p)) is the pixel value of the pixel block at the coordinate of W(x;p) in the i-th frame image, T(x) is the pixel value at the coordinate of x in the i-1th frame image The pixel value of the pixel block, I(W(x;p))-T(x) is the difference result of the above two pixel values.

Consider the

Part of it is normalized to improve the anti-interference ability of the optical flow acquisition method to the brightness change in this area, and formula 3 is obtained.

Among them, n is the number of pixels in the first pixel block, and B is the second gradient vector of the first pixel block in the i-1th frame image, including the sum of gradient values in the X direction and the sum of gradient values in the Y direction, namely (X The sum of gradient values in the direction, the sum of gradient values in the Y direction).

Correspondingly, formula 1 can be normalized to improve the anti-interference ability of the optical flow acquisition method to the brightness change of the region, and formula 4 can be obtained.

Among them, n is the number of pixels in the first pixel block.

Satisfying the exit condition includes, but is not limited to, reaching the preset number of iterations, the iteration result does not meet the convergence condition, and the iteration result exceeds the preset threshold.

Limited by the principle of the gradient-based optical flow algorithm, it is easy to cause large errors in the gradient flat area. This application also adds a confidence mechanism. The optical flow with a larger error has a lower confidence, and the optical flow with a smaller error has a higher confidence. Specifically, as shown in FIG. 11, the optical flow acquisition method further includes:

S1101: Determine the fourth similarity between the first pixel block and the fifth pixel block in the i-th frame of image.

Wherein, the coordinates of the fifth pixel block are obtained from the target optical flow. Specifically, the coordinates of the fifth pixel block are equal to the coordinates of the first pixel block plus the target optical flow, and the size of the fifth pixel block is the same as the size of the first pixel block.

In this step, obtaining the pixel block in the image of the i-th frame according to the optical flow and the first pixel block and calculating the similarity between the two pixel blocks are similar to steps S102 and S103, and will not be repeated here.

S1102, according to the first degree of similarity and the fourth degree of similarity, determine the confidence degree of the target optical flow.

The first similarity is equivalent to the similarity between the same position (the first pixel block) in the i-th frame image and the i+1-th frame image before the iterative calculation, and the fourth similarity is equivalent to the i-th image after the iterative calculation. The similarity between the same target in the frame image and the i+1th frame image.

Specifically, as shown in FIG. 12, step S1102 includes:

S11021: If the ratio of the value of the fourth degree of similarity to the value of the first degree of similarity is greater than the first threshold, it is determined that the target optical flow is of low confidence. Otherwise, step S11022 is executed.

The ratio of the value of the fourth degree of similarity to the value of the first degree of similarity is an evaluation of relative accuracy. As mentioned above, the smaller the value of similarity means the higher the degree of similarity. Therefore, the value of the fourth degree of similarity The smaller the ratio of the value to the value of the first similarity degree, the more accurate the tracking of the same target after iteration, otherwise the more inaccurate it is.

S11022: If the value of the fourth similarity is greater than the second threshold, determine that the target optical flow has a low confidence. Otherwise, step S11023 is executed.

Comparing the value of the fourth degree of similarity with the threshold value is an evaluation of absolute accuracy. The smaller the value of the fourth degree of similarity, the more accurate the tracking of the same target after iteration, otherwise the more inaccurate it is.

S11023. Determine that the target optical flow has a high degree of confidence.

The optical flow and its confidence obtained by the above-mentioned optical flow acquisition method are sparse, which can already meet the application requirements of motion detection and motion tracking. For example, based on the above information, the motion detection of the subject in the field of view can be realized, and the direction and speed of the subject's motion can be obtained, thereby realizing the motion tracking function.

The optical flow obtained by the above-mentioned optical flow acquisition method can also be densified to obtain pixel-level optical flow to achieve registration and alignment between pixels, which is used to assist in improving multi-frame denoising, multi-frame super-resolution, and multi-frame exposure fusion , Multi-frame anti-mosaic and other performance.

The optical flow algorithm can calculate the motion relationship between the pixel blocks of the two frames of images, including not only the global motion of the overall field of view, but also the local motion of objects in the scene. Therefore, the optical flow information can be regarded as a one-to-one mapping relationship established between the pixel blocks of the two frames of images, that is, the registration alignment between the pixel blocks of the image.

For applications such as multi-frame denoising, multi-frame super-resolution, multi-frame exposure fusion, and multi-frame demosaicing in image processing, the optical flow algorithm can provide the registration and alignment function between the pixel blocks of the image to assist its improvement The effect is to reduce the amount of matching calculations. If the registration between pixels can be achieved, performance will be further improved.

As shown in FIG. 13, the foregoing optical flow acquisition method further includes:

S1301: Determine the weighting coefficient of the first pixel block according to the first pixel in the i-1th frame and the target optical flow of the first pixel block including the first pixel.

Assuming that the coordinates of the first pixel in the i-1th frame are (x0, y0), the pixel value is P0, and the optical flow of the first pixel block is (dx, dy), add the coordinates of the first pixel to the first pixel The optical flow of the block obtains the second pixel in the i-th frame image, the coordinates of the second pixel are (x0+dx, y0+dy), and the pixel value of the second pixel is P1.

Determine the weighting coefficient of the first pixel block according to the pixel value of the first pixel and the pixel value of the second pixel

Among them, j represents the j-th first pixel block. When multiple first pixel blocks include the first pixel, the value range of j is greater than 1; abs represents the absolute value.

S1302: Determine the optical flow of the first pixel according to the weighting coefficient and the target optical flow of the first pixel block.

Optical flow of the first pixel

Among them, Flow _j represents the j-th first pixel block.

It should be noted that when the optical flow acquisition method described above is applied to multi-frame denoising, certain pre-denoising processing can be performed to improve the interference of noise to optical flow calculation. When the above-mentioned optical flow acquisition method is applied to multi-frame exposure fusion, the images of different exposures can be normalized to meet the brightness consistency assumption required by the optical flow algorithm.

In addition, due to the limitation of the computing power of the processor, there is a gap between the image rendering rate and the display frame rate, and the time-domain super-resolution frame interpolation function can be realized by using the result of the optical flow acquisition method of the present application.

As shown in FIG. 14, the foregoing optical flow acquisition method further includes:

S1401: Insert an intermediate image between the i-1th frame image and the i-th frame image according to the target optical flow.

The optical flow algorithm obtains the movement relationship between the real two frames of images, and on this basis, the movement relationship at a certain moment (or multiple moments) between the moments corresponding to the two frames of images can be inferred. That is to say, the super-resolution frame interpolation result can be obtained at a lower cost by using the reference frame image (for example, the image with an earlier time) and the above-mentioned mapping relationship inferred from the optical flow.

Assuming that there are a first image I1 and a second image I2, an intermediate image I1.5 of the first image I1 and the second image I2 is to be obtained. As shown in FIG. 13, the optical flow of the pixel can be obtained according to the target optical flow of the pixel block. For example, the optical flow of a pixel A in the first image I1 is F1_2.

Since the time between the two images is short enough, it can be approximately considered that the speed of the target in the image is linear, and the optical flow F1_2 can be halved to obtain the pixel point A from the first image I1 to the intermediate image I1.5 The optical flow F1_1.5=(F1_2)/2, the optical flow F1_1.5 represents the transformation relationship between the pixel point A from the first image I1 to the intermediate image I1.5. The above-mentioned processing is performed on all the pixels in the first image I1 to obtain the intermediate image I1.5.

It should be noted that for occlusion scenes and physical interaction scenes, some special considerations need to be added when inferring the optical flow information between two frames of images. For example, occlusion detection can be performed by a bidirectional optical flow acquisition method. Specifically, the optical flow of the first pixel block from the image of the i-1th frame to the image of the i-th frame can be solved first, denoted as F; then the optical flow of the first pixel block from the i-th image to the i-1th frame of image can be solved. Flow, denoted as F'. Theoretically, if there is no occlusion and the optical flow calculation is more accurate, the target at a certain position P on the i-1th frame image can be mapped from the optical flow F to the position P'on the i-th frame image; and then mapped by the optical flow F' Go back to the position P" on the i-1th frame image, and this P" and P basically coincide. Otherwise, it can be inferred that occlusion has occurred. In occlusion scenarios, optical flow with higher confidence is usually used.

The embodiments of the present application also provide an optical flow acquisition device, which is used to implement the above-mentioned various methods. The optical flow acquisition device may be a device such as a mobile phone, a tablet, a drone, a car, an electric vehicle, etc.

It can be understood that, in order to realize the above-mentioned functions, the optical flow acquisition device includes hardware structures and/or software modules corresponding to each function. Those skilled in the art should easily realize that in combination with the units and algorithm steps of the examples described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or computer software-driven hardware depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.

The embodiments of the present application may divide the optical flow acquisition device into functional modules according to the foregoing method embodiments. For example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. The above-mentioned integrated modules can be implemented in the form of hardware or software functional modules. It should be noted that the division of modules in the embodiments of the present application is illustrative, and is only a logical function division, and there may be other division methods in actual implementation.

FIG. 15 shows a schematic structural diagram of a device 150 for obtaining optical flow. The optical flow acquiring device 150 includes a determining module 1501, an acquiring module 1502, and a frame inserting module 1503. The determination module 1501 is used to implement steps S101-S103 in Figure 1 in the above method embodiment, steps S101-S103, S1101-S1102 in Figure 11, steps S101-S103, S1101, S11021-S11023 in Figure 12, and Figure 13 Steps S101-S103, S1301-S1302 in Figure 14, steps S101-S103 in Figure 14. The obtaining module 1502 is used to implement step S104 in FIG. 1, step S104 in FIG. 11, and step S104 in FIG. 13 in the foregoing method embodiment. The frame insertion module 1503 is used to implement step S1401 in FIG. 14 in the foregoing method embodiment.

Exemplarily, the determining module 1501 is configured to determine the first similarity between the first pixel block in the i-1th frame image and the second pixel block in the i-th frame image; wherein, the coordinates of the second pixel block are the same as the first pixel block in the i-th frame image. The coordinates of a pixel block are the same, and i is a positive integer.

The determining module 1501 is further configured to perform at least one of the following two processes: determining the second similarity between the first pixel block and the third pixel block in the i-th frame of image, wherein the coordinates of the third pixel block are determined by the first The pixel block is obtained from the historical optical flow from the i-2th frame image to the i-1th frame; the third similarity between the first pixel block and the fourth pixel block in the i-th frame image is determined, where the fourth pixel block The coordinates are obtained from the optical flow of adjacent pixel blocks of the first pixel block.

The obtaining module 1502 is configured to obtain the first pixel block from the i-1th frame image to the i-th frame according to at least one of the second similarity degree and the third similarity degree, the first similarity degree, and the gradient information of the first pixel block The target optical flow of the image.

In a possible implementation, the acquiring module 1502 is specifically configured to: determine the highest similarity according to at least one of the second similarity and the third similarity and the first similarity, and select the optical flow corresponding to the highest similarity as The initial value of the optical flow; combined with the gradient information of the first pixel block, the initial value of the optical flow is approximated by Gauss-Newton gradient descent iterative solution, and the iterative result obtained when the exit condition is met is used as the target optical flow.

In a possible implementation manner, the acquiring module 1502 is specifically configured to: determine at least two highest similarities according to at least one of the second similarity, the third similarity and the first similarity, and select the same as the at least two highest similarities. The optical flow corresponding to the similarity is used as at least two initial values of optical flow; combined with the gradient information of the first pixel block, the at least two initial values of optical flow are respectively solved by approximate Gauss-Newton gradient descent, and the exit condition is set as Similarly, when exiting due to meeting exit conditions, the iterative result with the smallest energy function is selected as the target optical flow.

In a possible implementation, the gradient information includes the sum of gradient values in the X direction, the sum of square gradient values in the X direction, the sum of gradient values in the Y direction, the sum of square gradient values in the Y direction, and the product sum of gradient values in the X direction and Y direction, to obtain The module 1502 is also used to: convolve each pixel of the first pixel block with the Sobel operator in the X direction to obtain a gradient matrix in the X direction, and combine each pixel of the first pixel block with the Sobel operator in the Y direction Calculate the convolution to obtain the gradient matrix in the Y direction; accumulate and sum all the gradient values of the gradient matrix in the X direction to obtain the sum of the gradient values in the X direction, and square each gradient value of the gradient matrix in the X direction to obtain the X direction. The sum of squares of gradient values; the cumulative sum of all gradient values in the Y-direction gradient matrix obtains the sum of gradient values in the Y-direction, and the cumulative sum of each gradient value in the Y-direction gradient matrix obtains the sum of squares of the gradient values in the Y direction; Multiply the gradient values at the same position of the X-direction gradient matrix and the Y-direction gradient matrix, and then accumulate and sum them to obtain the product sum of the X-direction and Y-direction gradient values.

In a possible implementation manner, the determining module 1501 is further configured to: determine the fourth similarity between the first pixel block and the fifth pixel block in the i-th frame of image, where the coordinates of the fifth pixel block are determined by the target optical flow Obtain; According to the first similarity and the fourth similarity, the confidence of the target optical flow is determined.

In a possible implementation manner, the determining module 1501 is specifically configured to: if the ratio of the value of the fourth degree of similarity to the value of the first degree of similarity is greater than the first threshold, determine that the target optical flow is of low confidence; otherwise, if If the value of the fourth similarity is greater than the second threshold, it is determined that the target optical flow is of low confidence; otherwise, the target optical flow is determined to be of high confidence.

In a possible implementation manner, the frame insertion module 1503 is configured to insert an intermediate image between the i-1th frame image and the i-th frame image according to the target optical flow.

In this embodiment, the optical flow acquisition device 150 is presented in the form of dividing various functional modules in an integrated manner. "Module" here can refer to digital signal processor (digital signal processor, DSP), application specific integrated circuit (ASIC), circuit, processor and memory that execute one or more software or firmware programs, integrated logic Circuits, and/or other devices that can provide the above-mentioned functions.

Since the optical flow obtaining device 150 provided in this embodiment can perform the above-mentioned method, the technical effects that can be obtained can refer to the above-mentioned method embodiment, which will not be repeated here.

As shown in FIG. 16, an embodiment of the present application also provides an optical flow acquisition device. The optical flow acquisition device 160 includes a processor 1602 and a memory 1601. The processor 1602 and the memory 1601 are coupled through a bus 1603, and the memory 1601 stores Computer instructions. When the processor 1602 executes the computer instructions in the memory 1601, the optical flow acquisition methods in FIGS. 1 and 11-14 are executed.

An embodiment of the present application also provides a chip, including a processor and an interface, used to call and run a computer program stored in the memory from the memory, and execute the optical flow acquisition methods in FIGS. 1 and 11-14.

The embodiment of the present application also provides a computer-readable storage medium that stores instructions in the computer-readable storage medium. When the instructions in the computer-readable storage medium run on a computer or a processor, the computer or the processor executes Figure 1, Figure 11-14 in the optical flow acquisition method.

The embodiment of the present application also provides a computer program product containing instructions. When the instructions are executed on a computer or a processor, the computer or the processor executes the optical flow acquisition methods in FIGS. 1 and 11-14.

The embodiment of the present application provides a chip system, which includes a processor, and is used for an optical flow obtaining apparatus to execute the optical flow obtaining method in FIG. 1 and FIG. 11-14.

In a possible design, the chip system also includes a memory for storing necessary program instructions and data. The chip system may include a chip, an integrated circuit, or may include a chip and other discrete devices, which is not specifically limited in the embodiment of the present application.

Among them, the optical flow acquisition device, chip, computer storage medium, computer program product, or chip system provided in this application are all used to implement the above-mentioned method. Therefore, the beneficial effects that can be achieved can refer to the above-mentioned The beneficial effects in the implementation manner will not be repeated here.

The processor involved in the embodiment of the present application may be a chip. For example, it can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), or a central processing unit. The central processor unit (CPU) can also be a network processor (NP), a digital signal processing circuit (digital signal processor, DSP), or a microcontroller (microcontroller unit, MCU) It can also be a programmable logic device (PLD) or other integrated chips.

The memory involved in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable read-only memory (programmable ROM, PROM), erasable programmable read-only memory (erasable PROM, EPROM), and electrically available Erase programmable read-only memory (electrically EPROM, EEPROM) or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of exemplary but not restrictive description, many forms of RAM are available, such as static random access memory (static RAM, SRAM), dynamic random access memory (dynamic RAM, DRAM), and synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous connection dynamic random access memory (synchlink DRAM, SLDRAM) ) And direct memory bus random access memory (direct rambus RAM, DR RAM). It should be noted that the memories of the systems and methods described herein are intended to include, but are not limited to, these and any other suitable types of memories.

It should be understood that in the various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not correspond to the embodiments of the present application. The implementation process constitutes any limitation.

A person of ordinary skill in the art may realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and conciseness of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed system, device, and method may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components may be combined or It can be integrated into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection between devices or units through some interfaces, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional units in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using a software program, it can be implemented in the form of a computer program product in whole or in part. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website, computer, server, or data center. Transmission to another website, computer, server or data center via wired (such as coaxial cable, optical fiber, Digital Subscriber Line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer or includes one or more data storage devices such as servers, data centers, etc. that can be integrated with the medium. The usable medium may be a magnetic medium (for example, a floppy disk, a hard disk, and a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)).

The above are only specific implementations of this application, but the protection scope of this application is not limited to this. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application. Should be covered within the scope of protection of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

A method for obtaining optical flow, which is characterized in that it comprises:

Determine the first similarity between the first pixel block in the i-1 frame image and the second pixel block in the i frame image; wherein the coordinates of the second pixel block are the same as the coordinates of the first pixel block , I is a positive integer;

Perform at least one of the following two procedures:

Determine the second similarity between the first pixel block and the third pixel block in the i-th frame of image, wherein the coordinates of the third pixel block are determined by the first pixel block from the i-2th frame of image The historical optical flow to the i-1th frame is obtained;

Determine the third similarity between the first pixel block and the fourth pixel block in the i-th frame of image, wherein the coordinates of the fourth pixel block are determined by the light of adjacent pixel blocks of the first pixel block. Flow

According to at least one of the second similarity and the third similarity, the first similarity, and the gradient information of the first pixel block, it is obtained that the first pixel block is from the i-1th The target optical flow from the frame image to the i-th frame image.
The method according to claim 1, wherein the coordinates of the third pixel block are equal to the coordinates of the first pixel block plus the first pixel block from the i-2th frame of image to the i-th frame. -1 frame of historical optical flow.
The method according to any one of claims 1-2, wherein the coordinates of the fourth pixel block are equal to the coordinates of the first pixel block plus the light of adjacent pixel blocks of the first pixel block. flow.
The method according to any one of claims 1-3, characterized in that, according to at least one of the second degree of similarity and the third degree of similarity, the first degree of similarity, and the first degree of similarity The gradient information of the pixel block to obtain the target optical flow of the first pixel block from the i-1 frame image to the i frame image includes:

Determining the highest similarity according to at least one of the second similarity and the third similarity and the first similarity, and selecting the optical flow corresponding to the highest similarity as the initial value of the optical flow;

In combination with the gradient information of the first pixel block, the initial value of the optical flow is approximated by Gauss-Newton gradient descent iterative solution, and the iterative result obtained when the exit condition is met is used as the target optical flow.
The method according to any one of claims 1-3, characterized in that, according to at least one of the second degree of similarity and the third degree of similarity, the first degree of similarity, and the first degree of similarity The gradient information of the pixel block to obtain the target optical flow of the first pixel block from the i-1 frame image to the i frame image includes:

Determine at least two highest similarities according to at least one of the second similarity and the third similarity and the first similarity, and select the optical flows corresponding to the at least two highest similarities as the At least two initial values of optical flow;

Combined with the gradient information of the first pixel block, the at least two initial values of optical flow are respectively solved by approximate Gauss-Newton gradient descent iterative solution, and the exit conditions are set to be the same. When the exit conditions are met, select The iteration result with the smallest energy function is used as the target optical flow.
The method according to any one of claims 1 to 5, wherein the gradient information includes the sum of gradient values in the X direction, the sum of square gradient values in the X direction, the sum of gradient values in the Y direction, and the sum of square gradient values in the Y direction. And the product sum of gradient values in the X direction and the Y direction, the method further includes:

Calculate the convolution of each pixel of the first pixel block with the X-direction Sobel operator to obtain the X-direction gradient matrix, and calculate the volume of each pixel of the first pixel block with the Y-direction Sobel operator Product to get the Y-direction gradient matrix;

Cumulatively sum all the gradient values of the X-direction gradient matrix to obtain the sum of the gradient values in the X-direction, and square each gradient value of the X-direction gradient matrix to obtain the sum of squares of the gradient values in the X-direction;

Cumulatively sum all the gradient values of the Y-direction gradient matrix to obtain the sum of the gradient values in the Y-direction, and square each gradient value of the Y-direction gradient matrix to obtain the sum of squares of the gradient values in the Y-direction;

After multiplying the gradient values at the same position of the X-direction gradient matrix and the Y-direction gradient matrix, the cumulative sum is obtained to obtain the product sum of the X-direction and Y-direction gradient values.
The method according to any one of claims 1-6, wherein the method further comprises:

Determining the fourth similarity between the first pixel block and the fifth pixel block in the i-th frame of image, wherein the coordinates of the fifth pixel block are obtained from the target optical flow;

According to the first degree of similarity and the fourth degree of similarity, the confidence degree of the target optical flow is determined.
The method according to claim 7, wherein the determining the confidence of the target optical flow according to the first similarity and the fourth similarity comprises:

If the ratio of the value of the fourth degree of similarity to the value of the first degree of similarity is greater than a first threshold value, determining that the target optical flow has a low confidence level;

Otherwise, if the value of the fourth similarity is greater than the second threshold, it is determined that the target optical flow is a low confidence level;

Otherwise, it is determined that the target optical flow is of high confidence.
The method according to any one of claims 1-8, wherein the method further comprises:

Inserting an intermediate image between the i-1th frame image and the i-th frame image according to the target optical flow.
An optical flow acquisition device, which is characterized in that it comprises:

The determining module is used to determine the first similarity between the first pixel block in the i-1th frame image and the second pixel block in the i-th frame image; wherein, the coordinates of the second pixel block are the same as those of the first pixel block. The coordinates of the pixel blocks are the same, i is a positive integer;

The determining module is also used to perform at least one of the following two processes:

Determine the second similarity between the first pixel block and the third pixel block in the i-th frame of image, wherein the coordinates of the third pixel block are determined by the first pixel block from the i-2th frame of image The historical optical flow to the i-1th frame is obtained;

Determine the third similarity between the first pixel block and the fourth pixel block in the i-th frame of image, wherein the coordinates of the fourth pixel block are determined by the light of adjacent pixel blocks of the first pixel block. Flow

The obtaining module is configured to obtain the first pixel block from the first pixel block according to at least one of the second similarity degree and the third similarity degree, the first similarity degree, and the gradient information of the first pixel block. The target optical flow from the image from the i-1th frame to the image from the i-th frame.
The optical flow acquisition device according to claim 10, wherein the coordinates of the third pixel block are equal to the coordinates of the first pixel block plus the first pixel block from the i-2th frame of image to the Describe the historical optical flow of the i-1th frame.
The optical flow acquisition device according to any one of claims 10-11, wherein the coordinates of the fourth pixel block are equal to the coordinates of the first pixel block plus the adjacent pixels of the first pixel block Block of optical flow.
The optical flow obtaining device according to any one of claims 10-12, wherein the obtaining module is specifically configured to:

Determining the highest similarity according to at least one of the second similarity and the third similarity and the first similarity, and selecting the optical flow corresponding to the highest similarity as the initial value of the optical flow;

In combination with the gradient information of the first pixel block, the initial value of the optical flow is approximated by Gauss-Newton gradient descent iterative solution, and the iterative result obtained when the exit condition is met is used as the target optical flow.
The optical flow obtaining device according to any one of claims 10-12, wherein the obtaining module is specifically configured to:

Determine at least two highest similarities according to at least one of the second similarity and the third similarity and the first similarity, and select the optical flows corresponding to the at least two highest similarities as the At least two initial values of optical flow;

Combined with the gradient information of the first pixel block, the at least two initial values of optical flow are respectively solved by approximate Gauss-Newton gradient descent iterative solution, and the exit conditions are set to be the same. When the exit conditions are met, select The iteration result with the smallest energy function is used as the target optical flow.
The optical flow acquisition device according to any one of claims 10-14, wherein the gradient information includes the sum of gradient values in the X direction, the sum of square gradient values in the X direction, the sum of gradient values in the Y direction, and the gradient in the Y direction. The sum of squares of the values and the product sum of the gradient values in the X direction and the Y direction, the acquisition module is also used for:

Calculate the convolution of each pixel of the first pixel block with the X-direction Sobel operator to obtain the X-direction gradient matrix, and calculate the volume of each pixel of the first pixel block with the Y-direction Sobel operator Product to get the Y-direction gradient matrix;

Cumulatively sum all the gradient values of the X-direction gradient matrix to obtain the sum of the gradient values in the X-direction, and square each gradient value of the X-direction gradient matrix to obtain the sum of squares of the gradient values in the X-direction;

Cumulatively sum all the gradient values of the Y-direction gradient matrix to obtain the sum of the gradient values in the Y-direction, and square each gradient value of the Y-direction gradient matrix to obtain the sum of squares of the gradient values in the Y-direction;

After multiplying the gradient values at the same position of the X-direction gradient matrix and the Y-direction gradient matrix, the cumulative sum is obtained to obtain the product sum of the X-direction and Y-direction gradient values.
The optical flow acquisition device according to any one of claims 10-15, wherein the determining module is further configured to:

Determining the fourth similarity between the first pixel block and the fifth pixel block in the i-th frame of image, wherein the coordinates of the fifth pixel block are obtained from the target optical flow;

According to the first degree of similarity and the fourth degree of similarity, the confidence degree of the target optical flow is determined.
The optical flow acquisition device according to claim 16, wherein the determining module is specifically configured to:

If the ratio of the value of the fourth degree of similarity to the value of the first degree of similarity is greater than a first threshold value, determining that the target optical flow has a low confidence level;

Otherwise, if the value of the fourth similarity is greater than the second threshold, it is determined that the target optical flow is a low confidence level;

Otherwise, it is determined that the target optical flow is of high confidence.
The optical flow obtaining device according to any one of claims 10-17, wherein the optical flow obtaining device further comprises a frame insertion module, configured to:

Inserting an intermediate image between the i-1th frame image and the i-th frame image according to the target optical flow.