CN117197507A

CN117197507A - Image block determining method and device, storage medium and electronic device

Info

Publication number: CN117197507A
Application number: CN202311473587.3A
Authority: CN
Inventors: 林亦宁; 陈曦
Original assignee: Beijing Shanma Zhijian Technology Co ltd; Hangzhou Shanma Zhiqing Technology Co Ltd; Shanghai Supremind Intelligent Technology Co Ltd
Current assignee: Beijing Shanma Zhijian Technology Co ltd; Hangzhou Shanma Zhiqing Technology Co Ltd; Shanghai Supremind Intelligent Technology Co Ltd
Priority date: 2023-11-07
Filing date: 2023-11-07
Publication date: 2023-12-08
Anticipated expiration: 2043-11-07
Also published as: CN117197507B

Abstract

The invention discloses a method and a device for determining an image block, a storage medium and an electronic device, wherein the method comprises the following steps: determining the motion attribute of a first image block in a current frame image through a target sub-template in a first template set, wherein the first template set comprises a plurality of sub-templates, and each sub-template in the plurality of sub-templates is used for updating the optimal matching point of the first image block in a previous frame image of the current frame image; determining a second template set corresponding to the first image block according to the motion attribute, wherein the first template set comprises: the second set of templates; the second set of templates includes the target sub-template; and determining a second image block matched with the first image block in the previous frame image according to the second template set. The method solves the problems of large calculated amount and low efficiency in the matching process of the image blocks in the related technology.

Description

Image block determining method and device, storage medium and electronic device

Technical Field

The present invention relates to the field of image processing technologies, and in particular, to a method and apparatus for determining an image block, a storage medium, and an electronic apparatus.

Background

Video shot change is a common scene in the video monitoring field, and the state that continuous motion occurs on a video picture caused by camera change cannot be perceived in real time. The video motion sensing method has wide application in the fields of video stability correction, video target matching and the like besides automatic scene distribution. In the prior art, for detecting the motion of a video lens, there are sensing methods such as a histogram method, a pixel matching difference method, edge extraction graph contrast, image frame background area contrast and the like. The method is based on global error matching of pictures, has obvious effect on detecting shot switching frames, but does not consider continuity and locality of real video motion, and cannot distinguish the direction of video motion and sense fine-granularity motion.

Further, video shot changes typically result in shifting or zooming in and out of the overall picture of the video. To distinguish local motion from global motion, the video global motion magnitude direction is counted based on motion estimation techniques. Motion estimation is an important technique in video coding standards, with important impact on coding speed, code rate and coding quality. In common and latest video coding standards such as h.264/h.265/av1/vp9, motion estimation is the highest in computational complexity in the whole system, and the time consumption is 60% -80% of the whole coding process.

Specifically, motion estimation refers to dividing each frame of a video sequence into non-overlapping image blocks, calculating each block to be matched to a reference frame in a search window according to a certain matching criterion, and finding out a block most similar to a current block, wherein the relative displacement between the matching block and the current block is a motion vector. The simplest motion Search algorithm is a Full Search algorithm (FS for short), that is, each point in the Search window is subjected to matching calculation, and the algorithm can accurately find the optimal matching block under the condition of proper Search range, but the calculation amount is large, so that the motion Search algorithm is not suitable for a scene of a large number of real-time transmissions. In order to quickly and accurately Search the motion vector of the image block, a plurality of quick Search algorithms are proposed, including early three-step Search templates, diamond Search templates, hexagonal Search templates, asymmetric cross-type multi-level hexagonal lattice Search algorithm (Unsymmetrical cross Multi-Hexagon-grid Search, abbreviated as UMHexagonS) and the like, and although the Search methods corresponding to the Search templates can reduce the operation amount by 80% compared with the full Search, the Search methods corresponding to the Search templates still have the defect of low motion estimation optimization efficiency.

Aiming at the problems of large calculation amount and low efficiency in the matching process of the image blocks in the related technology, no effective solution is proposed at present.

Accordingly, there is a need for improvements in the related art to overcome the drawbacks of the related art.

Disclosure of Invention

The embodiment of the invention provides a method and a device for determining an image block, a storage medium and an electronic device, which are used for at least solving the problems of large calculation amount and low efficiency in the matching process of the image block.

According to an aspect of an embodiment of the present invention, there is provided a method for determining an image block, including: determining the motion attribute of a first image block in a current frame image through a target sub-template in a first template set, wherein the first template set comprises a plurality of sub-templates, and each sub-template in the plurality of sub-templates is used for updating the optimal matching point of the first image block in a previous frame image of the current frame image; determining a second template set corresponding to the first image block according to the motion attribute, wherein the first template set comprises: the second set of templates; the second set of templates includes the target sub-template; and determining a second image block matched with the first image block in the previous frame image according to the second template set.

In one exemplary embodiment, determining, by a target sub-template in a first set of templates, a motion attribute of a first image block in a current frame image includes: taking a first search center point corresponding to the first image block as an origin, determining m points to be matched in the first image block according to the target sub-template, and determining m matching points in the previous frame image, wherein the points to be matched in the m points to be matched and the matching points in the m points to be matched have a one-to-one correspondence, and m is a positive integer; and determining the motion attribute of the first image block according to the m points to be matched, the m matching points and the first searching center point.

In an exemplary embodiment, determining the motion attribute of the first image block according to the m points to be matched, and the first search center point includes: matching the m points to be matched with the m matching points to obtain m matching values; determining a smallest matching value of the m matching values; and determining the motion attribute of the first image block according to the ratio of the minimum matching value to the origin matching value corresponding to the first search center point.

In an exemplary embodiment, determining a second set of templates corresponding to the first image block according to the motion attribute includes: determining that only the target sub-template is included in the second template set under the condition that the motion attribute is a first amplitude motion, wherein the first amplitude motion is a motion with a ratio larger than a first value, and the ratio is determined according to the target sub-template, and is a ratio of a minimum matching value of the first image block and the previous frame image to an original point matching value; determining x sub-templates in the plurality of sub-templates under the condition that the motion attribute is second amplitude motion, and determining a second template set according to the x sub-templates and the target sub-template, wherein the x sub-templates are sub-templates used for updating the best matching point in a small range in the plurality of sub-templates, and x is a positive integer, wherein the second amplitude motion is a motion with the ratio smaller than a second value and larger than a third value, and the first value is larger than the second value and the second value is larger than the third value; and determining the first template set as the second template set in the case that the motion attribute is a third-amplitude motion, wherein the third-amplitude motion is a motion in which the ratio is greater than a second value, less than a first value, or less than a third value.

In an exemplary embodiment, determining a second image block in the previous frame image that matches the first image block according to the second set of templates includes: determining a first matching point, wherein the first matching point is a minimum matching point of the first image block in the previous frame image, which is determined according to y sub-templates in the plurality of sub-templates, wherein the y sub-templates all take a first searching center point as an original point, the first searching center point is an initial searching center point of the first image block, the y sub-templates comprise the target sub-templates, y is a positive integer, and the y sub-templates and the x sub-templates do not comprise the same sub-templates; updating the first search center point through the first matching point to obtain a second search center point; and determining a target matching point through the x sub-templates and the second searching center point, and searching in the previous frame image according to the target matching point to obtain the second image block, wherein the target matching point is the best matching point obtained by final updating.

In an exemplary embodiment, determining the first matching point from y sub-templates of the plurality of sub-templates includes: determining a second matching point, wherein the second matching point is the minimum matching point of the first image block in the previous frame image, which is determined according to the target sub-template; determining a third matching point under the condition that y is greater than 1, wherein the third matching point is the minimum matching point of the first image block in the previous frame image, which is determined according to a current sub-template, and the current sub-template is any sub-template except the target sub-template in the y sub-templates; updating the third matching point to the first matching point under the condition that the matching value of the third matching point is smaller than that of the second matching point; and determining the second matching point as the first matching point in the case that the matching value of the third matching point is larger than the matching value of the second matching point.

In an exemplary embodiment, determining a target match point from the x sub-templates and the second search center point includes: determining the order in which the x sub-templates are used; in the case that the best matching points of the first image block are updated through the x sub-templates in the order of use, matching is performed on any one of the x sub-templates based on the second search center point to determine a current best matching point, and the second search center point is updated through the current best matching point; and determining the current searching center point as the target matching point under the condition that the current best matching point coincides with the second searching center point.

According to another aspect of the embodiment of the present invention, there is also provided an apparatus for determining an image block, including: a first determining module, configured to determine a motion attribute of a first image block in a current frame image through a target sub-template in a first template set, where the first template set includes a plurality of sub-templates, and each sub-template in the plurality of sub-templates is configured to update a best matching point of the first image block in a previous frame image of the current frame image; a second determining module, configured to determine a second template set corresponding to the first image block according to the motion attribute, where the first template set includes: the second set of templates; the second set of templates includes the target sub-template; and a third determining module, configured to determine, according to the second template set, a second image block that matches the first image block in the previous frame image.

According to a further aspect of embodiments of the present invention, there is also provided a computer readable storage medium having a computer program stored therein, wherein the computer program is arranged to perform the above-described method of determining image blocks when run.

According to still another aspect of the embodiments of the present invention, there is further provided an electronic device including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the method for determining the image block by using the computer program.

According to the method and the device, the motion attribute of the first image block in the current frame image is determined through the target sub-templates in the first template set, wherein the first template set comprises a plurality of sub-templates, and each sub-template in the plurality of sub-templates is used for updating the optimal matching point of the first image block in the previous frame image of the current frame image; determining a second template set corresponding to the first image block according to the motion attribute, wherein the first template set comprises: the second set of templates; the second set of templates includes the target sub-template; and determining a second image block matched with the first image block in the previous frame image according to the second template set. By adopting the technical scheme, the second template set corresponding to the first image block is determined by determining the motion attribute of the first image block, and then for the first image block with different motion attributes, the second image block matched with the first image block in the previous frame image can be determined through the second template set, so that the problems of large calculation amount and low efficiency in the matching process of the image block are solved, and in the process of motion estimation, the second template set corresponding to the first image block can be selected in a targeted manner according to the motion attribute of the first image block, and the efficiency of motion estimation can be improved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention and do not constitute a limitation on the invention. In the drawings:

fig. 1 is a hardware block diagram of a computer terminal of a method of determining an image block according to an embodiment of the present invention;

FIG. 2 is a flow chart of a method of determining image blocks according to an embodiment of the invention;

FIG. 3 is a flow chart of steps of a method of determining image blocks according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of a small diamond template for a method of determining image blocks according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of a large diamond template for a method of determining image blocks according to an embodiment of the present invention;

FIG. 6 is a schematic diagram of a non-cross-symmetric template of a method of determining image blocks according to an embodiment of the present invention;

FIG. 7 is a schematic diagram of a large hexagonal template of a method of determining image blocks according to an embodiment of the present invention;

FIG. 8 is a schematic diagram of a hexagonal template of a method of determining image blocks according to an embodiment of the present invention;

FIG. 9 is a diamond-shaped template diagram of a method of determining image blocks according to an embodiment of the invention;

Fig. 10 is a block diagram of a configuration of a determination device of an image block according to an embodiment of the present invention.

Detailed Description

In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.

It should be noted that the terms "first," "second," and the like in the description and the claims of the present invention and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

The method embodiments provided in the embodiments of the present invention may be executed in a computer terminal or similar computing device. Taking the example of running on a computer terminal, fig. 1 is a block diagram of the hardware structure of the computer terminal of the image block determining method according to the embodiment of the present invention. As shown in fig. 1, the computer terminal may include one or more (only one is shown in fig. 1) processors 102 (the processor 102 may include, but is not limited to, a microprocessor (Microprocessor Unit, abbreviated MPU) or programmable logic device (Programmable logic device, abbreviated PLD)) and a memory 104 for storing data, and in an exemplary embodiment, the computer terminal may further include a transmission device 106 for communication functions and an input-output device 108. It will be appreciated by those skilled in the art that the configuration shown in fig. 1 is merely illustrative and is not intended to limit the configuration of the computer terminal described above. For example, a computer terminal may also include more or fewer components than shown in FIG. 1, or have a different configuration than the equivalent functions shown in FIG. 1 or more than the functions shown in FIG. 1.

The memory 104 may be used to store a computer program, for example, a software program of application software and a module, such as a computer program corresponding to a method for determining an image block in an embodiment of the present invention, and the processor 102 executes the computer program stored in the memory 104 to perform various functional applications and data processing, that is, to implement the above-described method. Memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory remotely located relative to the processor 102, which may be connected to the computer terminal via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The transmission means 106 is arranged to receive or transmit data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of a computer terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, simply referred to as NIC) that can connect to other network devices through a base station to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is used to communicate with the internet wirelessly.

In order to solve the above-mentioned problems, in the present embodiment, there is provided a method for determining an image block, which is applied to the above-mentioned computer terminal, fig. 2 is a flowchart of a method for determining an image block according to an embodiment of the present invention, the flowchart including the steps of:

step S202, determining the motion attribute of a first image block in a current frame image through a target sub-template in a first template set, wherein the first template set comprises a plurality of sub-templates, and each sub-template in the plurality of sub-templates is used for updating the best matching point of the first image block in a previous frame image of the current frame image;

Step S204, determining a second template set corresponding to the first image block according to the motion attribute, where the first template set includes: the second set of templates; the second set of templates includes the target sub-template;

step S206, determining a second image block matching the first image block in the previous frame image according to the second template set.

Determining the motion attribute of a first image block in a current frame image through a target sub-template in a first template set, wherein the first template set comprises a plurality of sub-templates, and each sub-template in the plurality of sub-templates is used for updating the best matching point of the first image block in a previous frame image of the current frame image; determining a second template set corresponding to the first image block according to the motion attribute, wherein the first template set comprises: the second set of templates; the second set of templates includes the target sub-template; and determining a second image block matched with the first image block in the previous frame image according to the second template set. By adopting the technical scheme, the second template set corresponding to the first image block is determined by determining the motion attribute of the first image block, and then for the first image block with different motion attributes, the second image block matched with the first image block in the previous frame image can be determined through the second template set, so that the problems of large calculation amount and low efficiency in the matching process of the image block are solved, and in the process of motion estimation, the second template set corresponding to the first image block can be selected in a targeted manner according to the motion attribute of the first image block, and the efficiency of motion estimation can be improved.

Optionally, before the step S202, that is, before determining the motion attribute of the first image block in the current frame image by using the target sub-template in the first template set, the method further includes: determining a global search range of the first image block in the previous frame image by one of: receiving the global search range set by a target object for the first image block; and obtaining an average value of the motion vectors of the previous frame of image, and determining the global search range according to the average value.

Optionally, obtaining an average value of motion vectors of the previous frame image, and determining the global search range according to the average value includes: the global search scope is determined by the following formula:

；

where SA represents the global search scope,the average values c1, c2, and c3 are constant, wherein c1, c2, and c3 can be adjusted according to statistics, and preferably, can be set to 3, and 1 in sequence.

It will be appreciated that 1) in the case where the current frame image does not have the previous frame image, the global search range may be set by a target object, where the target object may be a user of an image capturing apparatus with a camera, a researcher in the field of video motion estimation, or the like. In the case of manual setting, the initial value of the global search range may be set to-128 to 128, and the corresponding search range may preferably be 1/8 of the first image block of the segmented current frame image. 2) Under the condition that the current frame image exists in the previous frame image, the global search range can be determined according to the previous frame image, and further compared with a scheme of manually setting the global search range, a reasonable global search range can be obtained through the average value of the motion vectors of the previous frame, the calculated amount of a video motion estimation method is further reduced, and the estimation efficiency is further improved. It should be noted that, the previous frame image and the current frame image may be continuous (or discontinuous, but there is a certain motion association) two frame images that are cut from continuous video, and the previous frame image may also be an independent reference frame image that is only used as a reference.

The motion attribute of the first image block can be determined according to the m points to be matched, m matching values corresponding to the m matching points and an origin matching value corresponding to the first search center point.

Alternatively, the first search center point corresponding to the first image block may be determined by using an intra-frame prediction or inter-frame prediction method.

Alternatively, the first search center point of the first image block may be determined according to the search center point of the first image block that is immediately before the first image block that is currently being searched, that is, according to the relative displacement between the first image block and the immediately before first image block, and the search center point of the immediately before first image block, specifically, the search center point of the immediately before first image block (the third image block) is moved according to the relative displacement, so as to obtain the first search center point. The motion estimation efficiency can also be improved to some extent.

Alternatively, the motion attribute may be determined by the following formula:

；

wherein DA represents the ratio, Representing said minimum match value, +.>Representing the origin matching value.

Illustratively, the target search template that determines the motion attribute is preferably a small diamond-shaped template having 4 search points (i.e., m is equal to 4). According to the small diamond template, taking the first searching center point as an origin, determining 4 points to be matched corresponding to the small diamond template in the first image block and 4 matching points corresponding to the small diamond template in the previous frame image, determining 4 matching points and 4 matching values of the points to be matched, and further determining the minimum matching value in the 4 matching values; and determining an origin matching value corresponding to the first search center point, and taking the ratio of the minimum matching value to the origin matching value as a basis for determining the motion attribute.

Further, determining the motion attribute of the first image block according to the ratio of the minimum matching value to the origin matching value includes: acquiring a motion attribute corresponding table, wherein the motion attribute corresponding table records the corresponding relation between the ratio and the motion attribute; and inquiring the motion attribute of the first image block from the motion attribute corresponding table according to the ratio.

It will be appreciated that the attribute of motion, and in particular the attribute of motion, may be determined from the home range of the ratios in the previous embodiments: 1) And under the condition that the ratio is larger than or equal to a first value, the motion attribute of the first image is a first amplitude motion. Alternatively, the first value is preferably 3, and the first amplitude motion may be understood as a stationary motion, and further, for the first image block belonging to the first amplitude motion, after determining the motion attribute according to the target sub-template, the surrounding image blocks may be searched for performing state verification. 2) In case the ratio is larger than the third value and smaller than the second value, the motion property of the first image block may be determined to be a second amplitude motion. Alternatively, the second value is preferably 0.6 and the third value is preferably 0.3, the second amplitude movement being understood as a small amplitude movement. The small motion may employ a target sub-template in combination with x sub-templates of the plurality of sub-templates for a small range search to perform a search process for the second image block. 3) For the rest of the ratios which are not in the interval to which the ratio of the first amplitude motion and the second amplitude motion belongs, the corresponding first image block can be confirmed as a third amplitude motion, and the third amplitude motion can be understood as a large amplitude motion. For large-scale motion, to accurately determine the second image block corresponding to the first image block, the first template set may be directly used as the second template set.

The first template set includes a double diamond template (small diamond template, large diamond template), a non-cross symmetric template, an eight-point large hexagon template, a small hexagon template and a diamond template, wherein the target sub-template is the small diamond template, and the large diamond template and the non-cross symmetric template can be used as y sub-templates (at this time, y=3); the small hexagonal template serves as an iterative template and the diamond template together serve as templates dedicated to small-scale searches (i.e., corresponding to the x sub-templates, where x=2). Further, determining the second image block includes: 1) Determining a motion attribute through the small diamond template, ending the searching process of the second template set under the condition that the motion attribute is static motion, and then searching only the first image blocks around for verification; 2) Under the condition that the motion attribute is large motion, a first matching point (namely, the minimum matching point of a first image block corresponding to the three sub-templates of the small diamond template, the large diamond template and the non-cross symmetric template) is determined together through the small diamond template, the large diamond template and the non-cross symmetric template; updating the first search center point through the first matching point to obtain a second search center point; further, the second searching center point is updated sequentially through the eight-point large hexagon template, the small hexagon template and the diamond template, the target matching point corresponding to the second searching center point can be further determined, and the second image block is determined through the target matching point; 3) In the case that the motion attribute is small-amplitude motion, the difference from the searching process of large-amplitude motion is that the small-amplitude motion only determines a first matching point according to the target sub-template, and then updates the first searching center point to obtain a second matching point, and the rest processes are the same as the large-amplitude motion.

Alternatively, in the case where y=1, that is, in the case where the motion attribute is small motion and only the target sub-template is included in the y sub-templates, the second matching point is determined as the first matching point.

Illustratively, where y=3, determining the first matching point by the small diamond template, the large diamond template, and the non-cross symmetric template collectively includes: and determining a second matching point through the small diamond template, determining a third matching point through the large diamond template (and/or the non-cross symmetrical template), comparing the matching value of the third matching point with the matching value of the second matching point, and taking the smaller matching point as the first matching point.

It should be noted that, among the y sub-templates, the target sub-template is used to determine the motion attribute, and the order of use of the remaining y-1 sub-templates is not limited. The above-mentioned determination scheme of the first matching value can be further understood as follows: and determining a common minimum matching value (fourth matching value) of the y sub-templates by taking the first search center point as an origin, and taking the first search center point as a first matching value.

It should be noted that, the x sub templates are in use sequence, for example: in the case of x=2, the small hexagonal templates and the small hexagonal templates in the diamond-shaped templates are used as the first used sub-templates, and the diamond-shaped templates are used as the second used sub-templates.

Under the condition that a small hexagon is used, determining the minimum matching point (current best matching point) corresponding to 6 searching points of the small hexagon by taking a second searching center as an original point through a small hexagon template, updating the second searching center point through the current best matching point, and continuously and circularly iterating to use the small hexagon template to finally obtain a target matching point.

Optionally, in the case of using the diamond template, only the determining process of the minimum matching point corresponding to the diamond template is performed, and the minimum matching point corresponding to the diamond template is determined as the target matching point.

Optionally, n sub-templates may also be employed for more accurate positioning of the second search center point before x sub-templates are used. The using sequence of n sub templates is determined, the smallest matching point (fifth matching point) corresponding to the current sub template in the n sub templates is determined according to the using sequence, and the second searching center point is updated through the smallest matching point corresponding to the current sub template in the n sub templates, wherein n is a positive integer. In fact, each of the n sub-templates continuously updates the second search center point, and the updated second search center point is used as the search origin of the next sub-template of the n sub-templates. For example, in the case where n=1, that is, the eight-point large hexagonal template is used with the first search center as the origin, the second search center point is determined and updated by the minimum matching point corresponding to the eight-point large hexagonal template.

It will be apparent that the embodiments described above are merely some, but not all, embodiments of the invention. In order to better understand the above-mentioned method for determining the image block, the following description will explain the above-mentioned process with reference to the embodiments, but is not intended to limit the technical solution of the embodiments of the present invention, specifically:

in order to realize video motion perception, the embodiment of the invention converts the motion perception problem into the problem of motion estimation plus overall motion duty ratio, motion amplitude and motion direction judgment. The key technology is how to perform fast motion estimation.

Specifically, the embodiment of the invention provides a fast motion state sensing scheme based on motion estimation, specifically as shown in fig. 3, including:

step S301, a current frame and a reference frame are input.

Step S302, determining a global search scope.

Firstly, image blocks are subjected to motion estimation, the initial value of the searching range of the image blocks (which is equivalent to the global searching range in the embodiment) is set to 128-128, and for rapid motion estimation, statistics finds that the searching step length is 1/8 of the size of the image blocks, so that the approximate accurate prediction can be realized.

Updating the global search range of the image: the motion amplitude of most frames in the natural video sequence is small, and the motion state between adjacent frames has continuity, so that the global search range of the frame can be limited according to the mv of the previous frame. Determination formula of global search scope:

；

Wherein,as the average value of mv amplitude of the previous frame, c1, c2, c3 are constant and are set to 3, 1 respectively. Can be adjusted according to the statistics.

Step S303, determining an image block to be matched.

In step S304, an image block initial Motion Vector (mv) is predicted.

And predicting the mv initial value of each image block. Intra prediction or inter prediction methods commonly used by encoders may be employed.

Step S305, determining whether the number of blocks to be predicted (corresponding to the first image block in the above embodiment) is less than 12% of the total number, and if so, proceeding to step S306; if it is equal to or greater than the predetermined value, the process advances to step S307.

Step S306 determines that the current frame (corresponding to the current frame image in the above embodiment) is the same as the previous frame (corresponding to the previous frame image in the above embodiment). Directly proceed to step S312.

Step S307, determining an image block motion state, wherein the motion attribute determined in step S307 includes: step S3071, standing; step S3072, small-amplitude movement; step S3073, the motion is large. Wherein the stationary movement proceeds to step S310, and the small-amplitude movement and the large-amplitude movement proceed to step S308.

Optionally, judging the motion attribute of the image block: and judging the motion attribute of the image block according to the matching result difference in the local range of the image block, wherein the motion attribute is divided into three types, namely static, small-amplitude motion and large-amplitude motion. Defining motion attributes by distortion ratio

；

Taking 960x540 video as an example,can be set to step for minimum SAD (sum of absolute differences) calculated for 2 small diamond templates,/L>Is the SAD of the origin. Statistics shows that the DA value has no obvious correlation with the video content, and when DA is more than or equal to 3, the image block is a still image block; 0.3<DA<At 0.6 the image block is small motion and the motion vector points near the minimum. Other types of motion are gross motion.

Step S308, updating the search range and the step size.

Step S309, the da_umh algorithm searches.

Before DA_UMH algorithm search is carried out, UMH motion templates in the prior art are optimized, and the method specifically comprises the following steps: the UMH algorithm (namely UMH moving templates) universal templates are respectively square templates, non-cross symmetrical templates, sixteen-point large-hexagon templates, small-hexagon templates and diamond templates according to stages. The optimized templates are respectively a double-diamond template, a non-cross symmetrical template, an eight-point large-hexagon template, a small-hexagon template and a diamond template according to the stages. The computational complexity of motion estimation with the optimized UMH template is shown in table 1 below.

Dynamically selecting a search algorithm (i.e., selecting an optimized UMH template) according to the motion attribute: when the video sequence moves severely, correct blocks can be quickly and accurately matched by adopting the UMH searching algorithm, but when the video moves slowly, repeated complex searching is carried out for a plurality of times by adopting the UMH algorithm, and long-distance mismatching is easy to occur. Thus, different search templates may be employed for motion estimation for image blocks of different motion attributes. The still image block only needs to search the surrounding image blocks for state verification; the small-sized moving image block only needs to adopt an iteration hexagonal template and a diamond template to predict in a small range; the large-scale motion is predicted by adopting an optimized UMH algorithm.

Optionally, the embodiment of the invention can dynamically set the search step size: aiming at the scene of quickly solving the motion state, the step length can be dynamically adjusted according to the search range, so that the search speed is increased.

Step S310, outputting a motion vector.

Step S311, the status is determined based on the full frame (corresponding to the current frame image in the above embodiment) mv.

Step S312, the motion state is output.

And judging the motion state of the frame according to the dynamic judgment strategy. The basic judgment conditions are a duty ratio of the current frame (corresponding to the current frame image in the above embodiment) mv to be larger than the average amplitude of the still frame (corresponding to the previous frame image in the above embodiment), the amplitude of the average mv, and an angular change of the average mv. And (3) setting a moving window threshold value for judging continuous rotation and small-range jitter of the video, wherein the motion segment exceeds 5 frames and the motion direction is consistent, and if the motion segment is continuous rotation, otherwise, the motion segment is small-range jitter.

Specifically, the specific implementation steps of motion state sensing through the optimized UMH algorithm are as follows:

step S401: a global search scope is determined.

Step S402: and filtering out image blocks which do not need to be matched according to service requirements, wherein part of the image blocks are too flat and have no useful texture information, so that global judgment can be influenced.

Step S403: the initial mv (inter prediction or intra prediction) is determined, and the search center point (MBD point) is updated. Wherein the MBD point is Motion-Compensated Block Difference point. MBD points are a technical term used to describe video compression and motion estimation. It refers to estimating motion by comparing one motion compensated block with a corresponding block in a reference frame and calculating the difference between them in each frame in the video. This difference is commonly referred to as MBD value, representing the degree of motion difference between the two blocks.

Step S404 specifically includes the following steps S4041-S4043:

step S4041: the small diamond template is searched, and the searching template is shown in fig. 4. The specific steps are that the same point of the coordinate position of the reference frame and the current MBD point is used as a search center, the point of the small diamond template (4 points) with the search center as the origin is matched with the block to be matched, and the minimum sad point is recorded. And calculating motion attribute according to the minimum sad point to origin sad ratio, and terminating the search by the still image block. Where SAD stands for "Sum of Absolute Differences (sum of absolute differences).

Step S4042: large diamond template search, the search template is shown in fig. 5. The small motion skips this step. The specific steps are that the point where the large diamond template (8 points) takes the searching center as the origin is located is matched with the block to be matched, sad is calculated and compared with the minimum sad point in the previous step, and a new minimum sad point is recorded.

Step S4043: the asymmetric cross template search, the small motion skips this step. The specific steps are that the matching operation is carried out on the points of the template shown in the figure 6 in the horizontal direction and the vertical direction which take the searching center as the origin, the matching operation is carried out on the points and the blocks to be matched, and compared with the minimum sad point in the previous step, the new minimum sad point is recorded. The MBD point is updated to the best match point of steps S4041-S4043.

Step S405 includes the following steps S4051-S4053:

step S4051: the method comprises the specific steps of searching by using a multi-layer improved large hexagon template, namely taking the same point of the coordinate position of a reference frame and the current MBD point as a search center, carrying out matching operation on the point of the multi-layer large hexagon template with the search center as an original point according to the template shown in fig. 7, comparing the point with a block to be matched, and recording a new minimum sad point. The MBD point is updated as the best match point for the current step.

Step S4052: the iterations use hexagonal searches. The specific steps are that the same point of the coordinate position of the reference frame and the current MBD point is used as a search center, matching operation is carried out on the points of the hexagonal template with the search center as the origin according to the template shown in fig. 8, the points are matched with the block to be matched, the optimal matching point is recorded, the MBD point is moved to the optimal matching point, the matching steps are repeated until the optimal point is at the center, and the MBD point is updated to be the optimal matching point of the current step.

Step S4053: diamond search, where the matching search has been substantially completed, searches for four points around the best point with diamond templates as shown in fig. 9, to search for more accurate matching blocks.

Step S406: the motion state of the frame (corresponding to the first image block in the above embodiment) is judged according to the dynamic judgment policy.

Compared with the traditional motion estimation method, the method provided by the embodiment of the invention has the following advantages:

1) The motion between two frames of the video has obvious relativity and consistency, and the searching range of the current frame can be limited according to the motion vector of the previous frame of the video. The search range can be reduced, and the mismatching caused by the calculation error of the matching rule can be reduced.

2) The image blocks are subjected to motion grading, the searching range and the searching step length of the image blocks can be dynamically adjusted, different searching templates are selected according to the searching range and the searching step length, repeated calculation of the searching templates can be reduced, the optimal matching blocks can be found more quickly, and the calculated amount of slightly moving image blocks can be reduced by more than 90% compared with UMH algorithm.

3) For the whole video, the dynamic-UMH searching algorithm (namely DA_UMH algorithm) can save more than 80% of calculated amount compared with the original UMH algorithm on average, and can save more than 90% of calculated amount for adjusting the initial step length of a scene requiring quick solving motion.

4) The motion estimation method can be applied to the video rotation sensing direction, can achieve 16 paths of video in real time in a single cpu scene, and has the accuracy and recall rate equivalent to those of other general search algorithms. Overall calculated amount comparison is as follows table 1:

table 1: comparison of block calculation times of several search algorithms

5) The motion estimation in the embodiment of the invention is applied to the field of visual motion perception, and whether the image frame has lens change relative to the previous frame is judged through motion estimation. For other motion estimation application scenes such as video encoding and decoding, lens shake correction and target matching, etc., the application can be performed after the relevant parameters are adjusted.

The embodiment also provides a device for determining an image block, which is used for implementing the above embodiment and the preferred implementation, and is not described in detail. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. While the devices described in the following embodiments are preferably implemented in software, implementations in hardware, or a combination of software and hardware, are also possible and contemplated.

Fig. 10 is a block diagram of a configuration of an image block determining apparatus according to an embodiment of the present invention, the apparatus including:

A first determining module 1002, configured to determine a motion attribute of a first image block in a current frame image through a target sub-template in a first template set, where the first template set includes a plurality of sub-templates, and each sub-template in the plurality of sub-templates is configured to update a best matching point of the first image block in a previous frame image of the current frame image;

a second determining module 1004 is configured to determine a second template set corresponding to the first image block according to the motion attribute, where the first template set includes: the second set of templates; the second set of templates includes the target sub-template;

a third determining module 1006 configured to determine, according to the second template set, a second image block in the previous frame image that matches the first image block

The device determines the motion attribute of a first image block in a current frame image through a target sub-template in a first template set, wherein the first template set comprises a plurality of sub-templates, and each sub-template in the plurality of sub-templates is used for updating the optimal matching point of the first image block in a previous frame image of the current frame image; determining a second template set corresponding to the first image block according to the motion attribute, wherein the first template set comprises: the second set of templates; the second set of templates includes the target sub-template; and determining a second image block matched with the first image block in the previous frame image according to the second template set. By adopting the technical scheme, the second template set corresponding to the first image block is determined by determining the motion attribute of the first image block, and then for the first image block with different motion attributes, the second image block matched with the first image block in the previous frame image can be determined through the second template set, so that the problems of large calculation amount and low efficiency in the matching process of the image block are solved, and in the process of motion estimation, the second template set corresponding to the first image block can be selected in a targeted manner according to the motion attribute of the first image block, and the efficiency of motion estimation can be improved.

In an exemplary embodiment, the first determining module 1002 is further configured to determine m points to be matched in the first image block according to the target sub-template with a first search center point corresponding to the first image block as an origin, and determine m matching points in the previous frame image, where there is a one-to-one correspondence between a point to be matched in the m points to be matched and a matching point in the m matching points, where m is a positive integer; and determining the motion attribute of the first image block according to the m points to be matched, the m matching points and the first searching center point.

In an exemplary embodiment, the first determining module 1002 is further configured to match the m points to be matched with the m matching points to obtain m matching values; determining a smallest matching value of the m matching values; and determining the motion attribute of the first image block according to the ratio of the minimum matching value to the origin matching value corresponding to the first search center point.

In an exemplary embodiment, the second determining module 1004 is further configured to determine that only the target sub-template is included in the second template set if the motion attribute is a first amplitude motion, where the first amplitude motion is a motion with a ratio greater than a first value, where the ratio is determined according to the target sub-template, and a ratio of a minimum matching value of the first image block and the previous frame image to an origin matching value; determining x sub-templates in the plurality of sub-templates under the condition that the motion attribute is second amplitude motion, and determining a second template set according to the x sub-templates and the target sub-template, wherein the x sub-templates are sub-templates used for updating the best matching point in a small range in the plurality of sub-templates, and x is a positive integer, wherein the second amplitude motion is a motion with the ratio smaller than a second value and larger than a third value, and the first value is larger than the second value and the second value is larger than the third value; and determining the first template set as the second template set in the case that the motion attribute is a third-amplitude motion, wherein the third-amplitude motion is a motion in which the ratio is greater than a second value, less than a first value, or less than a third value.

In an exemplary embodiment, the third determining module 1006 is further configured to determine a first matching point, where the first matching point is a minimum matching point of the first image block in the previous frame image determined according to y sub-templates of the plurality of sub-templates, where the y sub-templates each use a first search center point as an origin, the first search center point is an initial search center point of the first image block, the y sub-templates include the target sub-template, where y is a positive integer, and the y sub-templates and the x sub-templates do not include the same sub-template; updating the first search center point through the first matching point to obtain a second search center point; and determining a target matching point through the x sub-templates and the second searching center point, and searching in the previous frame image according to the target matching point to obtain the second image block, wherein the target matching point is the best matching point obtained by final updating.

In an exemplary embodiment, the third determining module 1006 is further configured to determine a second matching point, where the second matching point is a minimum matching point of the first image block in the previous frame image, which is determined according to the target sub-template; determining a third matching point under the condition that y is greater than 1, wherein the third matching point is the minimum matching point of the first image block in the previous frame image, which is determined according to a current sub-template, and the current sub-template is any sub-template except the target sub-template in the y sub-templates; updating the third matching point to the first matching point under the condition that the matching value of the third matching point is smaller than that of the second matching point; and determining the second matching point as the first matching point in the case that the matching value of the third matching point is larger than the matching value of the second matching point.

In an exemplary embodiment, the third determining module 1006 is further configured to determine a used order of the x sub-templates; in the case that the best matching points of the first image block are updated through the x sub-templates in the order of use, matching is performed on any one of the x sub-templates based on the second search center point to determine a current best matching point, and the second search center point is updated through the current best matching point; and determining the current searching center point as the target matching point under the condition that the current best matching point coincides with the second searching center point.

Embodiments of the present invention also provide a computer readable storage medium having a computer program stored therein, wherein the computer program is arranged to perform the steps of any of the method embodiments described above when run.

Alternatively, in the present embodiment, the above-described storage medium may be configured to store a computer program for performing the steps of:

s1, determining the motion attribute of a first image block in a current frame image through a target sub-template in a first template set, wherein the first template set comprises a plurality of sub-templates, and each sub-template in the plurality of sub-templates is used for updating the optimal matching point of the first image block in a previous frame image of the current frame image;

S2, determining a second template set corresponding to the first image block according to the motion attribute, wherein the first template set comprises: the second set of templates; the second set of templates includes the target sub-template;

and S3, determining a second image block matched with the first image block in the previous frame image according to the second template set.

In one exemplary embodiment, the computer readable storage medium may include, but is not limited to: a usb disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a removable hard disk, a magnetic disk, or an optical disk, or other various media capable of storing a computer program.

Specific examples in this embodiment may refer to the examples described in the foregoing embodiments and the exemplary implementation, and this embodiment is not described herein.

An embodiment of the invention also provides an electronic device comprising a memory having stored therein a computer program and a processor arranged to run the computer program to perform the steps of any of the method embodiments described above.

Alternatively, in the present embodiment, the above-described processor may be configured to execute the following steps by a computer program:

In an exemplary embodiment, the electronic apparatus may further include a transmission device connected to the processor, and an input/output device connected to the processor.

It will be appreciated by those skilled in the art that the modules or steps of the invention described above may be implemented in a general purpose computing device, they may be concentrated on a single computing device, or distributed across a network of computing devices, they may be implemented in program code executable by computing devices, so that they may be stored in a storage device for execution by computing devices, and in some cases, the steps shown or described may be performed in a different order than that shown or described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple modules or steps of them may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.

The above description is only of the preferred embodiments of the present invention and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method of determining an image block, comprising:

determining the motion attribute of a first image block in a current frame image through a target sub-template in a first template set, wherein the first template set comprises a plurality of sub-templates, and each sub-template in the plurality of sub-templates is used for updating the optimal matching point of the first image block in a previous frame image of the current frame image;

determining a second template set corresponding to the first image block according to the motion attribute, wherein the first template set comprises: the second set of templates; the second set of templates includes the target sub-template;

and determining a second image block matched with the first image block in the previous frame image according to the second template set.

2. The method of claim 1, wherein determining the motion attribute of the first image block in the current frame image from the target sub-templates in the first set of templates comprises:

Taking a first search center point corresponding to the first image block as an origin, determining m points to be matched in the first image block according to the target sub-template, and determining m matching points in the previous frame image, wherein the points to be matched in the m points to be matched and the matching points in the m points to be matched have a one-to-one correspondence, and m is a positive integer;

and determining the motion attribute of the first image block according to the m points to be matched, the m matching points and the first searching center point.

3. The method of claim 2, wherein determining the motion attribute of the first image block from the m points to be matched, and the first search center point comprises:

matching the m points to be matched with the m matching points to obtain m matching values;

determining a smallest matching value of the m matching values;

and determining the motion attribute of the first image block according to the ratio of the minimum matching value to the origin matching value corresponding to the first search center point.

4. The method of claim 1, wherein determining a second set of templates corresponding to the first image block from the motion attribute comprises:

Determining that only the target sub-template is included in the second template set under the condition that the motion attribute is a first amplitude motion, wherein the first amplitude motion is a motion with a ratio larger than a first value, and the ratio is determined according to the target sub-template, and is a ratio of a minimum matching value of the first image block and the previous frame image to an original point matching value;

determining x sub-templates in the plurality of sub-templates under the condition that the motion attribute is second amplitude motion, and determining a second template set according to the x sub-templates and the target sub-template, wherein the x sub-templates are sub-templates used for updating the best matching point in a small range in the plurality of sub-templates, and x is a positive integer, wherein the second amplitude motion is a motion with the ratio smaller than a second value and larger than a third value, and the first value is larger than the second value and the second value is larger than the third value;

and determining the first template set as the second template set in the case that the motion attribute is a third-amplitude motion, wherein the third-amplitude motion is a motion in which the ratio is greater than a second value, less than a first value, or less than a third value.

5. The method of claim 4, wherein determining a second image block of the previous frame image that matches the first image block based on the second set of templates comprises:

determining a first matching point, wherein the first matching point is a minimum matching point of the first image block in the previous frame image, which is determined according to y sub-templates in the plurality of sub-templates, wherein the y sub-templates all take a first searching center point as an original point, the first searching center point is an initial searching center point of the first image block, the y sub-templates comprise the target sub-templates, y is a positive integer, and the y sub-templates and the x sub-templates do not comprise the same sub-templates;

updating the first search center point through the first matching point to obtain a second search center point;

and determining a target matching point through the x sub-templates and the second searching center point, and searching in the previous frame image according to the target matching point to obtain the second image block, wherein the target matching point is the best matching point obtained by final updating.

6. The method of claim 5, wherein determining a first matching point from y of the plurality of sub-templates comprises:

Determining a second matching point, wherein the second matching point is the minimum matching point of the first image block in the previous frame image, which is determined according to the target sub-template;

determining a third matching point under the condition that y is greater than 1, wherein the third matching point is the minimum matching point of the first image block in the previous frame image, which is determined according to a current sub-template, and the current sub-template is any sub-template except the target sub-template in the y sub-templates;

updating the third matching point to the first matching point under the condition that the matching value of the third matching point is smaller than that of the second matching point;

and determining the second matching point as the first matching point in the case that the matching value of the third matching point is larger than the matching value of the second matching point.

7. The method of claim 5, wherein determining a target match point from the x sub-templates and the second search center point comprises:

determining the order in which the x sub-templates are used;

in the case that the best matching points of the first image block are updated through the x sub-templates in the order of use, matching is performed on any one of the x sub-templates based on the second search center point to determine a current best matching point, and the second search center point is updated through the current best matching point;

And determining the current searching center point as the target matching point under the condition that the current best matching point coincides with the second searching center point.

8. An image block determining apparatus, comprising:

a first determining module, configured to determine a motion attribute of a first image block in a current frame image through a target sub-template in a first template set, where the first template set includes a plurality of sub-templates, and each sub-template in the plurality of sub-templates is configured to update a best matching point of the first image block in a previous frame image of the current frame image;

a second determining module, configured to determine a second template set corresponding to the first image block according to the motion attribute, where the first template set includes: the second set of templates; the second set of templates includes the target sub-template;

and a third determining module, configured to determine, according to the second template set, a second image block that matches the first image block in the previous frame image.

9. A computer readable storage medium, characterized in that the computer readable storage medium comprises a stored program, wherein the program when run performs the method of any of the preceding claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that the memory has stored therein a computer program, the processor being arranged to execute the method according to any of the claims 1 to 7 by means of the computer program.