FR3111253A1 - IMAGE PROCESSING PROCESS AND EQUIPMENT FOR IMPLEMENTING THE PROCESS - Google Patents

Abstract

A method for processing an image in a set of images, in which the image is cut into a plurality of blocks, is proposed, which comprises: determining a plurality of motion vectors, each motion vector corresponding to a block of the plurality of blocks of the image, and pointing to a block correlated to the corresponding block in a reference image, the reference image being an image of the set of images distinct from the image; distributing motion vectors among the plurality of determined motion vectors into classes of a first histogram based on a matching criterion applied, for each classified motion vector, to a component of the motion vector; select a first class of the first histogram based on a selection criterion; and determining an overall motion vector of the image based on a motion vector corresponding to the selected first class. Abstract Figure: Figure 3

Description

IMAGE PROCESSING METHOD AND EQUIPMENT FOR IMPLEMENTING THE METHOD

The present disclosure relates to an image processing method and a device for implementing this method. It applies in particular to the coding of images of a video stream.

Video data is generally subject to source coding aimed at compressing it in order to limit the resources necessary for its transmission and/or storage. There are many encoding standards, such as H.264/AVC, H.265/HEVC and MPEG-2, which can be used for this purpose.

We consider a video stream comprising a set of images. In classic coding schemes, the images of the video stream to be encoded are typically considered according to an encoding sequence, and each is divided into sets of pixels which are also processed sequentially, for example starting at the top left and ending at the bottom. to the right of each image.

The encoding of an image of the stream is thus carried out by dividing a matrix of pixels corresponding to the image into several sets, for example blocks of fixed size 16 x 16, 32 x 32 or 64 x 64, and encoding these blocks of pixels according to a given processing sequence. Certain standards, such as H.264/AVC, provide for the possibility of breaking down blocks of size 16 x 16 (then called macro-blocks) into sub-blocks, for example of size 8 x 8 or 4 x 4, in order to perform encoding processing with finer granularity.

Current techniques for video compression of an image of a set of images can be divided into two main categories: on the one hand, so-called "Intra" compression, in which compression processing is carried out on the image from of the pixels of the image, and on the other hand the so-called "Inter" compression, in which the compression processing is carried out on the image from the pixels of one or more other images of the set of images .

In the Intra mode, the processing of a block (or set) of pixels typically includes a prediction of the pixels of the block carried out using causal pixels (previously coded) present in the image being encoded (known as " current image”), in which case we speak of “Intra prediction”. In the Inter mode, the processing of a block (or set) of pixels typically includes a prediction of the pixels of the block performed using pixels from previously coded images, in which case we speak of "Inter prediction" or "motion compensation".

This exploitation of spatial and/or temporal redundancies makes it possible to avoid transmitting or storing the value of the pixels of each block (or set) of pixels, by representing at least some of the blocks by a residual of pixels representing the difference (or the distance) between the prediction values of the pixels of the block and the actual values of the pixels of the predicted block. The pixel residual information is present in the data generated by the encoder after transform (for example, of DCT type) and quantization in order to reduce the entropy of the data generated by the encoder.

It is desirable to reduce as much as possible the additional information generated by the prediction of the pixels and present at the output of the encoder in order to increase the efficiency of a coding/compression scheme at a given level of distortion. Conversely, one can also seek to reduce this additional information to increase the efficiency of a coding/compression scheme at a given encoder output rate level.

A video encoder typically performs an encoding mode choice corresponding to a selection of encoding parameters for a set of processed pixels. This decision-making can be implemented by optimizing a bitrate and distortion metric, the encoding parameters selected by the encoder being those which minimize a bitrate-distortion criterion. The choice of encoding mode then has an impact on the performance of the encoder, both in terms of bitrate gain and visual quality.

The AOMedia Video 1 (AV1) codec is a royalty-free video codec developed by the Alliance for Open Media (AOM). Born in 2018, it is the successor of another royalty-free video codec, the Vp9 codec, and offers new coding tools that significantly increase its efficiency compared to its predecessor. The AV1 codec presents itself as a competitor to video codecs specified by the MPEG standards, and in particular the H.265 (HEVC) codec. The AV1 codec specification document is available online and accessible at the following URL: https://aomedia.googlesource.com/avl-spec/.

Essentially designed for video broadcasting on the Internet and on an IP network, the AV1 codec incorporates the main principles of video compression, including in particular the use of intra and inter-image redundancy, the use of an entropy coder and the consideration of the human visual system. It is announced as being, thanks to new tools, 35% more efficient than its competitor HEVC, in that it would allow a reduction in average bit rate of 35% at equivalent visual quality compared to the HEVC codec.

Just like HEVC, the AV1 video encoder is based on dividing the image into blocks of variable sizes, and tests a certain number of coding tools on them (for example temporal and spatial prediction modes, different types of transform, etc.), by exploiting the spatial redundancies on an image or the temporal redundancies on a plurality of images, in order to select, for each block, the most effective tool.

Several new inter-type tools (which exploit the temporal redundancies of an image sequence) have been introduced in AV1. These inter tools use already coded reference images (up to 7) to represent the movements of the blocks of an image (e.g. block B of the current image I is identical to block B' of the reference image I' to within a translation T).

Certain Inter coding tools (called "GLOBAL_MOTION" tools in the AV1 codec specification) use a series of so-called "global motion" parameters specific to the image (and therefore common to all the blocks) which aim to transcribe the movements of cameras (ranging from simple tracking shots to more complex movements such as zooming).

The advantage for an Inter mode of using information global to the image is to avoid transmitting redundant motion information a multitude of times (because it is common to many blocks), and therefore to greatly reduce the quantity of information to be transmitted to the decoder and reduce the video bit rate.

Among the tools using global parameters (GLOBAL_MOTION), the TRANSLATION mode is the most suitable for representing a lateral traveling of the camera (a 2D translation). This mode uses two of the eight global parameters of an image (one represents the component of the translation on the x axis and the other the component of the translation on the y axis) in order to materialize a global motion vector .

The AV1 specification is an example of a codec specification that provides for the possible use and transmission of one or more global motion vectors. According to the current AV1 specification, the transmission of global motion vectors is not mandatory. A symbol (flag) in the header (in English, the “frame header”) of each image indicates to the decoder whether or not the global vectors are present in the binary stream (in English, “bitstream”) to be decoded. If this symbol indicates that they are present, the decoder expects to receive as many global vectors as there are reference images. The AV1 specification providing that the determination of global vectors is solely the responsibility of the encoder (therefore non-normative), it does not currently contain any recommendation for determining global motion vectors. On the other hand, the global vectors, if they are used in an AV1 codec, must meet normative criteria (size for example).

There is thus a need for an image processing method making it possible to determine an overall motion vector of an image in a set of images or a video stream to be processed (for example to be encoded or analyzed).

Summary

This disclosure improves the situation.

According to a first aspect, there is proposed a method for processing an image in a set of images, in which the image is cut into a plurality of blocks, the method comprising: determining a plurality of motion vectors, each vector of movement corresponding to a block of the plurality of blocks of the image, and pointing to a block correlated to the block corresponding in a reference image, the reference image being an image of the set of images distinct from the picture ; distributing motion vectors among the plurality of determined motion vectors into classes of a first histogram based on a matching criterion applied, for each classified motion vector, to a component of the motion vector; selecting a first class of the first histogram on the basis of a selection criterion; and determining an overall motion vector of the image based on a motion vector corresponding to the selected first class.

The proposed method advantageously makes it possible to determine a global motion vector of an image of a set of images. In particular, the proposed method allows the determination of a global motion vector of the image based on the motion estimation of portions of the image. In addition, the proposed method advantageously integrates a spatio-temporal relevance criterion which makes it possible to work only with the vectors of portions which are relevant according to this criterion.

In addition, the proposed method has the advantage of proposing a determination of the global motion vector of an image in a set of images which does not mobilize significant machine resources and whose implementation, whether software and/or or material, is not very complex.

For example, in the context of its implementation within a video encoder, for example of the AV1 type, the proposed method does not significantly impact the complexity of the calculations performed by the encoder. Thus, the proposed method has the advantage of an implementation that is not very complex, of a very low computational complexity, and of a fast execution speed which in particular allows its integration into a real-time encoding solution. without significant impact. It thus makes it possible in particular to globally improve the quality of the coding within an image belonging to a video sequence.

The proposed method is particularly suitable, although not exclusively, for encoding or compressing an image of a set of images, for example according to a standard scheme of the AV1, H.120, H.261 type. , MPEG-1, H.262/MPEG-2, H.263, H.263+, H.263++, MPEG-4 part 2, H.264/AVC, H.265/HEVC (High Efficiency Video Coding ), or SHVC (Scalable HEVC), VVC, EVC, according to the current corresponding specifications or their evolutions, in particular in the applicable cases to accelerate the encoding at certain levels. But it is also suitable for the encoding or more generally the processing of images according to any encoding or video processing scheme using or compatible with the use of a global motion vector or global motion analysis.

The proposed method can advantageously be implemented within any video encoder using global motion vectors of the encoded images, such as for example, but in a non-limiting way, a video encoder conforming to the current AV1 specification or its evolutions.

The proposed method is however not limited to implementations within an image encoder, and can be implemented within a global motion estimation unit. The global motion estimation unit may be implemented in software form, in which case it will take the form of a program executable by a processor, in hardware form (or "hardware"), such as an application-specific integrated circuit ( ASIC), a system on chip (SOC), or in the form of a combination of hardware and software elements, in which case it may comprise an input interface, a memory operationally coupled with at least one processor configured for the implementation of the proposed method. Depending on the embodiment chosen, the global motion estimation unit may be implemented within a video encoder, a video decoder, for example to perform a global motion calculation, a device for video analysis, for example to extract semantic information or information from video content, and/or a video processing device, for example to isolate foregrounds from backgrounds in images of video content .

The characteristics exposed in the following paragraphs can, optionally, be implemented. They can be implemented independently of each other or in combination with each other:

In one or more embodiments, determining the plurality of motion vectors may include: determining a motion vector for each block of the plurality of blocks of the image.

In one or more embodiments, the distributed motion vectors may be distributed into first classes of the first histogram based on the match criterion applied to a first component of the classified motion vector, and into second classes of a second histogram based on the matching criterion applied to a second component of the classified motion vector, and the proposed method may further comprise: selecting a second class of the second histogram based on the selection criterion; and wherein the overall motion vector of the image is generated based on motion vector components corresponding to the selected first and second classes, respectively.

In one or more embodiments, the selection criterion used may determine a class having the greatest number of values.

In other embodiments, the selection criterion could determine a class on the basis of combinations of the values of a distortion metric respectively obtained for the motion vectors of classes. A combination of distortion metric values can be calculated for each class of a set of classes in which the selection criterion is applied. This set of classes can include all the classes of a histogram, or only a part of these classes. For example, the selection criterion used could determine a class among a set of classes on the basis of the respective averages calculated for one or more of the values of a distortion metric respectively obtained for the motion vectors of each class of the set . Depending on the embodiment, the averages calculated may be arithmetic averages, or be calculated according to any other average formula that can be used for implementing the proposed method.

In these embodiments, the global motion vector of the image could be determined on the basis of a motion vector associated with the highest value of the inverse of the combination of values of the distortion metric among the inverses of the respective combinations of the values of the distortion metric respectively obtained for the class motion vectors. A combination of the distortion metric values can be calculated for each class of a set of classes, which can include all the classes of a histogram, or only a part of these classes. For example, the global motion vector of the image could be determined on the basis of a motion vector associated with the inverse of the average of the values of the highest distortion metric among the inverses of the respective averages of the values of the distortion metric respectively obtained for the class motion vectors. Depending on the embodiment, the averages calculated may be arithmetic averages, or be calculated according to any other average formula that can be used for implementing the proposed method.

In one or more embodiments, the motion vectors distributed in classes may be selected from among the motion vectors determined on the basis of a criterion of relevance. In one or more embodiments, the relevance criterion may include a comparison of a computed distortion metric value for the motion vector to a threshold. In one or more embodiments, the distortion metric may use a distance measurement.

In one or more embodiments, the motion vectors distributed in classes may be selected from among the motion vectors determined on the basis of a size criterion. In one or more embodiments, the size criterion may be fulfilled in the case where one or more of the components of the motion vector representing a translation along a respective dimension of the corresponding reference image has a value belonging to a range of predetermined values.

In one or more embodiments, the correspondence criterion may be fulfilled in the case where one or more of the respective component values of two motion vectors, the components representing a translation along a respective dimension of the corresponding reference image, are the same.

In one or more embodiments, the correspondence criterion may comprise a two-by-two identity criterion, for two motion vectors, of the respective values of components representing a translation along a respective dimension of the corresponding reference image.

According to another aspect, an image processing device is proposed, which comprises: an input interface configured to receive an image from a set of images; an image processing unit, comprising a processor, operatively coupled to the input interface, and configured to perform image processing according to one or more embodiments of the proposed method.

According to another aspect, a computer program, loadable into a memory associated with a processor, and comprising code portions for the implementation of one or more embodiments of the proposed method during the execution of said program by the processor, is offered.

Another aspect relates to a set of data representing, for example by way of compression or encoding, a computer program as proposed.

According to another aspect, a non-transitory storage medium of a computer-executable program is provided, which comprises a set of data representing one or more programs, said one or more programs comprising instructions for, when executing said one or more programs by a computer comprising a processing unit operationally coupled to memory means and to an input/output interface module, causing the computer to implement one or more embodiments of the proposed method. Another aspect, there is provided a computer program comprising instructions for the implementation of all or part of a method as defined herein when this program is executed by a processor. In another aspect, there is provided a non-transitory, computer-readable recording medium on which such a program is recorded.

Other characteristics, details and advantages will appear on reading the following description of non-limiting embodiments, with reference to the appended drawings, in which:

Fig. 1

shows a video encoder diagram for the implementation of the proposed method according to one or more embodiments.

Fig. 2

shows an example of motion estimation analysis on a current image divided into blocks according to one or more embodiments.

Fig. 3

shows a diagram illustrating an exemplary implementation of the proposed method according to one or more embodiments.

Fig. 4a

shows an example of generation of motion vectors for a current image for the implementation of the proposed method according to one or more embodiments.

Fig. 4b

shows an example of a histogram for the implementation of the proposed method according to one or more embodiments.

In the following detailed description of embodiments of the invention, many specific details are presented to provide a more complete understanding. Nevertheless, those skilled in the art can appreciate that embodiments can be practiced without these specific details. In other cases, well-known features are not described in detail to avoid unnecessarily complicating the description.

This description refers to functions, engines, units, modules, platforms, and illustrations of diagrams of the methods and devices according to one or more embodiments. Each of the described functions, engines, modules, platforms, units, and diagrams may be implemented in hardware, software (including embedded software (“firmware”), or “middleware”), firmware, or any combination of these. In the case of software implementation, the functions, engines, units, modules and/or diagram illustrations may be implemented by computer program instructions or software code, which may be stored or transmitted on a computer-readable medium, including a non-transitory medium, or a medium loaded into the memory of a generic, specific computer, or other programmable data processing device or device to produce a machine, such that the computer program instructions or the software code executed on the computer or the apparatus or programmable data processing device, constitute means of implementing these functions.

Embodiments of a computer-readable medium include, but are not limited to, computer storage media and communication media, including any media that facilitates the transfer of a computer program from one location to another. . “Computer storage medium(s)” means any physical medium that can be accessed by computer. Examples of computer storage media include, but are not limited to, flash memory disks or components or any other flash memory devices (e.g. USB drives, memory sticks, memory sticks, disk drives), CD-ROMs or other optical data storage devices, DVDs, magnetic disk data storage devices or other magnetic data storage devices, data memory components, RAM, ROM, EEPROM, memory cards (“smart cards”), memories of the SSD (“Solid State Drive”) type, and any other form of medium usable for transporting or storing or memorizing data or data structures which can be read by a computer processor computer.

In addition, various forms of computer-readable media can transmit or carry instructions to a computer, such as a router, gateway, server, or any data transmission equipment, whether wired transmission ( via coaxial cable, optical fiber, telephone wires, DSL cable, or Ethernet cable), wireless (via infrared, radio, cellular, microwave), or virtualized transmission equipment (virtual router, virtual gateway, virtual tunnel end, virtual firewall). Instructions may, depending on embodiments, include code of any computer programming language or computer program element, such as, without limitation, assembly languages, C, C++, Visual Basic, HyperText Markup Language (HTML), Extensible Markup Language (XML), HyperText Transfer Protocol (HTTP), Hypertext Preprocessor (PHP), SQL, MySQL, Java, JavaScript, JavaScript Object Notation (JSON), Python, and bash scripting.

In addition, the terms "in particular", "for example", "example", "typically" are used in this description to designate examples or illustrations of non-limiting embodiments, which do not necessarily correspond to embodiments preferred or advantageous over other possible aspects or embodiments.

The terms "operably coupled", "coupled", "mounted", "connected" and their various variations and forms used in this description refer to couplings, connections, assemblies, which can be direct or indirect, and include in particular connections between electronic equipment or between portions of such equipment which allow operations and functioning as described in the present description. Further, the terms "connected" and "coupled" are not limited to physical or mechanical connections or couplings. For example, an operational coupling may include one or more wired connection(s) and/or one or more wireless connection(s) between two or more devices that allow simplex and/or duplex communication links. between equipment or portions of equipment. According to another example, an operational coupling or a connection can include a coupling by wired and/or wireless link to allow data communications between a server of the proposed system and other equipment of the system.

Figure 1 shows, by way of non-limiting example of an image processing device, an example of a video encoder according to one or more embodiments.

Referring to Figure 1, the encoder 100 receives at input 107 an input video stream 101 comprising a plurality of images to be processed to perform the encoding of the stream. Encoder 100 includes a controller 102, operatively coupled to input interface 107, which drives a motion estimation unit 106 and a prediction unit 104 for predictions according to different modes implemented by the encoder 100, such as Intra, Inter and Skip modes, as well as a global motion vector determination unit 103. The data received on the input interface 107 is transmitted as input to the motion vector determination units global 103, motion estimation 106, controller 102, and prediction 104. The set of controller 102, motion estimation unit 106, prediction unit 104, and global motion vector determination unit 103 forms a encoding unit 108 operatively coupled to input interface 107.

The global motion vector determination unit 103 is configured to implement one or more embodiments of the proposed method. It is thus able to generate data of global motion vectors which are supplied as input to the prediction unit 104. The motion estimation unit 106 is configured to perform a motion estimation analysis on the images to be process, and in particular for the implementation of one or more embodiments of the method proposed by the implementation of a preliminary analysis in order to cut the image to be processed into blocks and perform a motion estimation analysis on at least some of the blocks of the image, and is able to generate data, including motion estimation data, which are supplied to the controller 102, to the global motion vector determination unit 103 for setting implementation of the proposed method, as well as to the prediction unit 104 for the needs of prediction in Inter mode. The prediction unit 104 generates prediction data (for example Intra, Inter and/or Skip mode prediction data) which are supplied as input to the entropy coder 105. For example, the data supplied to the decoder for a prediction of type Inter may include a residual of pixels and information concerning one or more motion vectors. This information relating to one or more motion vectors can comprise one or more indices identifying a predictor vector in a list of predictor vectors known to the decoder. Controller 102 can be configured to generate control data which is also supplied as input to entropy encoder 105.

The controller 102 is configured to drive the motion estimation unit 106, the global motion vector determination unit 103 and the prediction unit 104 in order to control the prediction data which is respectively provided at the input of the encoder entropy 105 by the prediction unit 104. Depending on the encoding scheme implemented by the encoder 100, the controller 102 can further be configured to select among the different types of prediction mode (Intra mode, Inter mode or Skip mode depending on the coding modes implemented in the encoding unit 108) the one for which prediction data will be transmitted to the entropy coder 105. Thus, the encoding scheme can include a decision for each image set process aimed at choosing the type of prediction for which data will be transmitted to the entropy coder 105. This choice will typically be implemented by the controller 102, to decide on the application of the Inter prediction mode, the Intra prediction mode or the Skip prediction mode to the block being processed. This makes it possible to control the sending to the entropic coder of Intra prediction data or else of Inter or Skip prediction data according to the decision taken by the controller 102.

The encoder 100 can be a computer, a network of computers, an electronic component, or another device comprising a processor operatively coupled to a memory, as well as, according to the chosen embodiment, a data storage unit. , and other associated hardware such as a network interface and a media drive for reading removable storage media and writing to such media (not shown in the figure). The removable storage medium can be, for example, a compact disc (CD), a digital video/versatile disc (DVD), a flash disc, a USB key, etc. Depending on the embodiment, the memory, data storage unit, or removable storage medium contains instructions which, when executed by controller 102, cause controller 102 to perform or control interface portions of input 107, global motion vector determination 103, prediction 104, motion estimation 106 and/or data processing of the exemplary implementations of the proposed method described herein. The controller 102 can be a component implementing a processor or a calculation unit for the encoding of images according to the proposed method and the control of the units 107, 106, 103, 104, 105 of the encoder 100.

In addition, the encoder 100 can be implemented in software form, as described above, in which case it takes the form of a program executable by a processor, or in hardware form (or "hardware"), such as a application specific integrated circuit (ASIC), a system on chip (SOC), or in the form of a combination of hardware and software elements, such as for example a software program intended to be loaded and executed on an FPGA type component (of English “Field Programmable Gate Array”). SOCs (System On Chip) or system on chip are embedded systems that integrate all the components of an electronic system into a single chip. An ASIC (from the English "Application-Specific Integrated Circuit") is a specialized electronic circuit that brings together tailor-made functionalities for a given application. ASICs are usually configured when manufactured and can only be simulated by the user. FPGA-type programmable logic circuits (Field-Programmable Gate Array) are user-reconfigurable electronic circuits.

An encoder can also use hybrid architectures, such as architectures based on a CPU+FPGA, a GPU (from the English "Graphics Processing Unit") or an MPPA (from the English "Multi-Purpose Processor Array") .

Figure 2 is a diagram illustrating the proposed method according to one or more embodiments.

In one or more embodiments, an analysis of the image being processed is performed to generate analysis data that will be used when implementing the proposed method. The analysis can include a division of the current image into blocks followed by a motion estimation on one or more of the blocks of the current image.

In one or more embodiments, the current image is thus divided into blocks or coding units (in English “Coding Unit”, or CU), whose shape and/or size, which can vary from one block to another, can be determined as a function in particular of the size of the matrix of pixels representing the image, for example in macroblocks of square shape of 16×16 pixels. A set of blocks is thus formed for which a processing sequence is defined (sometimes also called “processing path”, “analysis path” or “analysis sequence”). In the case of square-shaped blocks, it is for example possible to process the blocks of the current image starting with the one located at the top left of the image, followed by the one immediately to the right of the previous one, until arriving at the end of the first line of blocks to move to the leftmost block in the line of blocks immediately below this first line, to complete processing by the lowermost and rightmost block of the image. This processing sequence is sometimes designated by the English terms “raster” sequence or “raster scan” of the image being processed.

In one or more embodiments, a motion estimation is then performed on one or more of the blocks of the current image, in order to generate motion estimation data for the current image.

In one or more embodiments, it will be possible to consider a “current block”, that is to say a block being processed in the current image. The processing of the current block may comprise the partitioning of the block into sub-blocks, in order to process the block with a finer spatial granularity than that obtained with the block. Depending on the embodiment, for example in the context of a video coding application, the processing of a block may also include the prediction of the pixels of the block, by exploiting the spatial correlation (in the same image) or temporal (in previously coded images) between pixels. When several types of prediction, such as an Intra type prediction, an Inter type prediction, and/or a skip type prediction are used for video encoding, the block pixel prediction may typically include the selection of a prediction type of the block and prediction information corresponding to the selected type, the whole forming a set of encoding parameters. The prediction of the block of pixels processed makes it possible to calculate a residual of pixels, which corresponds to the difference between the pixels of the current block and the pixels of the prediction block, which can be transmitted in certain embodiments to the decoder after transform and quantization .

The motion estimation for a current block according to one or more embodiments is described below in more detail.

Motion estimation analysis in a current frame typically uses one or more sets of pixels belonging respectively to one or more previously analyzed frames (for example, in a video encoder, one or more previously encoded frames), commonly referred to as frames( s) reference, according to a sequence of analysis of a set of images comprising the current image.

For a given block in the current image (the "current block" or "original block"), the motion estimation can consist in studying the displacement of the current block (belonging to the current image) with respect to the position of a block (or more generally, a set of pixels) belonging to a reference image by exploiting the temporal correlation between the pixels of the two blocks. The motion estimation of the current block can thus lead to the selection of a set of pixels (called "reference block") that most closely resembles in a reference image, by representing the motion of the current block as positioned in the current image by relative to the reference block as positioned in the reference image by one or more motion parameters of the current block. These parameters can define a vector, called motion vector, whose coordinates can represent the movement, according to each of the dimensions of the image, of a projection of the current block in the reference image to coincide with the position of the block of reference in the reference image. The motion vector can for example be a vector in a two-dimensional frame, for example horizontal and vertical, whose coordinates correspond respectively to a horizontal displacement and a vertical displacement to superimpose a projection in the reference image of the current block with the reference block.

Figure 2 illustrates an example of motion estimation according to one or more embodiments of the proposed method.

FIG. 2 shows the projection of a current block of a current image onto a reference image, as well as the translations in each of the dimensions of the image to be performed in order to reach a correlated block in the reference image (estimated block in the figure) with the current block. As illustrated in the figure, in one or more embodiments, the motion vector corresponds to the displacement of the projection of the current block in the reference image to reach the estimated block.

The motion estimation process is non-normative and is therefore likely to differ from one encoder to another.

For example, the motion estimation method may include searching in a more or less extensive area of the reference image, for example defined from the block of the reference image corresponding to the original block in the original image, in order to test the degree of resemblance (or correlation) of the original block with a more or less large number of candidate blocks of the reference image.

Thus, in one or more embodiments, the motion estimation makes it possible to search, for each current block of a set of blocks of the current image, for one or more reference images, for example by successive steps, the block having the most resemblance to the current block.

In one or more embodiments, the degree of correlation between a block and its displacement according to a motion vector may be estimated using a distortion metric, such as the Sum of Absolute Differences (in English “Sum of Absolute Differences”, or “SAD”): Or is the pixel at position of the current block and the pixel at position of the reference block. A low SAD will be interpreted as an indication that the two blocks are very similar.

In one or more embodiments, the relevance of the motion vector determined for a block to characterize the real motion of the block (and not to minimize the SAD) can be measured using a self-distortion value of the block, by example by calculating the distortion between the block and a version of this block shifted by a first predetermined number of pixel(s) according to a first dimension and by a second predetermined number of pixel(s) according to a second dimension of the block. This self-distortion value, referred to below as “AUTOSAD” in the case where the distortion metric used is the SAD metric, can be used to be compared with the distortion value calculated initially, for example by using the ratio between the distortion value initially calculated on the self-distortion value, to obtain a relevance value of the motion vector corresponding to the distortion value initially calculated. In the case where the distortion metric used is the SAD metric, it will thus be possible to calculate a value of SAD/AUTOSAD to measure the relevance of the value of SAD.

The AUTOSAD advantageously provides an indicator of the spatial complexity of the block. A complex block will contain for example details/pixels which make it different from one block to another, and/or which contains a lot of spatial information (for example a lot of texture). Since to determine a global motion vector, it is desirable to encode the real motion of the current block, it is advantageous to use the AUTOSAD parameter. Indeed, unlike a motion estimation carried out for the encoding of the block, we are not looking for a resembling block, but rather to characterize the movement of the block from one image to another. There may be in a following image a more similar block, but which does not represent the true translation of the current block. However, to determine a global motion vector, we seek to obtain a global characterization of the motion of the majority of the blocks of the image. To do this, in one or more embodiments, the proposed method uses the AUTOSAD to exclude vectors whose SAD would certainly be very low, but which would not truly represent the true movement of the block in the set of images.

Thus, if the self-distortion of a block is weak, it can be considered that there are not many details in the block, and that it will therefore be difficult to find a relevant vector (representing the actual translation of the block). In other words, it can be considered that the distortion criterion value does not necessarily represent the actual movement of the block. It may indeed be that a low distortion criterion value corresponds to a block (in a reference image) which is certainly close to the current block, but which does not characterize the movement of the current block with respect to the image of reference.

In one or more embodiments, it may be decided not to use a vector having a low self-distortion value (for example from AUTOSAD) for the determination of a global motion vector, even if its value of distortion (for example its SAD) would indicate that it is very similar to the block, given that we are not looking to find a block very similar to the current block, but a vector representing the real translation of this block in an image of reference.

The use of a self-distortion criterion thus advantageously makes it possible to avoid, for example, using a motion vector corresponding to a block identified in a reference image having a distortion value (for example an SAD) of less than the distortion value (e.g. SAD) of the original block translated with its actual motion.

In one or more embodiments, the relevance of a determined motion vector for a block can be measured by calculating a self-distortion criterion for that block, and comparing the distortion value with the self-distortion value. distortion. For example, the distortion value can be compared to a normalized self-distortion value, for example of the form: , Or And are two parameters. In the embodiments where the distortion criteria used are of the SAD type, the value of SAD obtained for a vector corresponding to the current block can be compared to , where AUTOSAD is the measured self-distortion for the current block.

In one or more embodiments, the parameter can be used to compensate for block deformation and/or motion estimation inaccuracy (1 pixel, ½ pixel, ¼ pixel), and be chosen constant. In one or more embodiments, the parameter can be used to compensate for the overall noise level of the image, and be chosen constant. Parameter value can be determined dynamically so as to adapt to the content of the image or of the video sequence in order to evolve according to the level of noise.

For example, in one or more embodiments, the value of the parameter can be chosen equal to 0. Thus, a comparison between a relevance criterion value comprising a ratio between a distortion value and a self-distortion value (for example, a value ) and a predetermined threshold (for example chosen equal to 2) can be used to select the determined motion vectors to be used for the determination of the global motion vector. For example, a vector corresponding to a current block for which the value of SAD/AUTOSAD is greater than 2 may not be used for determining the global motion vector of the current image, for example by not using it for constitute the histogram(s) from which the global vector is determined. Indeed, even if the value of SAD (obtained for the motion vector obtained for the current block for a reference image) is low (in particular in the case where a block has been found in the reference image which looks very similar to the current block, to which the motion vector corresponds, but which does not characterize the motion of the current block or characterizes it little), the value of AUTOSAD of the current block may also be low, making the SAD/AUTOSAD ratio high, which which could result in not selecting the motion vector for determining the global motion vector of the current image.

According to the embodiment, the motion estimation can be performed for a given current block for one or more reference images. For example, in the context of an implementation of the proposed method on an AV1 type video codec, the motion estimation performed on each current block among the blocks of the current image can be performed for up to 7 images thus generating, for each current block, 7 sets of motion estimation parameters for the current block corresponding respectively to the 7 reference images.

Several sets of motion vectors could thus be generated, corresponding respectively to a reference image of a set of reference images, and the proposed method could be implemented in one or more embodiments from one of these sets of motion vectors.

FIG. 3 illustrates a non-limiting example of implementation (200) of the proposed method according to one or more embodiments.

Referring to Figure 3, we consider an image being processed (current image) in a set of images. The current picture is broken into blocks, for example as part of its analysis or processing for compression according to a video codec specification.

In one or more embodiments, in particular when the proposed method is used for encoding the current image, provision may be made to encode each block of the image according to one of a plurality of coding modes comprising at least one temporal correlation prediction type coding mode using a plurality of images from the set of images.

In one or more embodiments, a motion estimation analysis can be performed on one or more blocks of the image, using for each block analyzed a reference image, which can be chosen from a set comprising several images of reference, depending on the embodiment. Depending on the embodiment, this analysis can generate motion estimation parameters for the reference image and for the blocks of the image for which it is performed, which can take include one or more components of a vector of movement, as well as possibly a measure of correlation (or distortion), for example of the SAD type, between the reference block (the block of the reference image) corresponding to the movement vector and the current block.

The motion estimation analysis thus makes it possible to obtain, for a reference image of the set of images considered, a set of motion vectors corresponding respectively to blocks of the current image and pointing to respective blocks. of the reference image, as well as, for each motion vector, a distortion (or correlation) parameter between the block corresponding to the vector and the block in the reference image towards which the vector points. Each motion vector can have two displacement components ( And ), corresponding for a first component ( ) to a displacement (for example a translation) of a projection of the current block in the reference image along a first dimension ( ) in a frame of the image, and for a second component ( ) to a displacement of the projection along a second dimension ( ) in the image marker. The motion estimation analysis may thus generate, in one or more embodiments, a set of vectors and their associated SAD values: .

The motion estimation analysis performed on the current image may thus comprise, in one or more embodiments, the determination (201) of a plurality of motion vectors, each vector corresponding to a block of the plurality of picture blocks. Each motion vector may be configured to point to a block correlated to the corresponding block in a respective reference image of a set of reference images, each reference image being an image of the set of images distinct from the picture.

In one or more embodiments, a set of reference frames may be predefined, including one or more reference frames, for implementing block motion estimation analysis of the current frame. A reference image corresponding to the current image will typically have been processed (for example encoded) prior to the current image, for example by following an image processing sequence of the set of images considered, possibly constituted dynamically. In the embodiments in which the set of images is a video sequence, a reference image may be an image located before or after the current image in the video sequence, depending on the embodiment chosen.

One or more motion vectors can thus for example be generated for a block of the current image, and correspond respectively to one or more reference images corresponding to the current image.

Several motion vectors can thus be obtained, for a plurality of blocks of the current image, each motion vector corresponding to a block and to a reference image, and representing for example the displacement of the corresponding block and its distortion measurement ( for example of the SAD type) associated.

For example, in one or more embodiments, motion estimation analysis performed on the current frame may generate a set motion vectors with two displacement components (for example a component corresponding to a displacement along a first dimension in a reference frame of the corresponding reference image and a component corresponding to a displacement along a second dimension in the reference), with which is associated a parameter distortion measurement (e.g. an SAD measurement) respectively.

One or more histograms can then be created based on a set of motion vectors (corresponding to a reference image).

In one or more embodiments, it is thus possible to distribute (202) motion vectors among the plurality of motion vectors determined in classes of a first histogram.

The distribution of the motion vectors to create the first histogram can be performed based on a match criterion chosen depending on the embodiment.

For example, in one or more embodiments, the correspondence criterion used may relate to one or more components of the motion vector to which it is applied to distribute the vector in a class of the histogram. Each component can advantageously be chosen from components of displacement (component and/or component ) determined for the motion vector to be distributed.

Furthermore, depending on the chosen embodiment, all or part of the motion vectors may be distributed for the generation of a histogram. In one or more embodiments, all the determined motion vectors may be distributed in the different classes of one or more histograms according to chosen correspondence criteria. In other embodiments, only part of the determined motion vectors will be used for the constitution of each histogram among the histogram(s) used.

Different embodiments for generating one or more motion vector histograms are described in more detail below.

In one or more embodiments, the proposed method provides for selecting (203) a first class of the first histogram on the basis of a selection criterion once the first histogram has been constituted.

In one or more embodiments, a global motion vector of the image can then be determined (204) based on a motion vector corresponding to the first class selected in the histogram.

In one or more embodiments, the proposed method may use only one histogram ( ), generated on the basis of a first motion vector classification mode among the set of motion vectors obtained by motion estimation analysis of the current image. The motion vectors used for this classification may correspond to at least one displacement parameter, for example in the form of one or more displacement coordinates along the directions of a standardized reference, and to a distance parameter between the blocks to which the vector corresponds located respectively in the current image and in a reference image (for example to characterize the displacement between a projection of the current block in the reference image and the block of the reference image towards which the vector points) .

In what follows, we consider the non-limiting example of two components of displacement (for example a component corresponding to a displacement along a first dimension in a reference frame of the reference image and a component corresponding to a displacement along a second dimension in the frame), and a component distortion measurement (for example an SAD measurement). Nevertheless, those skilled in the art will be able to realize that any other components of a motion vector suitable for classification in a histogram, such as for example any other motion components, and/or any other distortion measurement component for the motion vector, may be used instead of or in addition to the components , and or , which are described by way of example only.

In one or more embodiments, the correspondence criterion may comprise a two-by-two identity criterion, for two motion vectors, of the respective values of components representing a translation along a respective dimension of the corresponding reference image.

According to the first mode of classifying motion vectors, vectors having pairs of motion components ( ; ) identical can be distributed in the same class of motion vectors. For example, two vectors And having two-by-two identical displacement components ( And ) may be distributed in the same class on the basis of their identical components.

Thus, in one or more embodiments, the correspondence criterion can be defined as being fulfilled in the case where one or more of the respective values of components of two motion vectors, the components representing a translation according to a respective dimension of the corresponding reference image, are identical.

In these embodiments, the correspondence criterion applied to at least one component of the motion vector to be distributed in a class comprises an identity criterion of this component with the other vectors of the class. In the example described previously, the correspondence criterion comprises a two-by-two identity criterion of the displacement components of the vector with the displacement components of another vector of the class.

In one or more embodiments, a new class of H1 histogram vectors may be created to distribute a motion vector therein for which no H1 histogram class to distribute the vector based on the matching criterion n still exists when distributing the vector in one of the classes of the histogram.

As a variant, the distribution of the vectors in classes could be implemented by using a table, for example two-dimensional, whose entries correspond respectively to pairs of coordinates (components of displacement), which are filled in by classifying the vectors selected to generate the histogram . This implementation, which involves storing in memory an array of dimension(s) corresponding to the number of possible combinations of pairs of coordinates of vectors, has the advantage of allowing a very rapid distribution of vectors in classes, from because of its simplicity from a computational complexity point of view, since it does not use classes created dynamically but a priori.

In one or more embodiments, the motion vectors distributed in classes of the histogram can be selected beforehand from among the motion vectors determined on the basis of a relevance criterion. The purpose of the relevance criterion used may be to avoid selecting for the distribution into classes of the histogram motion vectors that are not very relevant, for example with regard to a corresponding distortion metric value, and which could have been selected for lack of better vectors obtained during motion estimation analysis. The relevance criterion may therefore, in one or more embodiments, be advantageously defined on the basis of one or more distortion metrics.

Therefore, in one or more embodiments, the relevance criterion may comprise a comparison of a distortion metric value calculated for the motion vector with a predetermined threshold. The predetermined threshold may advantageously be chosen according to the type of distortion metric used for the relevance criterion.

In one or more embodiments, the distortion metric value used may be of the SAD type, as described above. In other embodiments, the distortion metric value used may also use a so-called “AUTOSAD” metric, for example when using a criterion of the SAD/AUTOSAD type (ratio of the value of SAD to AUTOSAD value), as described above.

In what follows, we consider the non-limiting example of a relevance criterion using a distortion measure of the SAD/AUTOSAD type and corresponding threshold values. Nevertheless, those skilled in the art will be able to realize that any other criterion suitable for a measurement of the relevance of an assignment of a motion vector in a class of the histogram, on the basis in particular of any measurement of distortion that can be used for this purpose, may be used instead of or in addition to a criterion based on a measurement of SAD/AUTOSAD, which is only described here by way of non-limiting example.

It will be possible to choose, in one or more embodiments using a relevance criterion using a distortion measurement of the SAD/AUTOSAD type, a comparison threshold in the interval [-1/2; +8], for example equal to 2.

Thus, in one or more embodiments, a histogram is generated from motion vectors of the set of motion vectors determined for the current image, on the basis of a correspondence criterion according to which the vectors having identical displacement components in pairs along respective directions are distributed in the same class, possibly after having been selected on the basis of a relevance criterion comparing a value of SAD/AUTOSAD which is associated with them with a predetermined threshold, for example chosen equal to 2. For example, the vectors resulting from motion estimation analysis can first of all be selected on the basis of the relevance criterion, and only the vectors whose SAD/AUTOSAD value does not exceed the threshold can be retained. Once selected, these vectors can then be distributed in classes in order to constitute the histogram .

In one or more embodiments, the proposed method may use a histogram generated on the basis of the first motion vector classification mode among the set of motion vectors obtained by motion estimation analysis of the current image. The motion vectors used for this classification may correspond to at least one displacement parameter, for example in the form of one or more displacement coordinates along the directions of a standardized frame (corresponding respectively to dimensions), and to a distance parameter between the blocks to which the vector corresponds located respectively in the current image and in a reference image.

In one or more embodiments, the correspondence criterion may comprise a two-by-two identity criterion, for two motion vectors, of the respective values of components representing a translation along a respective dimension of the corresponding reference image.

According to the first mode of classifying motion vectors, vectors having pairs of motion components ( ; ) identical can be distributed in the same class of motion vectors. For example, two vectors And having two-by-two identical displacement components ( And ) may be distributed in the same class on the basis of their identical components.

Thus, in one or more embodiments, the correspondence criterion can be defined as being fulfilled in the case where one or more of the respective values of components of two motion vectors, the components representing a translation according to a respective dimension of the corresponding reference image, are identical.

Same as for the histogram , in one or more embodiments, a new class of histogram vectors can be created to distribute therein a motion vector for which no class of the histogram allowing to distribute the vector on the basis of the match criterion still exists when distributing the vector in one of the classes of the histogram . As a variant, it is possible to use a two-dimensional table, in the case of vectors with two coordinates, stored in memory and which is filled in by associating vectors therewith with their corresponding entries.

In one or more embodiments, the histogram could represent the means of the values of a distortion criterion, for example the SAD, of the vectors of each class. Thus, in one or more embodiments, a histogram is generated from motion vectors of the set of motion vectors determined for the current image, on the basis of a correspondence criterion according to which the vectors having identical displacement components in pairs along respective directions are distributed in the same class, then we determine for each class of the histogram an average of the distortion values (for example values of SAD) respectively associated with the vectors of the class. For example, we add each vector in the class that corresponds to it on the basis of the identity of its coordinates with those of the class, then when the histogram is finished (for example when all the vectors used to generate the histogram have been distributed in respective classes), a combination is made of the values of SAD corresponding to the vectors of all or part of the classes of the histogram. For example, for each class of the histogram, a linear combination of the values of SAD corresponding to the motion vectors of the class is calculated. In particular, in one or more embodiments, an average (for example an arithmetic average) of the values of SAD corresponding to the motion vectors of the class is calculated. For example, if the histogram H2 includes C classes , each class including a number motion vectors with which are respectively associated SAD values , we calculate a combination of these SAD values for the class , for example an average of these SAD values of type: .

In one or more embodiments, no selection may be made on the vectors resulting from the motion estimation analysis prior to the generation of the histogram .

Alternatively, similarly to the selection scheme described above for the histogram , a selection can be made on the motion vectors resulting from the motion estimation analysis on the basis of a relevance criterion, for example by comparing a value of SAD/AUTOSAD associated with each vector with a predetermined threshold, for example chosen equal to 2, so as not to distribute irrelevant vectors in the classes of the histogram which may have been retained during the motion estimation for lack of finding better vectors (in terms of distortion). For example, the vectors resulting from the motion estimation analysis can first of all be selected on the basis of the criterion of relevance, and can be retained only the vectors whose value of SAD/AUTOSAD does not exceed the threshold. Once selected, these vectors can then be distributed in classes in order to constitute the histogram.

Thus, in one or more embodiments, a histogram is generated from motion vectors of the set of motion vectors determined for the current image, on the basis of a correspondence criterion according to which the vectors having identical displacement components in pairs along respective directions are distributed in the same class, by determining for each class the mean of the SAD values of the vectors of the class, after having been selected on the basis of a criterion of relevance comparing a value of SAD/AUTOSAD which is associated with them with a threshold predetermined, for example chosen equal to 2. This will allow, unlike the histogram , to select a class, not on the basis of the number of elements of each class, but on the basis of a comparison of combinations of the distortion values associated with the vectors of each class, for example of a comparison of the means of the SAD values of the motion vectors of each class.

In one or more embodiments, the proposed method may use several histograms, each generated on the basis of a respective method of classifying motion vectors among the set of motion vectors obtained by motion estimation analysis of the current picture. The motion vectors used for this classification may correspond to at least one displacement parameter, for example in the form of one or more displacement coordinates along the directions of a standardized reference, and to a distance parameter between the blocks to which the vector corresponds located respectively in the current image and in a reference image.

In one or more embodiments, the correspondence criterion may comprise an identity criterion, for two motion vectors, of the respective values of components representing a translation along a respective dimension of the corresponding reference image. This correspondence criterion can thus be applied for each of the dimensions of the current image, to generate a corresponding histogram.

According to a second method of classifying motion vectors, the vectors having displacement components along an identical first dimension ( ) can be distributed in the same class of motion vectors of a first histogram . For example, two vectors And having displacement components along an identical first dimension ( ) can be distributed in the same class of the first histogram based on their identical components.

According to a third method of classifying motion vectors, the vectors having displacement components along an identical first dimension ( ) can be distributed in the same class of motion vectors of a second histogram . For example, two vectors And having components of displacements along an identical second dimension ( ) can be distributed in the same class of the second histogram based on their identical components.

In one or more embodiments, a new class of vectors of each histogram may be created to distribute therein a motion vector for which no class of the histogram for distributing the vector based on the matching criterion does not yet exist when distributing the vector in one of the classes of the histogram. As a variant, it will be possible to generate each set of classes by filling in tables stored in memory whose entries correspond respectively to classes.

It will thus be possible, in one or more embodiments, to distribute all or part of the motion vectors generated during the motion estimation analysis in two histograms, a first histogram bringing together the motion vectors having displacement components according to a first dimension (e.g. coordinates ) identical in classes, and the second histogram gathering the motion vectors having displacement components according to a second dimension (for example coordinates ) identical in classes. The first histogram H4 can therefore represent the numbers of coordinates identical, and the second histogram H5 represent the numbers of coordinates identical.

In one or more embodiments, the motion vectors distributed in classes of the first and/or second histograms may be selected beforehand from among the motion vectors determined on the basis of a relevance criterion, as described above for scenarios where only one histogram is used. Here again, the purpose of the relevance criterion used may be to avoid selecting for the distribution in classes of the histogram motion vectors that are not very relevant, for example with regard to a corresponding distortion metric value, and which would have could be selected for lack of better vectors obtained during the motion estimation analysis. The relevance criterion may therefore, in one or more embodiments, be advantageously defined on the basis of one or more distortion metrics.

Therefore, in one or more embodiments, the relevance criterion may comprise a comparison of a distortion metric value calculated for the motion vector with a predetermined threshold. The predetermined threshold may advantageously be chosen according to the type of distortion metric used for the relevance criterion.

In one or more embodiments, the distortion metric value used may be of the SAD type, as described above. In other embodiments, the distortion metric value used may also use a so-called “AUTOSAD” metric, for example when using a criterion of the SAD/AUTOSAD type (ratio of the value of SAD to AUTOSAD value), as described above.

It will be possible in particular to choose, in one or more embodiments using a relevance criterion using a distortion measurement of the SAD/AUTOSAD type, a comparison threshold in the interval [-1/2; +8], for example equal to 2.

Thus, in one or more embodiments, two histograms And are generated from motion vectors of the set of motion vectors determined for the current image, on the basis of a correspondence criterion according to which the vectors having identical displacement components according to a dimension of the image of corresponding reference are distributed in the same class of the respective histogram, possibly after having been selected on the basis of a relevance criterion comparing a value of SAD/AUTOSAD which is associated with them with a predetermined threshold, for example chosen equal to 2 For example, the vectors resulting from the motion estimation analysis can first of all be selected on the basis of the criterion of relevance, and can be retained only the vectors whose value of SAD/AUTOSAD does not exceed not the threshold. Once selected, these vectors can then be distributed in classes in order to constitute the first and second histograms And .

In one or more embodiments, the proposed method may use several histograms, each generated on the basis of a respective method of classifying motion vectors among the set of motion vectors obtained by motion estimation analysis of the current picture. Similar to the generation of histograms And , the motion vectors used for this classification may correspond to at least one displacement parameter, for example in the form of one or more displacement coordinates along directions of a standardized reference frame of the reference image, and to a distance parameter between the blocks to which the vector corresponds located respectively in the current image and in a reference image.

In one or more embodiments, as described above for histograms And , the correspondence criterion may comprise an identity criterion, for two motion vectors, of the respective values of components representing a translation along a respective dimension of the corresponding reference image. This correspondence criterion can thus be applied for each of the dimensions of the current image, to generate a corresponding histogram.

Histogram classes can thus be generated as described above for histograms And to build two histograms And .

In one or more embodiments, the histograms And may each represent combinations of values of a distortion criterion (calculated for the vectors of each class), for example the SAD, respectively associated with the vectors of each of their respective classes. Thus, in one or more embodiments, two histograms And may be generated from motion vectors of the set of motion vectors determined for the current image, on the basis of correspondence criteria according to which the vectors having identical motion components along a respective dimension are distributed in the same class, by representing for each class a combination of the values of SAD of the vectors of the class.

For example, for each class of each of the histograms And , we can calculate a linear combination of the values of SAD corresponding to the motion vectors of the class. In particular, in one or more embodiments, it will be possible to calculate an average (for example an arithmetic average) of the values of SAD corresponding to the motion vectors of the class. For example, if the histogram (Or ) includes C classes , each class including a number motion vectors with which are respectively associated SAD values , we calculate a combination of these SAD values for the class , for example an average of these SAD values of type: .

Thus, in one or more embodiments, the histogram may represent the averages of the values of SADs determined for the vectors of the respective classes of vectors at the coordinates identical, and the histogram may represent the averages of the values of SADs determined for the vectors of the respective classes of vectors at the coordinates identical. It is possible, for example, to calculate an average of the values of SADs associated with the motion vectors of the same class, for one or more classes, or even each of the classes of the histograms And .

In one or more embodiments, no selection may be made on the vectors resulting from the motion estimation analysis prior to the generation of the histogram Or .

Alternatively, similarly to the selection scheme described above for the histogram a selection can be made on the motion vectors resulting from the motion estimation analysis on the basis of a relevance criterion, for example by comparing a value of SAD/AUTOSAD associated with each vector with a predetermined threshold, by example chosen equal to 2, so as not to distribute in the classes of one and/or the other of the histograms irrelevant vectors which may have been retained during the motion estimation for lack of finding better vectors ( for example in terms of distortion). For example, the vectors resulting from the motion estimation analysis can first of all be selected on the basis of the criterion of relevance, and can be retained only the vectors whose value of SAD/AUTOSAD does not exceed the threshold. Once selected, these vectors can then be distributed in respective classes in order to constitute two histograms And , the distribution of vectors into classes being done similarly to what is described above for histograms And , according to one or more embodiments.

In one or more embodiments, the histograms And may each represent combinations of values of a distortion criterion (calculated for the vectors of each class), for example the SAD, respectively associated with the vectors of each of their respective classes.

Thus, in one or more embodiments, two histograms And may be generated from motion vectors of a set of motion vectors determined for the current image, on the basis of correspondence criteria according to which the vectors having identical motion components along a respective dimension are distributed in the same class, by representing for each class a combination (for example an average, for example arithmetic) of the values of SAD of the vectors of the class, after having been selected on the basis of a criterion of relevance comparing a value of SAD/AUTOSAD which is associated with them with a predetermined threshold, for example chosen equal to 2.

For example, for each class of each of the histograms And , we can calculate a linear combination of the values of SAD corresponding to the motion vectors of the class. In particular, in one or more embodiments, it will be possible to calculate an average (for example an arithmetic average) of the values of SAD corresponding to the motion vectors of the class. For example, if the histogram (Or ) includes C classes , each class including a number motion vectors with which are respectively associated SAD values , we calculate a combination of these SAD values for the class , for example an average of these SAD values of type: .

For example, the histogram may represent the averages of the values of SADs determined for the vectors of the respective classes of motion vectors at the coordinates identical, and the histogram may represent the averages of the values of SADs determined for the vectors of the respective classes of motion vectors at the coordinates identical, the vectors used to generate the histograms H8 and H9 having been previously selected from among the set of motion vectors generated by the motion estimation analysis on the basis of a relevance criterion. It is possible, for example, to calculate an average of the values of SADs associated with the motion vectors of the same class, for one or more classes, or even each of the classes of the histograms And .

Different embodiments of determining the global motion vector of the image are now described, with reference to the different embodiments for generating one or more histograms described above.

With reference to the embodiments in which a histogram of type the histogram described above is generated, a global motion vector for the current frame can be determined by selecting the class of the histogram (e.g. histogram ) having the largest number of elements. In the case where several classes of the histogram contain the largest number of elements, an additional selection criterion can be applied. For example, this additional selection criterion may consist in selecting the first class of the histogram from among the classes which contain the most elements. As a variant, this additional selection criterion can use a combination of the distortion values associated with the vectors of each of the classes which contain the most elements, such as for example an average of these distortion values.

The selected class corresponds to motion vector components, on the basis of which the class was formed. For example, the selected class corresponds to displacement components according to respective dimensions and or motion vectors included in the class.

A global motion vector can then be determined on the basis of a motion vector corresponding to the selected class. In the previous example, each vector of the class having components And , the global motion vector of the current frame can be determined to have these components And : .

With reference to the embodiments in which a histogram of type the histogram Or described above is generated, a global motion vector for the current image can be determined by selecting the class of the histogram (for example the histogram or the histogram H3) corresponding to the inverse value of the combination (for example the average) of the values of SAD highest among the values of combinations (for example the values of average) of values of SAD used to generate the histogram . In the case where several classes of the histogram can be selected because they fulfill the selection criterion, an additional selection criterion can be applied. For example, this additional selection criterion may consist in selecting the first class of the histogram from among the classes which can be selected.

To take the example of the histogram (this example being transposable to the embodiment of the histogram for what follows) in the case where the histogram understand classes , each class including a number motion vectors with which are respectively associated SAD values , a combination of these SAD values can be calculated for each class. For example, for the class , we can calculate a combination of SAD values , for example an average of these SAD values of type: . We will thus obtain mean values , each corresponding to a respective class . It will then be possible, in one or more embodiments, to determine the inverses of the mean values , and select a class from the histogram whose mean inverse value is the highest among the respective mean inverse values of the classes of the histogram .

The selected class will correspond to a combination (for example an average) of SAD values for vectors having two-by-two identical displacement components, and therefore indirectly to displacement components of motion vectors, on the basis of which the class has been constituted. For example, the selected class will correspond to displacement components according to respective dimensions And motion vectors corresponding to the selected class.

A global motion vector can then be determined on the basis of a motion vector corresponding to the selected class. In the previous example, each vector of the class having components And , the global motion vector of the current frame can be determined to have these components And : .

With reference to the embodiments in which two histograms of type histograms And described above are generated, the classes of each of these histograms corresponding to a displacement component along a respective dimension (for example a component according to a first dimension (for example horizontal) and a component according to a second dimension (for example vertical)), a global motion vector for the current image can be determined by selecting in each histogram corresponding to a dimension (for example in for the horizontal dimension, and in for the vertical dimension) the class with the largest number of elements. In the case where several classes of one of the histograms contain the greatest number of elements, an additional selection criterion can be applied. For example, this additional selection criterion may consist in selecting the first class of the histogram from among the classes which contain the most elements. Alternatively, this additional selection criterion can use a combination of the distortion values (for example) associated with the vectors of each of the classes which contain the most elements, as an average of these distortion values. In one or more embodiments, this additional selection criterion can test different combinations of classes resulting respectively from the histograms And (of which at least one histogram comprises a plurality of classes with a greater number of elements) to identify whether one or more of these combinations corresponds to one of the motion vectors obtained by the motion estimation analysis. If one of the combinations matches a motion vector obtained by the motion estimation analysis, it may be chosen and the overall motion vector determined based on that combination. For example, when a first histogram ( ) includes classes formed with motion vectors having components along a first dimension identical, and that a first histogram ( ) includes classes formed with motion vectors having components along a second dimension identical, we can select in the first histogram ( ) the class corresponding to the value most represented, i.e. having the largest number of elements, and select in the second histogram ( ) the class corresponding to the value the most represented, that is to say having the greatest number of elements.

The selected classes correspond respectively to motion vector components, on the basis of which each of them has been constituted. For example, the selected classes correspond respectively to displacement components according to respective dimensions And motion vectors respectively comprised in the classes.

A global motion vector can then be determined on the basis of the motion vector components corresponding respectively to the classes selected in the first and second histograms. In the previous example, the components of each of the classes having respectively values And , the global motion vector of the current frame can be determined to have these components And : .

With reference to the embodiments in which two histograms of type histograms And , Or And described above are generated, the classes of each of these histograms corresponding to a displacement component along a respective dimension (for example a component according to a first dimension (for example horizontal) and a component according to a second dimension (for example vertical)), a global motion vector for the current image can be determined by selecting in each histogram corresponding to a dimension (for example in Or for the horizontal dimension, and in Or for the vertical dimension) the class of the histogram corresponding to the inverse value of the combination (for example the average) of the values of SAD highest among the values of combinations (for example the values of average) of values of SAD used to generate the histogram. In the case where several classes of a histogram can be selected because they fulfill the selection criterion, an additional selection criterion can be applied. For example, this additional selection criterion may consist in selecting the first class of the histogram from among the classes which can be selected.

To take the example of the histogram (this example being transposable to the embodiment of the histograms , And for what follows) in the case where the histogram understand classes , each class including a number motion vectors with which are respectively associated SAD values , a combination of these SAD values can be calculated for each class. For example, for the class , we can calculate a combination of SAD values , for example an average of these SAD values of type: . We will thus obtain mean values , each corresponding to a respective class . It will then be possible, in one or more embodiments, to determine the inverses of the mean values , and select a class from the histogram whose mean inverse value is the highest among the respective mean inverse values of the classes of the histogram .

For each histogram, the selected class corresponds to a combination (for example an average) of SAD values for vectors having identical one-dimensional displacement components, and therefore indirectly to displacement components of motion vectors, on the basis which the class was formed. For example, the selected class corresponds, for the first histogram (for example Or ), to a displacement component along a first dimension motion vectors corresponding to the class selected in this first histogram, and for the second histogram (for example Or ), to a displacement component along a second dimension motion vectors corresponding to the class selected in this second histogram.

A global motion vector can then be determined on the basis of the motion vectors corresponding respectively to the classes selected in the first and second histograms. In the previous example, the global motion vector of the current frame can be determined to have the components And : .

Various embodiments proposed for determining a global motion vector for an image of a set of images are described below.

In one or more embodiments, a pre-analysis can be performed on the image to be processed, to cut this image into blocks, and then to perform a motion estimation analysis on these blocks. For example, the current image can be cut into blocks of dimension 16 x 16 pixels, then a motion estimation of these blocks can be performed in order to extract, for each block, a motion vector (determined by components) and a value of SAD associated with this motion vector.

Figure 4a shows an image cut into blocks of equal size. A motion estimation analysis was performed on each of the blocks of the image based on a reference image, generating at least one motion vector per block. Identical motion vectors (for example whose components correspond respectively to a horizontal displacement and a vertical displacement are two by two identical) are indicated by hatching of identical orientation. In the example illustrated, the motion estimation analysis of the image generates, for a given reference image, sets of identical motion vectors comprising respectively 51 vectors (of components (59x; 85y)), 11 vectors (of components (56x; 495y)), 6 vectors (of components (19x; 11y)), and 9 vectors (of components (1050x; 32y)). A value of SAD/AUTOSAD can be calculated for each vector, then compared to a predetermined threshold. In the example illustrated in figure 4a, among the 51 component vectors (59x; 85y), 45 vectors have an SAD/AUTOSAD value lower than 2 (the other 6 vectors having an SAD/AUTOSAD value higher than 2) , among the 11 component vectors (56x; 495y), 10 vectors have an SAD/AUTOSAD value less than 2 (1 other vector having an SAD/AUTOSAD value greater than 2), among the 6 component vectors (19x; 11y), 1 vector has a value of SAD/AUTOSAD less than 2 (the other 5 vectors have a value of SAD/AUTOSAD greater than 2), and all 9 component vectors (1050x; 32y) have a value of SAD/ AUTOSAD less than 2. In certain embodiments, the 6 component vectors (19x; 11y) are therefore not used for the construction of a histogram for the implementation of the proposed method.

In one or more embodiments, a histogram could then be created from all the collected vectors, without input conditions neither on their respective sizes, nor on their respective SAD values, in order to classify the number of motion vectors identical, for example by distributing the motion vectors having identical components in classes. It will then be possible to choose as global vector of the current image the vector (for example determined by the pair of components Δx and Δy corresponding respectively to dimensions of the current image) the most represented (in number) in this histogram.

In one or more embodiments, the values Δx and Δy of this global motion vector may advantageously be thresholded (clipped) over a predefined interval, such as for example the interval [-512; +511] in order to be normative, and thus respect a maximum size value imposed by a standard specification, for example.

In one or more embodiments, the histogram may be created not from all the collected vectors, but only from the collected vectors that satisfy one or more entry conditions in the histogram. Some of the collected vectors will therefore be selected to create the histogram, based on one or more criteria to be met. For example, in a particular embodiment, a histogram could be created only from collected vectors whose values Δx and Δy are included in a predefined interval, such as for example the interval [-512; +511] (without entry conditions on their SAD values), in order to distribute in classes of the histogram vectors which have a “natively” normative amplitude, and thus for example classify the number of identical vectors which have this normative amplitude. This will advantageously make it possible to avoid modifying the components of the global motion vector once determined so that they are normative. We can then choose as global vector of the current image the vector (for example determined by the pair of components Δx and Δy corresponding respectively to dimensions of the current image) most represented (in number) in the histogram. Given that the histogram will only contain vectors having a normative amplitude, the global vector thus obtained will advantageously not require any thresholding.

In one or more embodiments, a plurality of entry conditions could be used for the creation of the histogram. For example, in a particular embodiment, the histogram could be created solely from motion vectors fulfilling a size criterion and a distortion criterion. The selected vectors could for example be motion vectors whose values Δx and Δy belong to a predefined interval, such as for example the interval [-512; +511], and whose “SAD/AUTOSAD” value is less than 2.

In one embodiment, a self-distortion value (AUTOSAD) may be determined beforehand for all or part of the blocks of the image being processed. The histogram can only be generated by using vectors having a value of SAD/AUTOSAD lower than a predetermined threshold, for example chosen equal to 2. One can then choose as global vector of the current image the vector (for example determined by the pair of components Δx and Δy corresponding respectively to dimensions of the current image) most represented (in number) in this histogram.

FIG. 4b illustrates an example of a histogram constructed from the results of the motion estimation illustrated in FIG. 4a and using a preselection on the basis of a size criterion (the size, for example expressed in number of bits, of the vectors selected to construct the histogram that must belong to a predetermined interval, such as for example an interval of type corresponding to signed n-bit vectors (for example the interval [-512;+511] which corresponds to signed 10-bit vectors) and on the basis of a relevance criterion of the distortion value associated with the vector ( for example the SAD, the relevance of the SAD being measured by comparing a value of SAD/AUTOSAD with a predetermined threshold). Thus, the motion vectors obtained in the example of FIG. 4a which have a size that does not fulfill a pre-selection criterion on the size (for example the 9 vectors of components (1050x; 32y) do not have the desired size of [-512;+512]) and/or which do not fulfill a criterion of relevance of the SAD (for example the 6 vectors of components (59x; 85y), the vector of components (56x; 495y) and the 5 vectors of components (19x; 11y) which have an SAD/AUTOSAD value greater than 2) are not used for the construction of the histogram illustrated in FIG. 4b.

In one or more embodiments, two histograms could be generated from all the collected vectors, without entry conditions either on their respective sizes, or on their respective distortion values (for example their SAD values), in order to classify in each histogram the number of motion vectors having a displacement component ( Or ) identical, for example by distributing the motion vectors having components according to a dimension of the current image that are identical in respective classes of the histograms. For example, in a particular embodiment, the proposed method may include the creation of two histograms comprising for one only the values Δx of the movement vectors (horizontal translations) and for the other only the values Δy of the vectors of motion (vertical translations), without input conditions either on their amplitudes or on their distortion values (for example their SAD values), in order to classify the number of identical Δx and identical Δy in respective histograms.

We can then choose as global vector of the current image the vector (for example determined by the pair of components Δx and Δy corresponding respectively to dimensions of the current image) the most represented (in number) in the corresponding histogram. For example, the resulting global vector may come from the combination of the most represented Δx and Δy values, possibly thresholded over the interval [-512; +512].

In one or more embodiments, two histograms could be generated from all the vectors collected as described above, but using input conditions on their respective sizes and/or on their distortion values (for example their values of SAD) respectively. For example, in a particular embodiment, two histograms can be constructed (one for the Δx and one for the Δy), but only from the values of Δx and of Δy directly included in the interval [-512; +512]. The resulting global vector will thus be derived from the combination of the most represented values of Δx and of Δy and will advantageously not require thresholding.

Depending on the chosen embodiment, certain acts, actions, events or functions of each of the methods described in this document may be performed or occur in a different order from that in which they were described, or may be added, merged or else not be effected or not occur, as the case may be. Further, in some embodiments, certain acts, actions or events are performed or occur concurrently and not sequentially.

Although described through a certain number of detailed exemplary embodiments, the proposed piloting method and the device for the implementation of an embodiment of the method include various variants, modifications and improvements which will become evident in skilled in the art, it being understood that these different variants, modifications and improvements are part of the scope of the present disclosure, as defined by the following claims. Additionally, various aspects and features described above may be implemented together, or separately, or substituted for each other, and all of the various combinations and sub-combinations of the aspects and features are within the scope of this disclosure. Additionally, some systems and equipment described above may not incorporate all of the modules and functions described for the preferred embodiments.

Claims (11)

A method of processing an image in a set of images, wherein the image is cut into a plurality of blocks, the method comprising:
determining a plurality of motion vectors, each motion vector corresponding to a block of the plurality of blocks of the image, and pointing to a block correlated to the corresponding block in a reference image, the reference image being an image of the set of images distinct from the image;
distributing motion vectors among the plurality of determined motion vectors into classes of a first histogram based on a matching criterion applied, for each classified motion vector, to a component of the motion vector;
selecting a first class of the first histogram based on a selection criterion; And
determining an overall motion vector of the image based on a motion vector corresponding to the first selected class. A processing method according to claim 1, wherein determining a plurality of motion vectors comprises: determining a motion vector for each block of the plurality of blocks of the image. A method according to any preceding claim, wherein the distributed motion vectors are distributed into first classes of the first histogram based on the match criterion applied to a first component of the classified motion vector, and into second classes of a second histogram based on the match criterion applied to a second component of the classified motion vector, the method further comprising: selecting a second class of the second histogram based on the selection criterion; and wherein the overall motion vector of the image is generated based on motion vector components corresponding to the selected first and second classes, respectively. A method according to any preceding claim, wherein the selection criterion determines a class having the greatest number of values. Method according to one of Claims 1 to 3, in which the selection criterion determines a class on the basis of combinations of the values of a distortion metric respectively obtained for the class motion vectors. A method according to claim 5, wherein the overall motion vector of the image is determined based on a motion vector associated with the highest value of the inverse of the combination of values of the distortion metric among the inverses of the respective combinations of the values of the distortion metric respectively obtained for the class motion vectors. Method according to one of the preceding claims, in which the motion vectors distributed in classes are selected from the motion vectors determined on the basis of a relevance criterion. Method according to one of the preceding claims, in which the motion vectors distributed in classes are selected from the motion vectors determined on the basis of a size criterion. Method according to any one of the preceding claims, in which the correspondence criterion comprises a two-by-two identity criterion, for two motion vectors, of the respective values of components representing a translation along a respective dimension of the reference image corresponding. Image processing device comprising:
an input interface configured to receive one image from a set of images;
an image processing unit, comprising a processor, operatively coupled to the input interface, and configured to perform image processing according to the method of any of claims 1 to 9. Computer program, loadable into a memory associated with a processor, and comprising portions of code for the implementation of the steps of a method according to any one of claims 1 to 9 during the execution of said program by the processor.