EP4732537A1

EP4732537A1 - Video data encoding using machine learning techniques that predict image quality

Info

Publication number: EP4732537A1
Application number: EP24742413.8A
Authority: EP
Inventors: Robin Atkins; Jaclyn Anne Pytlarz
Original assignee: Dolby Laboratories Licensing Corp
Current assignee: Dolby Laboratories Licensing Corp
Priority date: 2023-06-22
Filing date: 2024-06-24
Publication date: 2026-04-29
Also published as: CN121533013A; WO2024264023A1

Abstract

A computer-implemented method for encoding video data according to predicted quality values generated by machine learning includes providing a target video data package to a neural network to generate a plurality of predicted quality values for the target video data package, each of the plurality of predicted quality values associated with a different set of target encoding parameters from a range of encoding parameters, the neural network trained using training data including a plurality of reference video data packages and reference quality values calculated for each reference video data package as encoded according to different sets of reference encoding parameters from the range of encoding parameters, setting target encoding parameters for the target video data package based on the plurality of predicted quality values, and sending a control signal to an encoder module to encode the target video data package using the target encoding parameters.

Description

VIDEO DATA ENCODING USING MACHINE LEARNING TECHNIQUES THAT PREDICT IMAGE QUALITY

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority from U.S. Provisional Patent Application Ser. No. 63/509,585, filed on 22 June, 2023 and European Application No. 23190618.1 filed on 9 August 2023, each of which is incorporated by reference herein in its entirety.

TECHNICAL FIELD

[0002] The present disclosure relates to image analysis and, more particularly, to machinelearning-implemented image analysis techniques for video data. Such analysis may be used to set one or more encoding parameters for video data.

SUMMARY

[0003] Encoding or compressing video data provides a variety of technical benefits. For example, encoded video data tends to be significantly smaller than the source video data. In addition to reducing storage space requirements, reducing video data size facilitates faster buffering and streaming of video data to end users, reduces bandwidth requirements for content delivery systems, increases compatibility across devices and platforms, and — since smaller video data packages require less computational power to process and transmit — provide significant energy savings by reducing computational demand for data centers and/or content delivery networks.

[0004] While encoding video data has many benefits, lossy compression algorithms often remove or reduce some visual information. Lossy compression algorithms can introduce artifacts and/or blurriness to encoded video data — particularly if lower bitrates and/or aggressive compression settings are used. Furthermore, video encoding can be computationally intensive and time-consuming, and video encoding often requires users to select from a variety of complex encoding parameters that can affect quality, file size, and compatibility. Since it can be computationally intensive and time-consuming to encode video data, selecting encoding parameters through iterative trial-and-error processes is often inefficient.

[0005] Furthermore, while video quality values may be calculated to quantify quality losses of encoded video data, these values typically require comparing the encoded video data to the original source video data. Thus, such quality values can only be generated after both (i) the original (or source) unencoded source video data is encoded and decoded and (ii) additional computational processes are performed that compare the source video data and the encoded video data. Since the computational processes required to analyze and compare the unencoded and encoded video data can often be nearly as computationally intensive as the encoding process, applying video quality values to estimate the video quality by comparing encoded video data to the source video data is often more computationally intensive than the encoding process itself. Additionally, since quality values depend on the statistics of the content (each video data package, such as a data file containing a complete piece of content or otherwise related content), the quality values generally need to be generated for each different piece of content (each different video data package) and/or regenerated when the encoding parameters used to encode the video data change. In some examples, each video data package may include a collection of video data related to a single piece of video content.

[0006] Given the intense computational demands associated with (i) encoding video data and (ii) analyzing the encoded data against the unencoded version, it may be computationally infeasible to perform real-time or near-real-time video quality analysis on encoded video data. What is needed, therefore, are optimized image analysis techniques that can dramatically improve the computational throughput of computer systems that generate encoded video at a particular visual quality. Such novel and inventive high-throughput computational techniques quickly generate encoded streams with predicted video quality and/or set — in real-time or near- real-time (for example, at software modules such as video rate distortion controllers) — encoding parameters for a particular piece of content based on the predicted video quality, which greatly reduces the overall computational burden and increases the throughput of the video data analysis and encoding process.

[0007] Various aspects of the present disclosure relate to high-throughput systems and methods for analyzing source video data and generating predicted video quality values across an envelope of encoding parameters, wherein encoding parameters can be set based on such predictions. In some examples, an envelope of encoding parameters may include one or more ranges of one or more types of encoding parameters.

[0008] For example, a computer-implemented method for encoding video data according to predicted quality values generated by machine learning includes providing a target video data package to a neural network to generate a plurality of predicted quality values for the target video data package, each of the plurality of predicted quality values associated with a different set of target encoding parameters from a range of encoding parameters, the neural network trained using training data including a plurality of reference video data packages and reference quality values calculated for each reference video data package as encoded according to different sets of reference encoding parameters from the range of encoding parameters, setting target encoding parameters for the target video data package based on the plurality of predicted quality values, and sending a control signal to an encoder module to encode the target video data package using the target encoding parameters.

[0009] In other features, the plurality of predicted quality values includes a first surface defining a first predicted quality value as a function of encoding parameters and a second surface defining a second predicted quality value as a function of encoding parameters. Setting target encoding parameters for the target video data package based on the plurality of predicted quality values includes selecting encoding parameters that maximize the first predicted quality value while meeting a quality threshold for the second predicted quality value. In other features, the predicted quality values include at least one of a peak signal-to-noise ratio (PSNR) metric, a structural similarity index measure (SSIM) metric, a multi-scale structural similarity index measure (MS-SSIM) metric, a Video Quality Metric (VQM) metric, a Video Multi-method Assessment Fusion (VMAF) score, and a bitrate.

[0010] In other features, the target encoding parameters includes at least one of an encoding codec, a quantization parameter (QP), an output bitrate, and an output resolution. In other features, the plurality of predicted quality values are represented as coefficients for a pre-defined parametric equation. In other features, the plurality of predicted quality values include independent variables for the parametric equation and the independent variables represent encoding parameters. In other features, the plurality of predicted quality values includes dependent variables for the parametric equation and the dependent variables represent predicted quality values. In other features, the parametric equation defines a differentiable curve. In other features, the parametric equation defines a differentiable surface. In other features, the method includes generating the training data by encoding each of the plurality of reference video data packages with each different set of encoding parameters.

[0011] In other features, the method includes providing metadata for the target video data package to the machine learning model, the metadata including at least one of a file type, selected target encoding parameters, intended playback software, and an intended playback device type. In other features, the method includes calculating an actual quality value by comparing an encoded target video data package with the target video data package and retraining the machine learning model using the actual quality value. In other features, the machine learning model includes a first machine learning model configured to generate a plurality of predicted first quality values and a second machine learning model configured to generate a plurality of predicted second quality values. In other features, a non-transitory computer-readable medium includes executable instructions, which when executed by an electronic processor of a machine causes the machine to perform the method.

[0012] In some embodiments, a system includes memory hardware configured to store instructions and processor hardware configured to execute the instructions. The instructions include providing a target video data package to a neural network to generate a plurality of predicted quality values for the target video data package, each of the plurality of predicted quality values associated with a different set of target encoding parameters from a range of encoding parameters, the neural network trained using training data including a plurality of reference video data packages and reference quality values calculated for each reference video data package as encoded according to different sets of reference encoding parameters from the range of encoding parameters, setting target encoding parameters for the target video data package based the plurality of predicted quality values, and sending a control signal to an encoder module to encode the target video data package using the target encoding parameters.

[0013] Other examples, embodiments, features, and aspects will become apparent by consideration of the detailed description and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0014] FIG. 1 is a block diagram of an example implementation of a system for implementing machine- learning-enabled techniques for analyzing source video data packages and generating predicted video quality values across an envelope of encoding parameters.

[0015] FIG. 2 is a flowchart of an example process for training a machine learning model and analyzing source video data packages using the trained machine learning model and generating predicted video quality for any given set of encoding parameters or across an entire envelope of encoding parameters.

[0016] FIGS. 3A-3B are flowcharts of an example process for generating training data suitable for training a machine learning model.

[0017] FIG. 4 is a flowchart of an example process for calculating an objective quality value for encoded video data. [0018] FIG. 5 is a flowchart of an example process for calculating a structural similarity index measure quality metric value for encoded video data.

[0019] FIG. 6 is a flowchart of an example process for converting video data to video tensors suitable for input to neural networks.

[0020] FIGS. 7A-7B are flowcharts of an example process for training a machine learning model.

[0021] FIGS. 8 and 9 show example predicted quality values generated by a trained machine learning model.

[0022] In the drawings, reference numbers may be reused to identify similar and/or identical elements.

DETAILED DESCRIPTION

[0023] FIG. 1 is a block diagram of example implementations of a system 100 including a machine learning platform 102 and a video processing platform 104. As will be described herein, the system 100 analyzes source video data (such as unencoded video data and/or video data that has been encoded at very high bitrates and/or quality settings [for example, a mezzanine video file encoded using the JPEG 2000 standard]) and generates predicted image quality values across an envelope of encoding parameters. For example, an envelope of encoding parameters may include one or more ranges of one or more types encoding parameters. The predicted image quality values are used to set encoding parameters for the source video data — for example, at a rate distortion controller. As illustrated in FIG. 1, the system 100 may include a communications system 106. The platforms of the system 100 — such as machine learning platform 102 and video processing platform 104 — may communicate via the communications system 106. Examples of the communications system 106 may include one or more networks, such as a General Packet Radio Service (GPRS) network, a Time-Division Multiple Access (TDMA) network, a Code-Division Multiple Access (CDMA) network, a Global System of Mobile Communications (GSM) network, an Enhanced Data Rates for GSM Evolution (EDGE) network, a High-Speed Packet Access (HSPA) network, an Evolved High-Speed Packet Access (HSPA+) network, a Long Term Evolution (LTE) network, a Worldwide Interoperability for Microwave Access (WiMAX) network, a 5th-generation mobile network (5G), an Internet Protocol (IP) network, a Wireless Application Protocol (WAP) network, or an IEEE 802.11 standards network, as well as any suitable combination of the above networks. In various implementations, the communications system 106 may also include an optical network, a local area network, and/or a global communication network, such as the Internet.

[0024] In some examples, the machine learning platform 102 includes shared system resources 108, a communications interface 110, and one or more data stores that include non- transitory computer-readable storage media, such as storage 112. In some implementations, the shared system resources 108 include one or more electronic processors, one or more graphics processing units, volatile computer memory, non-volatile computer memory, and/or one or more system buses connecting the components of the shared system resources 108, communications interface 110, and/or storage 112. In various implementations, storage 112 includes one or more software modules, such as machine learning training module 120 and/or machine learning module 122. Additional functionality of machine learning training module 120 and machine learning module 122 will be described further on in this specification with reference to FIGS. 2- 7B.

[0025] In some embodiments, the video processing platform 104 includes shared system resources 114, a communications interface 116, and one or more data stores that include non- transitory computer-readable storage media, such as storage 118. In some examples, the shared system resources 114 include one or more electronic processors, one or more graphics processing units, volatile computer memory, non-volatile computer memory, and/or one or more system buses connecting the components of the shared system resources 114, communications interface 116, and/or storage 118. In various implementations, storage 118 includes one or more software modules, such as video encoding module 124, video analysis module 126, and/or rate distortion controller module 128. Additional functionality of video encoding module 124, video analysis module 126, and/or rate distortion controller module 128 will be described further on in this specification with reference to FIGS. 2-7B.

[0026] In various implementations, components of the machine learning platform 102 communicate with components of the video processing platform 104 via the communications system 106. For example, components of the machine learning platform 102 communicate with the communications interface 110, and components of the video processing platform 104 communicate with communications interface 116. Communications interface 1 10 and communications interface 116 may then communicate with each other via the communications system 106.

[0027] While the machine learning platform 102 and the video processing platform 104 are shown as two separate platforms in FIG. 1, the machine learning platform 102 and the video processing platform 104 may also be integrated into the same platform. For example, shared system resources 108 and shared system resources 1 14 may include the same components. Similarly, storage 112 and storage 112 may include the same components. In some implementations, the machine learning platform 102 and the video processing platform 104 may communicate over dedicated wired and/or wireless connections. In some examples, machine learning platform 102 may be implemented as a sub-component of video processing platform 104.

[0028] FIG. 2 is a flowchart of an example process 200 for (i) training a machine learning model, (ii) analyzing source video data using the trained machine learning model, and (iii) generating predicted image quality values for any given set of encoding parameters or for an entire envelope of encoding parameters. It should be understood that training process described herein is optional as this part may be performed by a separate process, wherein, in such embodiments, the process 200 includes only the analysis and generation of values as described herein. The process 200 is described herein as being performed by the machine learning platform 102. However, it should be understood that the process 200 or portions thereof may be performed by a collection of platforms or devices.

[0029] At 202, the machine learning training module 120 generates training data. The machine learning training module 120 may generate training data from a library of reference video data — such as reference video data stored in storage 112 and/or storage 118. In some examples, the training data includes source video data as reference video data. The training data may also include — for each reference video data package (such as video data relating to each piece of content): (i) one or more different sets of encoding parameters, (ii) reference quality values calculated for the reference video data package as encoded according to each of the different sets of encoding parameters, and/or (iii) reference video tensor(s) suitable for input into a machine learning model. In some embodiments, the encoding parameters include different classes of parameters. For example, the encoding parameters may include an encoding codec, a quantization parameter (QP) for the encoder, an output bitrate, an output resolution, an output framerate, output audio parameters, constant rate factor (CRF) configurations, group of pictures (GOP) length, and/or buffer size. In some examples, the one or more encoding parameters include combinations of a single class of parameters. In other examples, the one or more encoding parameters include combinations of multiple classes of parameters.

[0030] For example, the different sets of encoding parameters may include different combinations of first encoding parameters (such as QP values) and second encoding parameters (such as bitrate values) across an envelope of encoding parameters. As one non-limiting example, if the encoding parameters include n unique first encoding parameters and m unique second encoding parameters, the different sets of encoding parameters may include n X m unique combinations of the first and second encoding parameters. In some examples, the unique combinations of first and second encoding parameters are stored in an n x m array or matrix. In some examples, the first encoding parameters include QP values in a range of between about 16 and about 32. In other examples, the first encoding parameters include QP values in a range of between about 0 and about 20. In various implementations, the first encoding parameters include QP values in a range of between about 20 and about 30. In some embodiments, the first encoding parameters include QP values in a range of between about 30 and about 40. In example embodiments, the first encoding parameters include QP values in a range of between about 40 and about 51.

[0031] In various implementations, the second encoding parameters include output resolutions in a range of between about 320 vertical lines and about 3840 vertical lines. In some examples, the second encoding parameters include output bitrate values in a range of between about 15 megabits per second (Mbps) and about 40 Mbps. In example embodiments, the second encoding parameters include output bitrate values in a range of between about 8 Mbps and about 12 Mbps. In other examples, the second encoding parameters include output bitrate values in a range of between about 2 Mbps and about 8 Mbps. In various implementations, the second encoding parameters include output bitrate values in a range of between about 300 kilobits per second (Kbps) and about 2 Mbps. In some embodiments, the encoder codec may be specified as the High Efficiency Video Coding (HEVC) MainlO codec. For example, the encoder codec may be the X.264 codec for creating digital video streams in the H.264/MPEG-4 AVC video coding format. In other examples, the encoder codec may be the X.265 codec for creating digital video streams in the HEVC/H.265 video compression format.

[0032] The reference quality values may include values used to evaluate the performance and effectiveness of video codecs and encoding settings — such as, for example, by quantifying how well the encoded video data retains the original source video data’s visual quality while minimizing artifacts and maintaining an acceptable file size or bitrate. Examples of quality values include objective metrics such as peak signal-to-noise ratio (PSNR) metrics, structural similarity index measure (SSIM) metrics, multi-scale structural similarity index measure (MS- SSIM) metrics, Video Quality Metric (VQM) metrics, and/or a Video Multi-method Assessment Fusion (VMAF) score. In various implementations, one or more reference quality values may be used for each reference video data package. The reference quality value may include — for each video data package — a reference quality value corresponding to each unique combination of encoding parameters (each different set of encoding parameters). For example, the reference quality value for a given combination of encoding parameters may indicate how well a reference video data package encoded with a particular combination of encoding parameters performs relative to the corresponding source video data package. In various implementations, reference video data may be pre-processed and converted into a format suitable for input into a machine learning model — such as a neural network. For example, each reference video data package may be pre-processed and converted into corresponding reference video vectors, arrays, and/or tensor(s). Additional details associated with generating training data at 202 will be described with reference to FIGS. 3A-3B.

[0033] At 204, the machine learning training module 120 trains a machine learning model using training data — such as training data generated at 202. In some examples, the machine learning model includes a neural network — such as a convolutional neural network (CNN). Examples of suitable neural networks include 3D convolutional neural networks (3D-CNN), convolutional long short-term memory recurrent neural networks (ConvLSTM), two-stream convolutional networks, long-term recurrent convolutional networks (LRCN), TimeDistributed layer-based networks, and/or transformer-based networks. In various implementations, the trained machine learning model may be stored to storage 112 and/or storage 118. Additional details associated with training the machine learning model using training data will be described with reference to FIGS. 7A-7B.

[0034] At 206, the machine learning module 122 provides (i) a target video data package and, optionally, (ii) one or more different sets of encoding parameters (such as any of the previously described encoding parameters) to the trained machine learning model to generate a predicted quality value for each different set of encoding parameters. The target video data package and optional encoding parameters may be input or selected by a user on a graphical user interface — for example, through a graphical user interface element generated and output to a display of the machine learning platform 102, video processing platform 104, and/or a separate user device (such as a workstation, terminal, mobile phone, tablet, or any other computing device) connected to the machine learning platform 102 and/or the video processing platform 104. In various implementations, the user selects a user interface element corresponding to the target video data package and — in response to the user’s selection — the machine learning platform 102 retrieves the target video data package from one or more data stores (such as storage 112 and/or storage 118). [0035] In some embodiments, the generated predicted quality value may be output to the graphical user interface. In various implementations, the encoding parameters may include any of the classes of encoding parameters included with the training data used to train the machine learning model. In some examples, the trained machine learning model may generate predicted quality values for each combination of encoding parameters (each different set of encoding parameters). In example embodiments, the trained machine learning model may generate predicted quality values that indicate a predicted quality assessment of the target video data package if the target video data package were to be encoded using a particular combination (set) of encoding parameters. For example, if the encoding parameters provided at 206 include n first encoding parameters and m second encoding parameters, the machine learning model may generate n X m predicted quality values — such as one or more predicted quality values correlated to each combination of encoding parameters.

[0036] In various implementations, the predicted quality values may be expressed as an n X m array. For example, predicted values from various quality values may be stored as additional dimensions at each combination (n, m) of encoding parameters. In some embodiments, predicted quality values may be generated for randomly sampled combinations sample of encoding parameters. In other examples, the predicted quality values may be expressed in parametric form — for example, as an equation defining a differentiable curve or a differentiable surface. If the predicted quality values include a single class of predicted quality value for an entire range of a single class of encoding parameters, then the predicted quality values may be expressed as an equation defining a differentiable curve. If the predicted quality values include a single class of predicted quality value for an entire range of two different classes of encoding parameters, then the predicted quality value may be expressed as an equation defining a differentiable surface. In various examples, the trained machine learning model may output one or more parametric equations. In some implementations, the trained machine learning model may output coefficients for one or more parametric equations.

[0037] Equation (1) below illustrates a function F x) that outputs a predicted quality value F for any given input encoding parameter x:

F(x) = a ■ (%) + b (1)

[0038] In equation (1) above, F(x) may define a differentiable curve, and function (x) represents a predefined function. Coefficients a and b are generated by the trained machine learning model. Thus, after the trained machine learning model outputs coefficients a and b. a user could input any encoding parameter x into equation (1) and quickly calculate the predicted quality value F for the given encoding parameter x.

[0039] Equation (2) below illustrates a function G (x, y) that outputs a predicted quality value G for any given input combination of encoding parameters (x, y) :

G (x, y ) = c ■ g (x, y) + d (2)

[0040] In equation (2) above, G(x, y) may define a differentiable surface, and function g(x, y) represents a predefined function. Coefficients c and d are generated by the trained machine learning model. As with equation (1), after the trained machine learning model outputs coefficients c and d. the user could input any combination of encoding parameters (x, y) into equation (1) and quickly calculate the predicted quality value G for the given combination of encoding parameters (x, y). In some examples, equations (1) and/or (2) can be solved for encoding parameters (x, y) if the user knows the desired predicted quality value F and/or G. In various implementations, the concepts described above with reference to equations (1) and (2) can be scaled to include any number and combination of input encoding parameters and any number and combination of output predicted quality values. For example, the machine learning model may output two predicted quality values G (x, y) and G₂(x, y) f°^{r an}y given input combination of encoding parameters (x, y).

[0041] At 208, the machine learning module 122 and/or rate distortion controller module 128 selects, based on one or more threshold quality values, one or more of the plurality of predicted quality values. For example, the machine learning module 122 and/or rate distortion controller module 128 may load one or more threshold quality values. The threshold quality values may represent minimum or targeted values for a particular quality value. For example, the threshold quality values may include a targeted value for one or more visual quality values (VMAF, SSIM, etc.). In some embodiments, the user can input threshold quality values into an input field of the graphical user interface. The machine learning module 122 and/or rate distortion controller module 128 may parse the predicted quality values to select one or more of the predicted quality values for the target video. For example, the machine learning module 122 and/or rate distortion controller module 128 compares the predicted quality values output from the trained machine learning model and the one or more threshold quality values to identify the one or more predicted quality values closest to the target quality values.

[0042] In examples where the machine learning model is configured to output two predicted quality values G₁(x, y) and G₂(x, y), machine learning module 122 and/or rate distortion controller module 128 may be configured to find encoding parameters (x, y) that maximize the quality values G^ and G₂. For example, machine learning module 122 and/or rate distortion controller module 128 sets a first threshold for G_r and finds combinations of encoding parameters (%i,yi) that meet or exceed the first threshold. Machine learning module 122 and/or rate distortion controller module 128 sets a second threshold for G₂ and finds combinations of encoding parameters x₂₁ y₂) that meet or exceed the second threshold. In some embodiments, machine learning module 122 and/or rate distortion controller module 128 finds combinations of encoding parameters (x, y) that meet or exceed both the first quality threshold for G and the second quality threshold for G₂. For example, machine learning module 122 and/or rate distortion controller module 128 finds combinations of encoding parameters (x, y) that are present in both (x₁₍ y_x) and (x₂, y₂)-

[0043] In various implementations, machine learning module 122 and/or rate distortion controller module 128 selects the combination of encoding parameters (x, y) that maximizes the first quality values G_t while still meeting quality constraints (e.g., the second quality threshold) for the second quality values G₂. For example, machine learning module 122 and/or rate distortion controller module 128 selects the combination of encoding parameters from (%i, i) that maximizes G and that also lie on the surface described by G₂ (x y- . In some embodiments, machine learning module 122 and/or rate distortion controller module 128 selects the combination of encoding parameters (x, y) that maximizes the second quality values G₂ while still meeting quality constraints (e.g., the first quality threshold) for the first quality values G . For example, machine learning module 122 and/or rate distortion controller module 128 selects the combination of encoding parameters from (x₂, y₂) that maximizes G₂ and that also lie on the surface described by G₁ x₂,y₂).

[0044] At 210, the machine learning module 122 and/or rate distortion controller module 128 sets target encoding parameters for the target video based on the set of encoding parameters associated with the selected one of the plurality of predicted quality values. For example, machine learning module 122 and/or rate distortion controller module 128 sets target encoding parameters for the target video based on the set of encoding parameters selected at 210. The target encoding parameters may include the encoding parameters associated with the selected predict quality value or may be generated according to these parameters. For example, in some embodiments, depending on how close the corresponding value was to a threshold, the encoding parameters associated with the selected value can be modified (e.g., scaled) when used to set the target encoding parameters for the target video. As noted above, the predicted quality values may be expressed as parametric equations, and, in such implementations, the machine learning module 122 and/or rate distortion controller module 128 solves the parametric equations to determine the target encoding values required to achieve the desired target quality values.

[0045] In various implementations, contextual data (such as metadata) may be provided to the machine learning module 122 and/or rate distortion controller module 128 (for example, at 206 along with the target video data). Examples of metadata include at least one of a file type, the selected target encoding parameters, intended playback software, and an intended playback device type. The machine learning module 122 may automatically select target encoding parameters from the predicted quality values based on the metadata.

[0046] The target encoding parameters may be output to the graphical user interface. At 212, the machine learning module 122 and/or rate distortion controller module 128 passes the target encoding parameters to the video encoding module 124, and the video encoding module 124 encodes — using the target encoding parameters — the target video into an encoded video. In various implementations, the encoded video is stored to storage 1 12 and/or storage 118. In some examples, the machine learning module 122 and/or the video encoding module 124 calculates an actual quality value by comparing the encoded target video data package with the target video data package. The machine learning module 122 then retrains the machine learning model using the actual quality value.

[0047] In some embodiments, the first predicted quality value G generated at 206 may be a predicted quality Q represented by equation (7) below:

[0048] In equation (7) above, Q represents a surface defined as a function of variables x₁ and x₂. For example, may be a first encoder parameter (such as a quantization parameter), x₂ may be a second encoder parameter (such as resolution), and a, b, c, and d may be coefficients predicted by the machine learning model for a given input image.

[0049] In various implementations, the second predicted quality value G₂ generated at 206 may be a predicted bitrate B represented by equation (8) below:

B(x_{1 ;} x₂) = e ■ x ² + f ■ x₂ ² + g ■ x • x₂ + h (8)

[0050] As in equation (7), B represents a surface defined as a function of variables x and x₂ and e. f. g. and h may be coefficients predicted by the machine learning model for the given input image. In some examples, at 208, the rate distortion controller module 128 is instructed to select target quality values by maximizing quality Q at a target bitrate B. For example, since a value for target bitrate B is defined, equation (8) can be solved for one of x and x₂. In various implementations, rate distortion controller module 128 first solves for x in terms of x₂. For example, equation (8) may be expressed as quadratic equation (9) below and solved for x_± using the quadratic formula.

[0051] Rate distortion controller module 128 then substitutes the solution (for x_r) of equation (9) into equation (7) (the surface for quality Q), which then yields two different equations Q'(x₂) that are each a function of a single variable x₂. Rate distortion controller module 128 then finds the maximum of each of the two different equations i?'(x₂). For example, rate distortion controller module 128 calculates the derivative — of each of the two different dx₂ equations Q'(x₂), sets the derivatives equal to zero, and solves for x₂. Rate distortion controller module 128 then finds corresponding values of x by substituting x₂ into the solution of equation (9). Rate distortion controller module 128 then tests equation (7) Q(x₁, x₂) using all valid combinations of x and x₂. Rate distortion controller module 128 saves the value of Q calculated for each combination of x₁ and x₂ to an array Q*.

[0052] At 210, rate distortion controller module 128 selects the combination of x_r and x₂ from Q* corresponding to the highest calculated value of Q. At 212, rate distortion controller module 128 encodes the target video data package using the selected combination of x (for example, a quantization parameter) and x₂ (for example, a resolution). In various implementations, rate distortion controller module 128 passes the selected combination of x_r and x₂ to video encoding module 124 and video encoding module 124 encodes the target video data package using the selected combination of x and x₂.

[0053] While FIG. 2 shows a single machine learning model generating multiple types of predicted quality values, in other implementations, multiple machine learning models may be used to generate predicted quality values. For example, each of the multiple machine learning models may be used to generate a specific type of predicted quality value. Thus, in examples where two types of predicted quality values are needed, a first machine learning model may generate the first type of predicted quality value and a second machine learning model may generate the second type of predicted quality value. In various implementations, a single machine learning model may be trained to output more than one type of predicted quality value. For example, the single machine learning model may be trained to output the first type of predicted quality value and the second type of predicted quality value. In some examples, the single machine learning model may be trained to output any number of types of predicted quality values. [0054] FIGS. 3A-3B are flowcharts of an example process 300 for generating training data. At 302, the machine learning training module 120 may generate combinations of encoding parameters. For example, the machine learning training module 120 selects one or more classes of encoding parameters — such as those previously described with reference to FIG. 2 — and generates random combinations of encoding parameters for each class. At 304, the machine learning training module 120 commands the video encoding module 124 to load a plurality of reference video data packages. The plurality of reference video data packages may be a source video data packages selected for training the machine learning model. At 306, the machine learning training module 120 commands the video encoding module 124 to select the initial reference video data package from the plurality of reference video data packages. At 308, the machine learning training module 120 commands the video encoding module 124 to select an initial combination of encoding parameters from the encoding parameters generated at 302. At 310, the machine learning training module 120 commands the video encoding module 124 to encode the selected reference video data package using the selected combination of encoding parameters. The encoded reference video data package may be saved to storage 112 and/or storage 118.

[0055] At 312, the machine learning training module 120 commands the video analysis module 126 to generate reference quality values for the encoded video data package. For example, the encoded video data package is first decoded. During decoding, the encoded video data package is interpreted according to the codec used to encode the video data package, and the video data package is reconstructed. Decoding can be important because some effects of the encoding process — such as loss of detail, blurring, blocking, and/or ringing — may only be apparent once the video data package is reconstructed. After the encoded video data package is decoded, the machine learning training module 120 generate objective reference quality values by comparing the decoded video data package to the reference video data package. Additional details associated with generating reference quality values will be described further on in this specification with reference to FIGS. 4-6. At 314, the machine learning training module 120 associates the generated reference quality values with the selected reference video data package and the selected combination of encoding parameters. The selected reference video data package, generated reference quality values, and/or the selected combination of encoding parameters may be stored as training data.

[0056] At 316, the machine learning training module 120 determines whether there is another combination of encoding parameters that has not yet been processed at 310-314. In response to determining that another unprocessed combination of encoding parameters is present (“YES” at decision block 316), the machine learning training module 120 commands the video encoding module 124 to select the next combination of encoding parameters and encodes the selected video data package with the selected next combination of encoding parameters at 310. In response to determining that another unprocessed combination of encoding parameters is not present (“NO” at decision block 316), the machine learning training module 120 may compile the generated training data and perform any processing needed to input the training data to the machine learning model. For example, the machine learning training module 132 may convert the selected reference video data package to reference video tensor(s) suitable for input to the machine learning model. The reference video tensor(s) are associated with the selected reference video data package in the training data. Additional details associated with converting reference video data packages to video tensors will be described further on in this specification with reference to FIG. 7.

[0057] At 322, the machine learning training module 120 determines whether there is another reference video data package in the plurality of reference video data packages that has not yet been processed at 308-320. In response to determining that another unprocessed reference video data package is present (“YES” at decision block 322), the machine learning training module 120 commands the video encoding module 124 to select the next reference video data package at 324 and again select the initial combination of encoding parameters at 308. In response to determining that another unprocessed reference video data package is not present (“NO” at decision block 322), the machine learning training module 120 saves the training data. In various implementations, the training data may be saved to storage 112 and/or storage 118.

[0058] In response to determining that another set of unprocessed quality values is present (“YES” at decision block 336), the machine learning training module 120 selects the next set of reference quality values at 338 and again determines whether the selected reference quality values exceed or are equal to the target quality values at decision block 332. In response to determining that another set of unprocessed reference quality values is not present (“NO” at decision block 336), the machine learning training module 120 converts the selected reference video data package to reference video tensor(s) at 320. Additional details associated with converting video data packages to video tensor(s) will be described further on in this specification with reference to FIG. 6.

[0059] FIG. 4 is a flowchart of an example process 400 for calculating an objective quality value — such as a Video Multi-method Assessment Fusion (VMAF) score — for encoded video data (for example, after the encoded video data is decoded). While process 400 describes generating a specific type of quality value, it should be understood that quality values are not limited to perceived quality values (such as VMAF scores). Instead, any suitable quality value may be used (such as any combination of those previously described with reference to process 200). At 402, the video analysis module 126 loads an encoded video data package. At 404, the video analysis module 126 loads a reference video data package corresponding to the encoded video data package. For example, the video analysis module 126 loads the source video data package used to generate the encoded video data package as the reference video data package. At 406, the video analysis module 126 prepares and aligns the encoded video data package and the reference video data package. For example, the video analysis module 126 may prepare the video data packages by analyzing and/or processing the video data packages to ensure that the encoded and reference video data packages have the same resolution, frame rate, and/or duration. The video analysis module 126 may also align the encoded and reference video data packages to ensure that they are synchronized. At 408, the video analysis module 126 decodes the aligned reference video data package into a first sequence of frames. At 410, the video analysis module 126 decodes the aligned encoded video data package into a second sequence of frames.

[0060] At 412, the video analysis module 126 calculates primary values for each pair of corresponding frames from the first sequence of frames and the second sequence of frames. In various implementations, the primary values may include a mean squared error (MSE) metric, a SSIM metric, a visual information fidelity (VIF) metric, a detail loss metric (DLM), and/or a mean co-located pixel difference (MCPD) metric. At 414, the video analysis module 126 normalizes the primary values to a common scale. For example, each primary metric may be normalized to a scale of 0-100. At 416, the video analysis module 126 provides the normalized primary values to a trained machine learning model to generate a perceived quality metric for each corresponding pair of frames. In various implementations, the trained machine learning model may be the VMAF model, and the perceived quality metric may be the VMAF score. At 418, the video analysis module 126 generates an average perceived quality metric for the encoded video data package by averaging perceived quality values for each frame pair of the video data packages.

[0061] FIG. 5 is a flowchart of an example process 500 for calculating a structural similarity index quality measure metric for encoded video data packages. While process 500 describes generating a specific type of quality value, it should be understood that quality values are not limited to structural similarity index measure metrics. Instead, any suitable quality value may be used (such as any combination of those previously described with reference to process 200). At 502, the video analysis module 126 loads the encoded video data package. At 504, the video analysis module 126 loads a reference video data package corresponding to the encoded video data package. For example, the reference video data package may be the unencoded source video data package used to generate the encoded video data package. At 506, the video analysis module 126 aligns the reference video data package and the encoded video data package to ensure that the video data packages are synchronized. At 508, the video analysis module 126 extracts corresponding frames from the reference video data package and the encoded video data package. At 510, the video analysis module 126 converts the extracted frames into grayscale. At 512, the video analysis module 126 calculates the structural similarity index measure metric value for each pair of corresponding converted frames. In various implementations, the structural similarity index measure SS1M may be calculated for each reference frame x and encoded frame y pair (x, y) according to equation (3) below.

SSlM x, y) = l x, y) ■ c(x, y~) - y x,y) (3)

[0062] As shown in equation (3) above, the structural similarity index measure SSIM(x, y) for each reference frame x and encoded frame y pair (x, y) may be the product of a luminance comparison function /(x,y) of the frames, a contrast comparison function c(x, y) of the frames, and a structure comparison function s(x, y) of the frames. In various implementations, the luminance comparison function I (x, y) for any given reference frame x and encoded frame y pair (x, y) may be calculated according to equation (4) below.

[0063] As shown in equation (4), the luminance comparison function l(x, y) may be calculated as a product over sum of the average pixel intensities of the reference frame and the average pixel intensities of the encoded frame fi_y. In some embodiments, a constant c, may be added to the numerator and denominator to avoid instability when the denominator is close to zero.

[0064] In some examples, the contrast comparison function c(x, y) of for any given reference frame x and encoded frame y pair (x, y) may be calculated according to equation (5) below.

[0065] As shown in equation (5), the contrast comparison function c(x, y) may be calculated as a product over sum of the standard deviations of the pixel intensities of the reference frame a_x and the standard deviations of the pixel intensities of the encoded frame a_y. In some embodiments, a constant c₂ may be added to the numerator and denominator to avoid instability when the denominator is close to zero.

[0066] In some examples, the structural comparison function s(x, y) of for any given reference frame x and encoded frame y pair (x, y) may be calculated according to equation (6) below.

[0067] As shown in equation (6), the contrast comparison function c(x, y) may be calculated as the covariance o_xy of the pixel intensities of reference frame x and the encoded frame y divided by the product of the average pixel intensities of the reference frame a_x and the average pixel intensities of the encoded frame a_y. In some embodiments, a constant c₃ may be added to the numerator and denominator to avoid instability when the denominator is close to zero.

[0068] At 514, the video analysis module 126 generates an average structural similarity index measure metric value for the encoded video data packages by averaging structural similarity index measure metric values for each frame pair of the video data packages.

[0069] FIG. 6 is a flowchart of an example process 600 for generating video tensors suitable for input to neural networks. At 602, the machine learning training module 120, machine learning module 122, video encoding module 124, and/or video analysis module 126 loads video data. At 604, the machine learning training module 120, machine learning module 122, video encoding module 124, and/or video analysis module 126 extracts frames from the loaded video data. At 606, the machine learning training module 120, machine learning module 122, video encoding module 124, and/or video analysis module 126 rescales the extracted frames into the desired dimensions required by the neural network. At 608, the machine learning training module 120, machine learning module 122, video encoding module 124, and/or video analysis module 126 normalizes pixel values in each extracted frame. In some examples, the pixel values may be normalized to a range of between about 0 and about 1. In various implementations, the pixel values may be normalized to a range of between about -1 and about 1.

[0070] At 610, the machine learning training module 120, machine learning module 122, video encoding module 124, and/or video analysis module 126 converts the processed frames to a multidimensional input matrix (or a tensor). In various implementations, the RGB channels of each frame may be separated. This results in three two-dimensional matrices for each frame. In some embodiments, the processed frames may be stacked along the temporal dimensions to create a three-dimensional matrix (height X width x time) or a higher-dimensional matrix (height X width X channels X time). Thus, the video tensor is a spatiotemporal representation of the video data.

[0071] FIGS. 7A-7B are flowcharts of an example process 700 for training a machine learning model. At 704, the machine learning training module 120 initializes the machine learning model. In some examples, the machine learning model may include a neural network. The machine learning training module 120 may define the architecture of the neural network — including the number of layers, neurons per layer, and activation functions. The machine learning training module 120 may initialize weights and biases of the neural network with small random values — for example, using Xavier or He Initialization methods. In various implementations, the input layer is sized to accept video tensors for single-channel (Luma or Y) 4K (3840x2160 pixel) video. For example, every three adjacent frames of the 4K video may be concatenated together, resulting in an input layer with dimensions 3840x2160x3. In some embodiments, subsequent layers of the neural network are constructed of decreasing width and height but increasing depth using learnable convolutional filters. In example implementations, the output layer is sized to correspond with the indices of a defined table of possible encoding parameters. In other examples, the output layer is sized to return optimal encoding parameters directly (for example, scalar values corresponding to QP and resolution).

[0072] At 704, the machine learning training module 120 loads the training data — such as training data generated at 202 of process 200 and/or according to process 300. At 706, the machine learning training module 120 divides the training data into batches. In various implementations, the batches include a predefined number of reference videos (and their corresponding reference quality values, encoding parameters, and/or video tensor(s)). In some examples, the batches include 32, 64, or 128 reference videos. In various implementations, each video is broken into batches, and each batch is sized to include a certain length of the video. For example, each batch may be sized to include tensors correlating to two-second video segments.

[0073] At 708, the machine learning training module 120 selects the initial batch of the training data. At 710, the machine learning training module 120 selects the initial reference video data package in the selected batch. At 712, the machine learning training module 120 provides the reference video tensor and/or encoding parameters associated with the selected reference video data package to the initialized neural network to generate an output. At 714, the machine learning training module 120 calculates an error between the output and the reference quality values associated with the selected reference video data package. At 716, the machine learning training module 120 determines whether the end of the selected batch has been reached. In response to determining that the end of the selected batch has not been reached (“NO” at decision block 716), the machine learning training module 120 selects the next reference video data package in the selected batch at 718 and provides the reference video tensor and/or encoding parameters associated with the selected reference video data package to the initialized neural network at 712. In response to determining that the end of the selected batch has been reached (“YES” at decision block 716), the machine learning training module 120 calculates an average loss for the selected batch by averaging errors (for example, errors calculated at 714) for all reference video data packages in the selected batch at 720.

[0074] At 724, the machine learning training module 120 calculates a gradient of the average loss with respect to the neural network’s weights and biases using the chain rule of calculus. At 726, the machine learning training module 120 updates the weights and biases of the neural network using the calculated gradients and a predefined learning rate. At 728, the machine learning training module 120 determines whether the end of the epoch is reached. In some examples, an epoch is completed after an error has been calculated data associated with each reference video data package in the training dataset. In other words, an epoch is completed after the machine learning training module 120 cycles through the entire training dataset. In response to determining that the end of the epoch has not been reached (“NO” at decision block 728), the machine learning training module 120 selects the next batch at 730 selects the initial reference video data package in the selected batch at 710.

[0075] In response to determining that the end of the epoch has been reached (“YES” at decision block 728), the machine learning training module 120 determines whether a training condition is met. In some examples, the training condition may be whether the average errors between the output and reference quality values are below a threshold. In various implementations, the threshold may be about 5%. In other examples, the training condition may be met when a certain number of epochs is reached. In response to determining that the training condition has not been met (“NO” at decision block 732), the machine learning training module 120 again selects the initial batch at 708. In response to determining that the training condition has been met (“YES” at decision block 732), the machine learning training module 120 saves the neural network with updated weights and biases as the trained neural network.

[0076] FIGS. 8 and 9 show example predicted quality values generated by the trained machine learning model. In the examples of FIGS. 8 and 9, the predicted quality values are parametric equations that each define a surface as a function of resolution and bitrate. [0077] The systems and methods described in this specification provide a variety or novel and inventive solutions to technical problems related to digital video analysis systems. For example, as previously described, source video data packages — particularly high resolution, high bitrate data packages — tend to be large. To reduce computational, storage, and/or energy demands for data centers and/or content delivery systems, it is often desirable to encode video data packages to reduce their file sizes. Because encoded video data packages typically have smaller file sizes than raw source video data packages, they tend to require less space to store (and it tends to be more computationally efficient to deliver and play them back). However, as previously discussed, finding the optimal encoding parameters may be difficult and computationally intensive. While quality values — such as those described in this specification — quantitatively describe the relative quality losses of encoded video data compared to their unencoded sources, the cumulative process of generating an encoded video data object and calculating quality values for the encoded video data object is ultimately more computationally intensive than encoding the video data object in the first place. For example, generating quality values requires users (i) first encode the source video data object and then (ii) perform a series of computationally intensive operations comparing the encoded video data object to the unencoded source video data object to generate the quality values. It is often infeasible to perform such operations in real-time or near-real-time.

[0078] The systems and methods described in this specification provide novel and inventive solutions that allow for predicted performance values to be generated for target video data and a given combination of encoding parameters (or an entire envelope of predicted performance values to be generated for the target video data and an entire envelope of encoding parameters) without first having to encode the target video data object and perform computationally intensive comparison operations. By drastically reducing the computational operations required for users to access predicted performance values, the computational throughput of any computer system that calculates performance values can be greatly increased.

[0079] Furthermore, instead of generating a single set of quality values corresponding to the specific encoding parameters used to encode video data, the systems and methods described in this specification allow for an entire range of predicted quality values to be generated for an entire envelope of encoding parameters all at once. In examples where the range of predicted quality values is generated in parametric form, quality values for combinations of encoding parameters — even encoding parameters that were not input into the machine learning model and/or used to train the machine learning model — can be extracted with high precision (and in real-time or near-real-time). Additionally, after a range of quality values has been generated (for example, as individual numerical values or in parametric form), the required encoding parameters that achieve any desired quality values may be quickly and efficiently determined in real-time or near-real-time.

[0080] Examples where predicted quality values are expressed in parametric equations that define a fully differentiable surface or curve may offer unique technical benefits. For example, the H.265 — or High Efficiency Video Coding (HEVC) — video compression standard uses algorithms that optimize bitrate, video quality, and encoding complexity. These algorithms can be sensitive to small changes in the target bitrate (where even small changes in bitrate may lead to unpredictable fluctuations in encoded video quality). However, because the predicted quality values (such as bitrate) may be generated as fully differentiable surfaces or curves, local minima or maxima are eliminated from the predicted quality values as the bitrate increases or decreases. This reduces unstable responses caused by large changes in bitrate, greatly increasing the overall stability of implementations where target videos are encoded using the H.265 codec.

[0081] The systems and methods described in this specification provide a variety of technical benefits for various practical applications. For example, bitrate ladders include a set of encoded video streams with different bitrates and resolutions that are encoded from the same source video data object. Bitrate ladders are used in adaptive streaming techniques to deliver the best video quality for each viewer. Bitrate ladders allow for video data to be streamed to each device with optimal encoding settings for that device. The systems and methods described in this specification allow encoding parameters for an entire bitrate ladder to be calculated simultaneously. For example, the machine learning model could generate predicted quality values that define two surfaces. One surface could be quality as a function of QP and resolution, and the second surface could be bitrate as a function of QP and resolution. To determine which set of encoding parameters to use for a given ladder sequence, an iso-bitrate curve (which may be all encoding parameters that achieve the target bitrate) could be derived from the second surface. The iso-bitrate curve could be superimposed on the predicted quality surface, and a point with the highest quality could be selected. Additionally or alternatively, the encoding parameters may be found by solving the intersection of the iso-bitrate curve and the predicted quality surface. Encoding parameters corresponding to that point could be selected as target encoding parameters. Because multiple encoding parameters can be found for multiple target bitrates from the surfaces, this process could be completed simultaneously for multiple target bitrates in the ladder (for example, 10 Mbps, 1 Mbps, and/or 100 Kbps), thus deriving encoding parameters for an entire bitrate ladder without repeating the procedure for each target bitrate. [0082] In various implementations, the video processing platform 104 may be implemented as part of an active encoding platform. An active encoding platform may include a system that processes and encodes digital video data packages into various formats and bitrates for streaming, broadcast, or storage. Active encoding platforms may continuously encode and optimize the content in real-time or near-real-time to meet the specific requirements of various playback devices and/or network conditions. Thus, the video processing platform 104 may continuously generate training data based on the video data packages being continuously encoded and analyzed, and the machine learning platform 102 may continuously adjust and/or retrain the machine learning model based on the continuously updated training data. In some implementations, some or all of the reference video data packages used to generate training data (for example, at 202) may be pulled from the active encoding platform (rather than being pulled from the library of reference video data). In such examples, source video data, encoded video data, and/or quality values comparing the source video data to the encoded video data are pre- processed and do not need to be generated during process 200. By eliminating the need to encode video data and generate quality values during execution of process 200, such implementations further decrease computational requirements and increase computational throughput.

[0083] In example embodiments, training data is generated for N reference video data packages (for example, by the machine learning training module 120). Each video data package is encoded with M combinations of randomly generated first and second encoding parameters (for example, quantization parameters and resolutions). The encoding parameters may be specified by the user (for example, via the graphical user interface). For each unique combination of the M randomly generated first and second encoding parameters, (i) the bitrate R of the encoded video data package is recorded as Rfri (where represents a particular first encoding parameter and mi represents a particular second encoding parameter), and (ii) the encoded video data package is decoded and analyzed to generate a quality value D that is recorded as After bitrates R and quality values D are generated for each of the M combinations of first and second encoding parameters, an optimal combination of encoding parameters m_opt(n) is selected to achieve target bitrate r or a target quality value d. In examples where m_opt(n) is selected to achieve the target bitrate r, then m_opt(n is selected as the set of encoding parameters m that achieve a bitrate less than or equal to the target bitrate r while having the highest quality value d. In examples where m_opt(n) is selected to achieve the target quality value d, then m_opt n) is selected as the set of encoding parameters m that achieve a quality value meeting or exceeding the target quality value d while having the lowest bitrate r. [0084] After the training data is generated, the machine learning model (such as a neural network) is trained (for example, by the machine learning training module 120) using backpropagation to predict the optimal encoding parameters m_opt (n) for each reference video data package in the training data. For example, during training, a video tensor corresponding to each reference video data package is input to the machine learning model to generate predicted optimal encoding parameters m_pred (n) . The generated predicted optimal encoding parameters ^mpred(ⁿ) ^are compared to the optimal encoding parameters m_opt(n') (from the training data set for that reference video data package) to generate an error (for example, a difference between m_pred(n) and m_opt(n)), and the parameters of the machine learning model are adjusted until the error is minimized. In inference (after training), a video tensor corresponding to a target video is input to the machine learning model (for example, at the machine learning module 122) and the machine learning model generates predicted optimal encoding parameters m_pred(n) for the target video. The target video data package is then encoded using the predicted optimal encoding parameters m_pre (n) (for example, at the video encoding module 124).

[0085] The following paragraphs provide examples of systems, methods, and devices implemented in accordance with this specification.

[0086] Example 1 includes a computer-implemented method for encoding video data according to predicted quality values generated by machine learning comprising. The method includes providing a target video data package to a neural network to generate a plurality of predicted quality values for the target video data package, each of the plurality of predicted quality values associated with a different set of target encoding parameters from a range of encoding parameters, the neural network trained using training data including a plurality of reference video data packages and reference quality values calculated for each reference video data package as encoded according to different sets of reference encoding parameters from the range of encoding parameters; setting target encoding parameters for the target video data package based on the plurality of predicted quality values; and sending a control signal to an encoder module to encode the target video data package using the target encoding parameters.

[0087] Example 2 includes the subject matter of Example 1 and further specifies that the plurality of predicted quality values includes a first surface defining a first predicted quality value as a function of encoding parameters and a second surface defining a second predicted quality value as a function of encoding parameters, and setting target encoding parameters for the target video data package based on the plurality of predicted quality values includes selecting encoding parameters that maximize the first predicted quality value while meeting a quality threshold for the second predicted quality value.

[0088] Example 3 includes the subject matter of any of Examples 1-2 and further specifies that the predicted quality values include at least one of a peak signal-to-noise ratio (PSNR) metric, a structural similarity index measure (SSIM) metric, a multi-scale structural similarity index measure (MS-SSIM) metric, a Video Quality Metric (VQM) metric, a Video Multi-method Assessment Fusion (VMAF) score, and a bitrate.

[0089] Example 4 includes the subject matter of any of Examples 1-3 and further specifies that the target encoding parameters includes at least one of an encoding codec, a quantization parameter (QP), an output bitrate, and an output resolution.

[0090] Example 5 includes the subject matter of any of Examples 1, 3, and 4 and further specifies that the plurality of predicted quality values are represented as coefficients for a predefined parametric equation.

[0091] Example 6 includes the subject matter of Example 5 and furthers specifies that the plurality of predicted quality values include independent variables for the parametric equation and wherein the independent variables represent encoding parameters.

[0092] Example 7 includes the subject matter of Example 6 and further specifies that the plurality of predicted quality values includes dependent variables for the parametric equation and wherein the dependent variables represent predicted quality values.

[0093] Example 8 includes the subject matter of Example 7 and further specifies that the parametric equation defines a differentiable curve.

[0094] Example 9 includes the subject matter of Example 7 and further specifies that the parametric equation defines a differentiable surface.

[0095] Example 10 includes the subject matter of any of Examples 1-9 and further specifies that the different sets of reference encoding parameters include randomized sets of encoding parameters.

[0096] Example 11 includes the subject matter of any of Examples 1-10 and further specifies generating the training data by encoding each of the plurality of reference video data packages with each different set of encoding parameters. [0097] Example 12 includes the subject matter of any of Examples 1-11 and further specifies training the machine learning model.

[0098] Example 13 includes the subject matter of any of Examples 1-12 and further specifies providing metadata for the target video data package to the machine learning model, the metadata including at least one of a file type, selected target encoding parameters, intended playback software, and an intended playback device type.

[0099] Example 14 includes the subject matter of any of Examples 1-13 and further specifies that the plurality of reference video data packages is generated by an active encoding platform.

[00100] Example 15 includes the subject matter of any of Examples 1-14 and further specifies calculating an actual quality value by comparing an encoded target video data package with the target video data package and retraining the machine learning model using the actual quality value.

[00101] Example 16 includes the subject matter of any of Examples 1-15 and further specifies that the machine learning model includes a first machine learning model configured to generate a plurality of predicted first quality values and a second machine learning model configured to generate a plurality of predicted second quality values.

[00102] Example 17 includes the subject matter of any of Examples 1-16 and further specifies that the neural network includes an input layer and a plurality of subsequent layers, each subsequent layer has a decreased width and height than a previous layer, and each subsequent layer has an increased depth than the previous layer.

[00103] Example 18 includes the subject matter of any of Examples 1-18 and further specifies that training data is repeatedly generated and the machine learning model is repeatedly retrained using the training data.

[00104] Example 19 includes a non-transitory computer-readable medium comprising executable instructions, which when executed by an electronic processor of a machine causes the machine to perform the method of any of Examples 1-18.

[00105] Example 20 includes a system that includes memory hardware configured to store instructions and processor hardware configured to execute the instructions. The instructions include providing a target video data package to a neural network to generate a plurality of predicted quality values for the target video data package, each of the plurality of predicted quality values associated with a different set of target encoding parameters from a range of encoding parameters, the neural network trained using training data including a plurality of reference video data packages and reference quality values calculated for each reference video data package as encoded according to different sets of reference encoding parameters from the range of encoding parameters; setting target encoding parameters for the target video data package based the plurality of predicted quality values; and sending a control signal to an encoder module to encode the target video data package using the target encoding parameters.

[00106] The foregoing description is merely illustrative in nature and does not limit the scope of the disclosure or its applications. The broad teachings of the disclosure may be implemented in many different ways. While the disclosure includes some particular examples, other modifications will become apparent upon a study of the drawings, the text of this specification, and the following claims. In the written description and the claims, one or more steps within any given method may be executed in a different order — or steps may be executed concurrently or omitted — without altering the principles of this disclosure. Similarly, instructions stored in a non-transitory computer-readable medium may be executed in a different order — or concurrently or omitted — without altering the principles of this disclosure. Unless otherwise indicated, the numbering or other labeling of instructions or method steps is done for convenient reference and does not necessarily indicate a fixed sequencing or ordering.

[00107] Unless the context of their usage unambiguously indicates otherwise, the articles “a,” “an,” and “the” should not be interpreted to mean “only one.” Rather, these articles should be interpreted to mean “at least one” or “one or more.” Likewise, when the terms “the” or “said” are used to refer to a noun previously introduced by the indefinite article “a” or “an,” the terms “the” or “said” should similarly be interpreted to mean “at least one” or “one or more” unless the context of their usage unambiguously indicates otherwise.

[00108] Spatial and functional relationships between elements — such as modules — are described using terms such as (but not limited to) “connected,” “engaged,” “interfaced,” and/or “coupled.” Unless explicitly described as being “direct,” relationships between elements may be direct or include intervening elements. The phrase “at least one of A, B, and C” should be construed to indicate a logical relationship (A OR B OR C), where OR is a non-exclusive logical OR, and should not be construed to mean “at least one of A, at least one of B, and at least one of C.” The term “set” does not necessarily exclude the empty set. For example, the term “set” may have zero elements. The term “subset” does not necessarily require a proper subset. For example, a “subset” of set A may be coextensive with set A, or include elements of set A. Furthermore, the term “subset” does not necessarily exclude the empty set.

[00109] In the figures, the directions of arrows generally demonstrates the flow of information — such as data or instructions. However, the direction of an arrow does not imply that information is not being transmitted in the reverse direction. For example, when information is sent from a first element to a second element, the arrow may point from the first element to the second element. However, the second element may send requests for data to the first element, and/or acknowledgements of receipt of information to the first element.

[00110] Throughout this application, the term “module” or the term “controller” may be replaced with the term “circuit.” A “module” may refer to, be part of, or include processor hardware that executes code and memory hardware that stores code executed by the processor hardware. The term “module” may include one or more interference circuits. In various implementations, the interference circuits may implement wired or wireless interfaces that connect to or are part of communications systems. Modules may communicate with other modules using the interference circuits. In various implementations, the functionality of modules may be distributed among multiple modules that are connected via communications systems. For example, functionality may be distributed across multiple modules by a load balancing system. In various implementations, the functionality of modules may be split between multiple computing platforms connected by communications systems.

[00111] The term “code” may include software, firmware, and/or microcode, and may refer to programs, routines, functions, classes, data structures, and/or data objects. The term “memory hardware” may be a subset of the term “computer-readable medium.” The term computer- readable medium does not encompass transitory electrical or electromagnetic signals or electromagnetic signals propagating through a medium — such as on an electromagnetic carrier wave. The term “computer-readable medium” is considered tangible and non-transitory. Modules, methods, and apparatuses described in this application may be partially or fully implemented by a special-purpose computer that is created by configuring a general-purpose computer to execute one or more particular functions described in computer programs. The functional blocks, flowchart elements, and message sequence charts described above serve as software specifications that can be translated into computer programs by the routine work of a skilled technician or programmer.

[00112] It should also be understood that although certain drawings illustrate hardware and software as being located within particular devices, these depictions are for illustrative purposes only. In some embodiments, the illustrated components may be combined or divided into separate software, firmware, and/or hardware. For example, instead of being located within and performed by a single electronic processor, logic and processing may be distributed among multiple electronic processors. Regardless of how they are combined or divided, hardware and software components may be located on the same computing device or they may be distributed among different computing devices — such as computing devices interconnected by one or more networks or other communications systems.

[00113] In the claims, if an apparatus or system is claimed as including an electronic processor or other element configured in a certain manner, the claim or claimed element should be interpreted as meaning one or more electronic processors (or other element as appropriate). If the electronic processor (or other element) is described as being configured to make one or more determinations or one or execute one or more steps, the claim should be interpreted to mean that any combination of the one or more electronic processors (or any combination of the one or more other elements) may be configured to execute any combination of the one or more determinations (or one or more steps).

Claims

CLAIMS What is claimed is:

1. A computer-implemented method for encoding video data according to predicted quality values generated by machine learning, the method comprising: providing a target video data package to a neural network to generate a plurality of predicted quality values for the target video data package, each of the plurality of predicted quality values associated with a different set of target encoding parameters from a range of encoding parameters, the predicted quality values expressed in parametric equations that define a fully differentiable surface or curve each representing a different type of predicted quality value, the neural network trained using training data including a plurality of reference video data packages and reference quality values calculated for each reference video data package as encoded according to different sets of reference encoding parameters from the range of encoding parameters; setting target encoding parameters for the target video data package based on the plurality of predicted quality values; and sending a control signal to an encoder module to encode the target video data package using the target encoding parameters.

2. The method of claim 1 wherein: the plurality of predicted quality values includes: a first surface defining a first predicted quality value as a function of encoding parameters and a second surface defining a second predicted quality value as a function of encoding parameters; and setting target encoding parameters for the target video data package based on the plurality of predicted quality values includes: selecting encoding parameters that maximize the first predicted quality value while meeting a quality threshold for the second predicted quality value.

3. The method of any one of claims 1-2, wherein the predicted quality values include at least one of a peak signal-to-noise ratio (PSNR) metric, a structural similarity index measure (SSIM) metric, a multi-scale structural similarity index measure (MS-SSIM) metric, a Video Quality Metric (VQM) metric, a Video Multi-method Assessment Fusion (VMAF) score, and a bitrate.

4. The method of any one of claims 1-3, wherein the target encoding parameters includes at least one of an encoding codec, a quantization parameter (QP), an output bitrate, and an output resolution.

5. The method of any one of claims 1, 3, and 4, wherein the plurality of predicted quality values are generated using coefficients for a pre-defined parametric equation.

6. The method of claim 5 wherein the plurality of predicted quality values are generated using independent variables for the parametric equation and wherein the independent variables represent encoding parameters.

7. The method of claim 6 wherein the plurality of predicted quality values are generated using dependent variables for the parametric equation and wherein the dependent variables represent encoding parameters.

8. The method of claim 7, wherein the parametric equation defines a differentiable curve.

9. The method of claim 7, wherein the parametric equation defines a differentiable surface.

10. The method of any preceding claim, wherein the different sets of reference encoding parameters include randomized sets of encoding parameters.

11. The method of any preceding claim, further comprising generating the training data by encoding each of the plurality of reference video data packages with each different set of encoding parameters.

12. The method of any preceding claim, further comprising training the machine learning model.

13. The method of any preceding claim, further comprising providing metadata for the target video data package to the machine learning model, the metadata including at least one of a file type, selected target encoding parameters, intended playback software, and an intended playback device type.

14. The method of any preceding claim, wherein the plurality of reference video data packages are generated by an active encoding platform.

15. The method of any preceding claim, further comprising calculating an actual quality value by comparing an encoded target video data package with the target video data package and retraining the machine learning model using the actual quality value.

16. The method of any one preceding claim, wherein the machine learning model includes a first machine learning model configured to generate a plurality of predicted first quality values and a second machine learning model configured to generate a plurality of predicted second quality values.

17. The method of any preceding claim, wherein: the neural network includes an input layer and a plurality of subsequent layers; each subsequent layer has a decreased width and height than a previous layer; and each subsequent layer has an increased depth than the previous layer.

18. The method of any preceding claim, wherein: training data is repeatedly generated and the machine learning model is repeatedly retrained using the training data.

19. A non-transitory computer-readable medium comprising executable instructions, which when executed by an electronic processor of a machine causes the machine to perform the method of any preceding claim.

20. A system comprising: memory hardware configured to store instructions and processor hardware configured to execute the instructions, wherein the instructions include: providing a target video data package to a neural network to generate a plurality of predicted quality values for the target video data package, each of the plurality of predicted quality values associated with a different set of target encoding parameters from a range of encoding parameters, the predicted quality values expressed in parametric equations that define a fully differentiable surface or curve each representing a different type of predicted quality value, the neural network trained using training data including a plurality of reference video data packages and reference quality values calculated for each reference video data package as encoded according to different sets of reference encoding parameters from the range of encoding parameters, setting target encoding parameters for the target video data package based the plurality of predicted quality values, and sending a control signal to an encoder module to encode the target video data package using the target encoding parameters.