BRPI0713321B1 - efficient fixed-point approximations of discrete direct and inverse cosine transforms - Google Patents

Abstract

Efficient fixed-point approximations of discrete direct and inverse cosine transforms. Techniques for approximating computation of an inverse discrete cosine transform using fixed point calculations are described. According to these techniques, scaled coefficient matrices are generated by multiplying the coefficients in coefficient matrices encoded by scaling factors. Next, the polarized coefficient matrices are generated by adding a midpoint bias value to a coefficient dc of the scaled coefficient matrix. Fixed-point arithmetic is then used to apply a transform to the polarized coefficient matrices. values in the resulting arrays are then shifted right to derive arrays of pixel component values. Array of pixel component values are then combined to create pixel arrays. The pixel arrays generated by these techniques closely resemble the uncompressed pixel arrays using the ideal inverse discrete cosine transform ("idct").

Description

(54) Title: EFFICIENT APPROXIMATIONS FOR FIXED POINT OF DIRECT AND REVERSE DISCRETE TRANSFORMED (73) Holder: QUALCOMM INCORPORATED, Sociedade Norte Americana. Address: 5775 Morehouse Drive, San Diego, California 92121-1714, UNITED STATES OF AMERICA (US), North American (72) Inventor: YURIY REZNIK.

Validity Term: 10 (ten) years from 12/04/2018, subject to legal conditions

Issued on: 12/04/2018

Digitally signed by:

Liane Elizabeth Caldeira Lage

Director of Patents, Computer Programs and Topographies of Integrated Circuits

1/69

EFFICIENT APPROACHES FOR FIXED POINTS OF DIRECT AND REVERSE DISCRETE COSINE TRANSFORMED

Field of Invention [001] Disclosure refers to computer and multimedia graphics, specifically the compression of graphics, images, and video information.

Description of the Prior Art [002] Many existing image and video encoding standards employ compression techniques to allow high resolution images and video to be stored, or transmitted as relatively compact files or data streams. Such coding standards include the Photographic Experts Board (JPEG), the Moving Image Experts Group (MPEG) -1, MPEG-2, MPEG-4 part 2, H.261, H.263, and other video coding standards image or video.

[003] According to many of these standards, video frames are compressed using spatial encoding. These frames can be original frames (ie, i-frame) or they can be residual frames generated by a time coding process that uses motion compensation. During spatial coding, frames are divided into equal sized pixel blocks. For example, an uncompressed frame can be split into a set of 8x8 pixel blocks. For each pixel block, the pixel components are separated into arrays of pixel component values. For example, each block

of pixels can be divided into an array in Y values in components in pixel, an array of U values in components in pixel, and an array of V values in components in pixel. In this example, Y values in components pixels indicate luminance and U values and V values of components pixel represent values in

Petition 870180133578, of 9/24/2018, p. 8/169

2/69 chrominance.

[004] Furthermore, during spatial coding, a direct discrete cosine transform (FDCT) is applied to each array of pixel component values in a frame being encoded. An ideal one-dimensional FDCT is defined by:

n = 0 where s is the array of N original values, t is the array of N transformed values, and the coefficients c are given by c (o) = c (£) = 4ΪΪΝ for 1 <k <N - 1.

[005] An ideal two-dimensional FDCT is defined by the formula:

π (2ηι + l) z π (2η + l) y 2N ^C ° ^S 2N

ΛΓ-1 ΛΓ-1 t (i, j) = c (i, Σ ^s ( ^m ') ^cos n = lm = 0 where s is the array of N original values, t is the array of N transformed values, and c ( i, j) is given by c (i, j) = c (zjc (yj, and with c (k) defined as in the one-dimensional case.

[006] A coefficient matrix is produced when the block of pixel component values is transformed using the FDCT. This coefficient matrix can then be quantized and coded using, for example, arithmetic or Huffman codes. A video bit stream represents the combined result of performing this process on all color component value blocks in a series of video frames in an uncompressed series of video frames.

[007] An uncompressed video frame can be derived from a video bit stream by reversing this process. Specifically, for each matrix of coefficients

Petition 870180133578, of 9/24/2018, p. 9/169

3/69 the bit stream is uncompressed and the uncompressed values are de-quantized in order to derive matrices of transformed coefficients. An inverse discrete cosine transform (IDCT) is then applied to each matrix of transformed coefficients in order to derive arrays of pixel component values. An ideal one-dimensional IDCT is defined by:

k = 0 where s is the array of N original values, t is the array of N transformed values, and the coefficients c are given by c (o) = ylí / N, c (k) = 4ΪΪΝ for 1 <k <N - 1 .

An ideal two-dimensional IDCT is defined by the formula:

Λ / -1IV — 1 s (m, n) = /) ^cos í = 0 7 = 0 π (2ηι + l) z 2N

COS π · (2 «+ ι) /

2N

The arrays resulting from pixel component values are then reassembled into pixel blocks and these pixel blocks are reassembled to form a decoded frame. If the decoded frame is an i-frame, the frame is now completely decoded. However, if the uncompressed frame is a predictive frame or a bipredictive frame, the decoded frame is simply a decoded residual frame. A complete frame is generated by building a reconstructed frame using motion vectors associated with the decoded frame and then adding the reconstructed frame to the decoded residual frame.

[008] Under ideal circumstances, no information is lost when using an FDCT to encode or an

IDCT to decode a block of component values from

Petition 870180133578, of 9/24/2018, p. 10/169

4/69 pixel. Consequently, under these ideal circumstances, a decoded version of a video frame is identical to the original version of the video frame. However, computing an FDCT or an IDCT can be computationally difficult because computing the FDCTs and IDCTs involves using real numbers and significant numbers of multiplication operations. For this reason, actual numbers used in FDCTs and IDCTs are often approximated using limited precision numbers. Rounding errors result from the use of numbers of limited precision to represent real number values. Furthermore, quantization and dequantization can contribute to additional errors.

[009] Errors in the compacting and unpacking process can result in significant differences between the original uncompressed frame and the final uncompressed frame. For example, colors in the final uncompressed frame may differ from colors in the original uncompressed frame. Furthermore, errors caused by a mismatch between the implementation of the IDCT encoder and the implementation of the IDCT decoder can accumulate during the encoding and decoding of predicted frame sequences. These accumulated errors are commonly referred to as the IDCT variation.

Summary of the Invention [0010] Techniques to approximate computation of an inverse discrete cosine transform using fixed point calculations. According to these techniques, staggered coefficient matrices are generated by multiplying coefficients in matrices of coefficients encoded by scaling factors. Next, the polarized coefficient matrices are generated by adding a midpoint polarization value to a DC coefficient of the staggered coefficient matrix.

Petition 870180133578, of 9/24/2018, p. 11/169

5/69

Fixed point arithmetic is then used to apply a transform to the matrix of polarized coefficients. Values in the resulting arrays are then shifted to the right to derive arrays of pixel component values. Arrays of pixel component values are then combined to create arrays of pixels. The pixel arrays generated by these techniques closely resemble the decompressed pixel arrays using the ideal inverse discrete cosine transform (IDCT).

[0011] In one aspect, a method comprises scaling each coefficient into an 8x8 matrix of coefficients encoded by one of a factor A, a factor B, a factor C, a factor D, a factor E, a factor F, a factor

G, an H factor, a factor I, or a factor J in order to produce a matrix of staggered coefficients. In this method, / 1 = 1024, 5 = 1138, C = 1730, £) = 1609, £ = 1264, £ = 1922,

G = 1788, H = 2923, I = 2718, and J = 2528. The method also comprises using repeated applications of a unidirectional staggered fixed-point transform to transform the staggered coefficient matrix into a transformed coefficient matrix. In addition, the method comprises transformed right shift coefficients in the transformed coefficient matrix in order to produce an adjusted coefficient matrix. Each adjusted coefficient in the adjusted coefficient matrix approximates a corresponding value in an array of values that would be produced when applying an ideal two-dimensional IDCT to the coded coefficient matrix. Furthermore, the method comprises displaying an 8x8 pixel block. Each pixel in the 8x8 pixel block includes a pixel component value based on an adjusted coefficient in the adjusted coefficient matrix.

Petition 870180133578, of 9/24/2018, p. 12/169

6/69 [0012] In another aspect, a device comprises a scaling module that scales each coefficient in an 8x8 matrix of coefficients encoded by one of a factor A, a factor B, a factor C, a factor D, a factor E , a F factor, a G factor, a H factor, a I factor, or a J factor to produce a staggered coefficient matrix, where A = 1024, 2? = 1138, C = 1730, £) = 1609,

E = 1264, F = 1922, G = 1788, H = 2923, I = 2718, and J = 2528. The device also comprises an inverse transform module that uses repeated applications of a fixed point scaled one-dimensional transform to transform the matrix of scaled coefficients into a matrix of transformed coefficients. Furthermore, the device comprises a displacement module on the right which displaces the transformed coefficients in the transformed coefficient matrix to the right in order to produce an adjusted coefficient matrix. Each adjusted coefficient in the adjusted coefficient matrix approximates a corresponding value in an array of values that would be produced when applying an ideal two-dimensional IDCT to the coded coefficient matrix. The device also comprises an output module that produces an 8x8 pixel block. Each pixel in the pixel block includes a pixel component value based on an adjusted coefficient in the adjusted coefficient matrix.

[0013] In another aspect, a device comprises mechanisms to scale coefficients in an 8x8 matrix of coefficients encoded by one among a factor A, a factor B, a factor C, a factor D, a factor E, a factor F, a factor G, an H factor, a factor I, or a factor J to produce a matrix of scaled coefficients, where, 4 = 1024, 5 = 1138, C = 1730, £) = 1609, 5 = 1264,

Petition 870180133578, of 9/24/2018, p. 13/169

£ 7/69 £ 1922, (7 = 1788, Η = 2923, I = 271S, and J = 2528. In addition, the device comprises mechanisms for using repeated applications of a one-dimensional staggered fixed-point transform to transform the coefficient matrix staggered in a transformed coefficient matrix. Furthermore, the device comprises mechanisms to shift the transformed coefficients in the transformed coefficient matrix to the right to produce an adjusted coefficient matrix. Each adjusted coefficient in the adjusted coefficient matrix approximates a corresponding value in an array of values that would be produced by applying an ideal two-dimensional IDCT to the array of encoded coefficients. In addition, the device comprises mechanisms for outputting an 8x8 pixel block. Each pixel in the pixel block includes a pixel component value based on in an adjusted coefficient in the adjusted coefficient matrix.

[0014] In another aspect, the invention is directed to a computer-readable medium containing instructions. The instructions cause a programmable processor to scale each coefficient into an 8x8 matrix of coefficients encoded by one of a factor A, a factor B, a factor C, a factor D, an factor E, a factor F, a factor G, a factor H, factor I, or factor J in order to produce a staggered coefficient matrix, where A = 1024, = 1138, C = 1730, £) = 1609, H = 2923, I = 2718, and J = 2528

E = 1264, F = 1922, G = 1788, The instructions also cause the programmable processor to use repeated applications of a one-dimensional staggered fixed-point transform to transform the staggered coefficient matrix into a transformed coefficient matrix. In addition, the instructions make the processor

Petition 870180133578, of 9/24/2018, p. 14/169

Programmable 8/69 shift the transformed coefficients in the transformed coefficient matrix to the right in order to produce an adjusted coefficient matrix. Each adjusted coefficient in the adjusted coefficient matrix approximates a corresponding value in an array of values that would be produced when applying an ideal two-dimensional IDCT to the coded coefficient matrix. The instructions also cause the programmable processor to emit signals that cause a display unit to display an 8x8 pixel block. Each pixel in the pixel block includes a pixel component value based on an adjusted coefficient in the adjusted coefficient matrix.

[0015] In some cases, the computer-readable medium may form part of a computer program product, which can be sold to manufacturers and / or used in a video encoding device. The computer program product may include the computer-readable medium, and in some cases, may also include packaging materials.

[0016] Details of one or more examples are presented in the attached drawings and in the description below. Other features, objects and advantages of the invention will be evident from the description and drawings, and from the claims.

Brief Description of the Figures

[0017] Figure 1 is one diagram of blocks illustrating a device example that encodes and decodes files from media. [0018] Figure 2 is one diagram of blocks illustrating Details copies in one module of coding. [0019] Figure 3 is one diagram of blocks illustrating exemplary details on one module of

decoding.

Petition 870180133578, of 9/24/2018, p. 15/169

9/69 [0020] Figure 4 is a flow chart illustrating an exemplary operation of the coding module.

[0021] Figure 5 is a flow chart illustrating an exemplary operation of the decoding module.

[0022] Figure 6 is a block diagram illustrating exemplary details of an inverse discrete cosine transform (IDCT) module.

[0023] Figure 7 is a flow chart illustrating an exemplary operation of the reverse transform module.

[0024] Figure 8 is a block diagram illustrating exemplary details of a direct discrete cosine transform (FDCT) module.

[0025] Figure 9 is a flow chart illustrating an exemplary operation of the vector direct transform module.

[0026] Figure 10A is a flow chart illustrating an exemplary one-dimensional transform.

[0027] Figure 10B is a flow chart illustrating an exemplary scaled one-dimensional transform.

[0028] Figure 11 is a flowchart illustrating a scaled, exemplary one-dimensional transform used by the inverse transform module.

Detailed Description of the Invention [0029] Figure 1 is a block diagram illustrating an exemplary device 2 that encodes and decodes media files. Device 2 comprises a personal computer, a telephone via mobile radio, a server, a network device, a computer device integrated into a vehicle, a gaming platform, a portable video game device, a computer workstation, a computer kiosk, digital signage, a central computer, a television set-top box, a network telephone,

Petition 870180133578, of 9/24/2018, p. 16/169

10/69 a personal digital assistant, a video game platform, a mobile media player, a home media player, a digital video projector, a personal media player (for example, an iPod), or other type of electronic device .

[0030] The device 2 can include a media source 4 to generate media data. Media source 4 can comprise a digital video or still camera to capture image data. The media source 4 can be integrated into device 2 or can be attached to device 2 as a peripheral device. Media source 4 may also comprise a microphone for recording audio data. Media source 4 can provide media data to a processor 6. Processor 6 may comprise a digital signal processor (DSP), a microprocessor, or some other type of integrated circuit.

[0031] When processor 6 receives media data from media source 4, an encoding module 8 can encode the media data. The encoding module 8 may comprise software executed by processor 6. Alternatively, the encoding module 8 may comprise specialized hardware within processor 6 that encodes the media data. In yet another alternative, the encoding module 8 can comprise any combination of software and hardware to encode the media data.

[0032] The encoding module 8 can store the encoded media data in a media repository 10. 0 media repository 10 can comprise flash memory, random access memory, a hard disk drive, or some other type of drive volatile or non-volatile data storage.

Petition 870180133578, of 9/24/2018, p. 17/169

11/69 [0033] A decoding module 12 can retrieve encrypted media data from the media repository 10. The decoding module 12 can comprise software run by processor 6. Alternatively, the decoding module 12 can comprise specialized hardware within of processor 6 that decodes the encoded media data. In yet another alternative, the decoding module 12 may comprise a combination of software and hardware that collaborate to decode the encoded media data.

[0034] A media presentation unit 14 on device 2 can display media data decoded by decoding module 12. For example, media presentation unit 14 can comprise a computer monitor that displays image or video media data . In another example, the media presentation unit 14 may comprise an audio output device (e.g., a speaker) that presents audio media data. The media presentation unit 14 can be integrated into device 2 or can be connected via a wired or wireless link to device 2 as a peripheral device.

[0035] The device 2 can also comprise a network interface 16. The network interface 16 can facilitate communication between the device 2 and a computer network via a wired or wireless link. For example, network interface 16 can facilitate communication between device 2 and a mobile phone network. Device 2 can receive media files via the network interface 16. For example, device 2 can receive photographs, video clips, streaming video (for example, television, video conferencing, movies), video clips. audio (for example, ringing tones, music, files

Petition 870180133578, of 9/24/2018, p. 18/169

12/69

ΜΡ3), streaming audio (for example, digital radio stations, voice calls, etc.) through network interface 16. When network interface 16 receives a media file or video bit stream, the network interface 16 can store the media file or video bit stream in the media repository 10.

[0036] A video signal can be described in terms of a sequence of images, which includes frames (an entire image), or fields (for example, an image comprising odd or even lines in a frame). In addition, each frame or field can include two or more slices, or subparts of the frame or field. As used herein, whether alone or in combination with other words, the term frame can refer to an image, a frame, a field or a slice of it.

[0037] When encoding module 8 encodes a series of video frames, encoding module 8 can start by selecting one of the video frames to be i-frame. For example, coding module 8 can select each eighth frame as an i-frame. I-frames are frames that do not reference other frames. After selecting the i-frames, the coding module 8 uses spatial coding to code the i-frames. Furthermore, the encoding module 8 can use time coding to encode the remaining frames.

[0038] To use spatial encoding to encode a frame, encoding module 8 can split the frame data into blocks of pixels. For example, encoding module 8 can break frame data into pixel blocks that are eight pixels wide and 8 pixels high (that is, each pixel block contains 64 pixels). The coding module 8 can then

Petition 870180133578, of 9/24/2018, p. 19/169

13/69 separate pixel component values from pixels in each pixel block in separate arrays of pixel component values. The pixel component values of a pixel are the values that characterize the appearance of the pixel. For example, each pixel can specify a pixel component value Y, a pixel component value Cr, and a pixel component value Cb. The pixel component value Y indicates the pixel luminance, the pixel component value Cr indicates the red chrominance of the pixel, and the pixel component value Cb indicates the blue chrominance of the pixel. In this example, when the encoding module 8 separates the pixel component values from a pixel block, the encoding module 8 can obtain an array of Y pixel component values, an array of pixel component values Cr, and an array of Cb pixel component values.

[0039] After separating the pixel component values in arrays of pixel component values, the coding module 8 generates arrays of coefficient values of left pixel components in the arrays of pixel component values. For each matrix of adjusted coefficients, the coding module 8 uses fixed point arithmetic to repeatedly apply a one-dimensional transform to the matrix of adjusted coefficients, thereby generating matrices of transformed coefficients. Then, the coding module 8 generates a matrix of scaled coefficients by scaling the matrix of coefficients transformed by a set of scaling factors. Each of these escalation factors is an integer value. The scaling factors were selected in such a way that factors within the one-dimensional transform can be approximated adjusted by displacement to

Petition 870180133578, of 9/24/2018, p. 20/169

14/69 using simple rational numbers.

[0040] Each scaled coefficient in the scaled coefficient matrix approximates a corresponding value in an array of values that would be produced by applying an ideal two-dimensional direct discrete cosine transform (FDCT) to a corresponding array of color component values. An ideal one-dimensional FDCT is defined by:

= c (k) Ys (n) cos ^ + ¹ ^) n = 0 where s is the array of N original values, t is the array of N transformed values, and the coefficients c are given by c (o) = yfl / N, c (k) = y2 / N for 1 <k <N - 1.

An ideal two-dimensional FDCT is defined by the formula:

zz. \ χ ~ 7 χ ^- * ζ \ τγ (2τ7ζ + l) z 7t (2zz + 1) / t (i, j) = c (i, j) 2 ^ 2, ^s V ⁿ > ⁿ ) ^cos - cos ΛΓ-1 ΛΓ-1 n = lm = 0

2N

2N where s is the array of N original values, t is the array of N transformed values, and c (i, j) is given by c (i, j) = c (zjc (yj, and with c (k) defined as in the one-dimensional case.

[0041] After deriving a matrix of scaled coefficients, the coding module 8 generates a matrix of quantized coefficients when quantizing the coefficients in the matrix of scaled coefficients. Quantizing the scaled coefficients can reduce the amount of information associated with the high frequency coefficients in the scaled coefficient matrix. After generating the matrix of quantized coefficients, the coding module 8 can apply an entropy coding scheme to the matrix of quantized coefficients. For example, the

Petition 870180133578, of 9/24/2018, p. 21/169

15/69 coding 8 can apply a Huffman coding scheme to the quantized coefficients in the coefficient matrix. When the coding module 8 applies the entropy coding scheme to each of the matrices of quantized coefficients, the coding module 8 can output the coded matrices as part of a video bit stream.

[0042] To use time coding to encode a frame, encoding module 8 can divide the frame into macroblocks. Depending on the coding pattern used, these macroblocks can be of fixed or variable size and can be overlapping or non-overlapping. For example, each macroblock can be a 16x16 pixel block. For each macroblock in the frame, coding module 8 may attempt to identify a macroblock of origin in one or more reference frames. Depending on the coding standard, reference frames can be i-frames, predictive frames, or bi-predictive frames. If the coding module 8 is able to identify a source macroblock in a frame of reference, the coding module 8 records a motion vector for the macroblock. The motion vector includes a value of x that indicates the horizontal displacement of the macroblock relative to the identified macroblock of origin and a value of y that indicates the vertical displacement of the macroblock relative to the identified macroblock of origin. If coding module 8 is unable to identify a source macroblock for the macroblock, coding module 8 may not be required to record a motion vector for the macroblock. Then, the coding module 8 generates a reconstructed frame. The reconstructed frame contains the frame that would result from the movement of the macroblocks of the reference frames according to the motion vectors

Petition 870180133578, of 9/24/2018, p. 22/169

16/69 recorded for the current frame. After generating the reconstructed frame, coding module 8 subtracts pixel component values in each pixel of the reconstructed frame from the corresponding pixel component values in corresponding pixels of the current frame, resulting in a residual frame. The coding module 8 can then use an entropy coding scheme to compress the motion vectors for the macroblocks of the current frame. In addition, the coding module 8 uses the spatial coding technique described above to compress the residual frame.

[0043]

The decoding module 12 can perform a similar process as the encoding module

8, but in reverse spatial process

For example, in order to perform a decoding step, the decoding module 12 can apply an entropy decoding scheme for each coded matrix of quantized coefficients in an encoded video bit stream. The decoding module 12 can then quantize coefficients in each matrix of quantized coefficients, thereby generating a matrix of quantized coefficients for each matrix of quantized coefficients. For each matrix of quantized coefficients, the

decoding 12 generates a matrix in coefficients staggered when staggering the matrix in coefficients quantized using a set in factors in staggering . These factors staggering can be the same scheduling factors used at the process in

spatial coding discussed above. After generating a matrix of scaled coefficients, the decoding module 12 uses fixed point arithmetic to repeatedly apply a one-dimensional transform to the matrix of quantized coefficients, thereby generating a matrix of coefficients

Petition 870180133578, of 9/24/2018, p. 23/169

17/69 apply column to

in the transformed matrix of transformed. For example, the decoding module 12 can generate a matrix of intermediate coefficients by applying the one-dimensional transformation to each line vector in a matrix of scaled coefficients. In this example, the decoding module 12 can then generate the matrix of coefficients transformed to one-dimensional at each intermediate coefficient vector. After generating a matrix of transformed coefficients, the decoding module 12 generates a matrix of adjusted coefficients by shifting the transformed coefficients in the transformed coefficient matrix on the right.

[0044] Adjusted coefficients in the adjusted coefficient matrix approximate the values that would be produced when applying an ideal two-dimensional inverse discrete cosine transform (IDCT) to the matrix of quantized coefficients. The ideal one-dimensional IDCT is defined by the formula:

ΛΓ-1 í (£) = c (£) Zs (zz) cos π {2η +1) £) n = 0

2N where s is the array of N original values, t is the array of N transformed values, and the coefficients c are given by c (o) = JÜN, c (k) = 4ΪΪΝ for \ <k <N - \. An ideal two-dimensional IDCT is defined by the formula:

π (2ηι + l) z tt (2zz +1) 7

ΛΓ-1ΛΓ-1 s (zw, zz) = 7 W ^z '' 7 ') cos í = 0 j = 0 cos2N

These blocks of pixel component values can then be reassembled into blocks of pixels and these blocks of pixels can be reassembled to form the uncompressed video frame.

Petition 870180133578, of 9/24/2018, p. 24/169

18/69 [0045] After generating the adjusted coefficient matrix, the decoding module 12 can then cut the adjusted coefficients in the adjusted coefficient matrix to ensure that the adjusted coefficients are within the allowed range for a pixel component value . The decoding module 12 can then reassemble the arrays of clipped coefficients into blocks of pixels. After reassembling the blocks of pixel component values into pixel blocks, the decoding module 12 can generate an image by reassembling the pixel blocks.

[0046] In order to decode a predictive frame, the decoding module 12 can use the spatial decoding technique described above to decode the matrices of quantized coefficients in the residual image for the predictive frame. In addition, the decoding module 12 can use the entropy decoding scheme to decode the motion vectors of the predictive frame. Then, the decoding module 12 can generate a reconstructed frame by moving macroblocks from the reference frames of the predictive frame according to the motion vectors. After generating the reconstructed frame, the decoding module 12 adds pixel component values in each pixel of the decoded residual frame to the corresponding pixel component values in corresponding pixels of the reconstructed frame. The result of this addition is the reconstructed predictive framework.

[0047] The techniques described in this invention can provide several advantages. For example, because these techniques apply fixed-point arithmetic, these techniques can be applied to smaller, less complex devices, such as mobile phones, personal digital assistants, and personal media players. Besides that,

Petition 870180133578, of 9/24/2018, p. 25/169

19/69 these techniques can be applied to formats that include the recommendations H.261, H.263, H.264, T.81 (JPEG) of the International Telecommunication Union Standardization Sector (ITU-T), as well as the media formats Moving Image Experts Group (MPEG) -l, MPEG-2, and MPEG-4 part 2 MEC / International Organization for Standardization (ISO).

[0048] Figure 2 is a block diagram illustrating exemplary details of the coding module 8. The coding module 8 can comprise a set of modules. These modules may comprise subsets of software instructions from the coding module 8. Alternatively, these modules may comprise hardware and firmware such as one or more special purpose hardware. In another alternative, these modules may comprise special-purpose software and hardware instructions.

[0049] As illustrated in the example in Figure 2, the encoding module 8 includes a frame control module 20 that controls whether the encoding module 8 processes a video frame as an i-frame, a predictive frame, or a frame bi-predictive. For example, when encoding module 8 receives a video frame, frame control module 20 can determine whether a bit rate indicator associated with the video frame indicates that the frame is an i-frame, a predictive frame , or a bipredictive picture. If the frame control module 20 determines that the bitstream indicator indicates that the frame is an i-frame, the frame control module 20 can cause the video frame to be processed by a set of modules that perform spatial encoding in the video frame. On the other hand, if the frame control module 20 determines that the frame is a predictive frame

Petition 870180133578, of 9/24/2018, p. 26/169

20/69 or a bi-predictive frame, the frame control module 20 can cause the video frame to be processed by a set of modules that perform time coding.

[0050] The encoding module 8 includes a series of modules for applying spatial encoding to video frames. These modules include a block divider module 22, a component extraction module 24, a direct transform module 26, a quantization module 28, and an entropy coding module 30. The block divider module 22 can receive frames of non-tagged video from media source 4, network interface 16, or another source. When the block divider module 22 receives a non-encoded video frame, the block divider module 22 can separate the frame into blocks of pixels. The block divider module 22 can provide pixel blocks to a component extraction module 24.

[0051] When the component extraction module 24 receives a block of pixels, the component extraction module 24 can convert pixel component values of each pixel into a different color format. For example, the component extraction module 24 can convert each pixel from a Red-Green-Blue (RGB) color format to the YCrCb color format. After converting the pixels in the block to a different color format, the component extraction module 24 can separate the pixel component values from the pixels in the block into arrays of pixel component values. For example, the component extraction module 24 can extract an array of Y values, an array of Cr values, and an array of Cb values from a pixel block. The Y values specify the pixel brightness, the Cr values specify the red chrominance of the pixels, and the Cb values specify the blue chrominance of the pixels.

Petition 870180133578, of 9/24/2018, p. 27/169

21/69

When the component extraction module 24 extracted the matrix of pixel component values, the component extraction module 24 can provide each matrix separately to a Discrete Cosine Transform (DCT) module 26.

[0052] When the direct transform module 26 receives an array of pixel component values, the direct transform module 26 generates an array of scaled coefficients. Each coefficient in this matrix of scaled coefficients approximates a coefficient that is produced using an ideal direct discrete cosine transform to transform the matrix of pixel component values.

[0053] The direct transform module 26 uses fixed point arithmetic to apply a one-dimensional transform to the arrays of pixel component values. Using fixed-point arithmetic can be advantageous in some circumstances. For example, smaller devices, such as mobile phones, could not include a floating point unit required to perform floating point arithmetic. The direct transform module 26 can begin a process of generating the staggered coefficient matrix by shifting each of the pixel component values to the left. For example, the direct transform module 26 can generate an array of adjusted coefficients by shifting each of the pixel component values to the left by a number of precision bits (that is, the number of mantissa bits) of representations of fixed point of numbers that the direct transform module 26 uses when applying the one-dimensional transform plus a number of precision bits removed when scaling the transformed coefficients that result from applying the transform. After moving to the

Petition 870180133578, of 9/24/2018, p. 28/169

22/69 left of each of the pixel component values, the direct transform module 26 can perform the transform in each of the line vectors of the adjusted coefficient matrix. When performing a discrete cosine transform on each of the line vectors of the adjusted coefficient matrix, it generates an intermediate coefficient matrix. Then, the direct transform module 26 can perform the transform in each of the column vectors of the intermediate coefficient matrix. When performing the transform in each of the column vectors of the intermediate coefficient matrix, it results in a matrix of transformed coefficients.

[0054] After generating the transformed coefficient matrix, the direct transform module 26 scales transformed coefficients at different positions in the coefficient matrix transformed by different scale factors. As described below, the decoding module 12 can use the reciprocal of these scale factors in the application of an inverse transform. When the direct transform module 26 has finished scaling the coefficients transformed by the scale factors, the direct transform module 26 can emit the matrix resulting from scaled coefficients for the quantization module 28.

[0055] When the quantization module 28 receives a coefficient matrix from the direct transform module 26, the quantization module 28 can quantize the scaled coefficients. The quantization module 28 can quantize the scaled coefficients in a variety of ways depending on the coding pattern being employed. For example, according to the MPEG-4 part 2 standard, the quantization module 28 can use the following quantization matrix to quantize the coefficients in a

Petition 870180133578, of 9/24/2018, p. 29/169

23/69 matrix of scaled coefficients for an i-frame:

8 17 18 19 21 23 25 27 17 18 19 21 23 25 27 28 20 21 22 23 24 26 28 30 21 22 23 24 26 28 30 32 22 23 24 26 28 30 32 35 23 24 26 28 30 32 35 38 25 26 28 30 32 35 38 41 27 28 30 32 35 38 41 45

In addition, in this example, the quantization module 28 can use the following quantization matrix to quantize coefficients in a scaled coefficient matrix for a predictive or bi-predictive framework:

16 17 18 19 20 21 22 23 17 18 19 20 21 22 23 24 18 19 20 21 22 23 24 25 19 20 21 22 23 24 26 27 20 21 22 23 25 26 27 28 21 22 23 24 26 27 28 30 22 23 24 26 27 28 30 31 23 24 25 27 28 30 31 33

[0056] After the quantization module 28 generates a matrix of quantized coefficients, the entropy coding module 30 can compress the matrix of quantized coefficients using an entropy coding scheme. To compress the matrix of quantized coefficients using an entropy coding scheme, the entropy coding module 30 can organize the quantized coefficients into a vector by obtaining a zigzag pattern of the coefficients. That is, the

Petition 870180133578, of 9/24/2018, p. 30/169

24/69 entropy coding 30 can arrange all quantized coefficients in the two-dimensional matrix of quantized coefficients in a one-dimensional vector of quantized coefficients in a predictable one. The entropy coding module 30 can then apply an entropy coding scheme, such as Huffman coding or arithmetic coding, to the quantized coefficient vector.

[0057] The encoding module 8 also includes one or more modules to perform the temporal encoding of the video frames. As illustrated in the example in Figure 2, the coding module 8 includes a motion estimation module 32, a reconstructed frame generation module 34, and a residual frame generation module 36. Motion estimation module 32 attempts to identify a source macroblock in a reference image for each macroblock in a current video frame. The motion estimation module 32 can try to identify a source macroblock for a macroblock in the current frame by looking for macroblocks in the reference image that contain similar pixels as the macroblock. The motion estimation module 32 can search for areas of different sizes according to different coding standards in order to identify the source macroblock for a macroblock in the current frame. For example, the motion estimation module 32 can search for a source macroblock within an area that is 32 pixels wide by 32 pixels high, with the current macroblock in the center of the search area. When motion estimation module 32 identifies a source macroblock for a macroblock in the current frame, motion estimation module 32 calculates a motion vector for the macroblock in the current frame. The motion vector for the macroblock in the current frame

Petition 870180133578, of 9/24/2018, p. 31/169

25/69 specifies an x value that indicates the difference in the horizontal position between the identified macroblock of origin and the macroblock of the current frame. After motion estimation module 32 has calculated a motion vector or has been unable to identify a source macroblock for each macroblock in the current frame, motion estimation module 32 can provide the calculated motion vectors for the current frame for the reconstructed board generation module 34.

[0058] The reconstructed frame generation module 34 can use motion vectors and reference frames to generate a reconstructed frame. The reconstructed frame generation module 34 can generate the reconstructed frame by applying the motion vectors for each macroblock in the current frame to the original macroblocks in the reference frames. In fact, the reconstructed frame generation module 34 creates a frame in which the macroblocks of the reference frames have been moved to the positions indicated by the corresponding motion vectors of the current frame.

[0059] Residual frame generation module 36 can generate the residual frame by subtracting pixel component values in the reconstructed frame from the corresponding pixel component values in the current frame. The residual picture generally includes less information than the reconstructed picture or the current picture. After the residual frame generation module 36 generates the residual frame, the residual frame generation module 36 provides the residual frame to the block divider module 22 in order to begin the spatial encoding process of the residual frame. In addition, the motion estimation module 32 can provide the motion vectors for the current frame to the entropy coding module 30 in order to compress the motion vectors.

Petition 870180133578, of 9/24/2018, p. 32/169

26/69 movement. After the residual frame is spatially encoded and the entropy encoding module 30 has encoded the motion vectors, the flow output module 38 can format the spatially encoded residual frame and the encoded motion vectors as part of a video bit stream. .

[0060] Figure 3 is a block diagram illustrating exemplary details of the decoding module 12. The decoding module 12 may comprise an entropy decoding module 44, a quantizing module 46, an inverse transform module 48, a module pixel reconstruction module 50, a frame buffer 51, a block combiner module 52, a frame control module 53, a reference frame storage module 54, a reconstructed frame generation module 56, and a generation of predictive board 58. Some or all of these modules may comprise subsets of the software instructions of the decoding module 12. Alternatively, some or all of these modules may comprise special-purpose hardware or firmware. Alternatively, these modules may comprise special-purpose software and hardware instructions or firmware.

[0061] When the decoding module 12 receives a bit stream containing a video frame, the entropy decoding module 44 can apply an entropy decoding scheme to the matrices of quantized coefficients in the video frame. The bit stream can include a value that tells entropy decoding module 44 which entropy decoding scheme to apply to the matrices of quantized coefficients in the bit stream. In addition, entropy decoding module 44 can apply the same or a different scheme

Petition 870180133578, of 9/24/2018, p. 33/169

27/69 entropy decoding to decode the motion vectors of the video frame.

[0062] After the entropy decoding module 44 applies the entropy decoding scheme to the matrices of quantized coefficients in the video file, a quantization module 46 can quantize the coefficients in each of the matrices of quantized coefficients. Depending on the coding standard, the quantization module 46 can quantify the coefficients in a variety of ways. For example, according to the MPEG-4 part 2 standard, the quantization module 466 can use the two quantization matrices listed above in two different ways. First, the quantization module 46 can use these quantization matrices to perform H.263-style quantization. In H.263 style dequantization, the dequantization module 46 obtains reconstructed coefficients F [v] [u] from the quantized values QF [v] [u] as follows:

| F [v] [«] | • (2 x | QF [v] [m] | +1) x scale_ quantized, if QF [v] [«] 0, scale_ quantized is odd (2 x | QF [v] [m] | +1) x scalar_ quantized -1, if QF [v] [«] 0, scaled_ quantized is even

F [v] [u]: F [v] [u] = Sign (QF [v] [u]) x | F [v] [u] | , which only involves a multiplication by the scale_quantizer, and

Second, the quantization module 46 can use these quantization matrices to perform MPEG-1/2 style quantization. In MPEG-1/2 style dequantization, the dequantization module uses additional weighting matrices W [w] [v] [u] where w indicates which weighting matrix is being used:

0, if QF [v] [u] = 0 ((2xQF [v] [u] + k) xW [w] [v] [u] xescala_quantizad0) / 16, if QF [v] [u] Φ 0 | F [v] M | =

Petition 870180133578, of 9/24/2018, p. 34/169

28/69 where:

OJ intra blocks [5z'na (0F [v] [M]) non-intra blocks [0063] As described in detail below, the inverse transform module 48 scales each of the coefficients quantified by specific scaling factors and adds a term from midpoint polarization to the DC coefficient of the matrix resulting from staggered coefficients. The DC coefficient is the coefficient at position (0.0) of the staggered coefficient matrix. Next, the inverse transform module 48 uses repeated applications of a one-dimensional fixed-point transform to transform the staggered coefficient matrix into a transformed coefficient matrix. After transforming the staggered coefficient matrix into the transformed coefficient matrix, the inverse transform module 48 shifts each transformed coefficient to the right into the transformed coefficient matrix in order to produce an adjusted coefficient matrix. Each adjusted coefficient in the adjusted coefficient matrix approximates a corresponding value in an array of values that would be produced when applying an ideal two-dimensional IDCT to the coded coefficient matrix. In addition, the inverse transform module 48 can cut each of the adjusted coefficients in the adjusted coefficient matrix in order to produce a matrix of cut coefficients. The coefficients of cut have values comprised in ranges appropriate for the resulting pixel component value format. For example, the coefficients of cut can have values that are included in the range [-256, 255]. After the reverse transform module 48 cuts the coefficients

Petition 870180133578, of 9/24/2018, p. 35/169

29/69 adjusted, the resulting cut-off values are pixel component values.

[0064] After the inverse transform module 48 emits an array of pixel component values, the pixel reconstruction module 50 can generate an array of pixels by combining the array of pixel component values with the array of component values pixel values associated with equivalent positions within a video frame. For example, the pixel reconstruction module 50 may receive an array of pixel component values Y, an array of pixel component values Cb, and an array of pixel component values Cr from the reverse transform module 48. Each of these three arrays can include pixel components for a single 8x8 pixel block. Each of the pixels can include a pixel component value Y, a pixel component value Cb, and a pixel component value Cr. After generating the pixel array, the pixel reconstruction module 50 can provide the pixel block to the block combiner module 52.

[0065] When the block combiner module 52 receives a pixel block, the block combiner module 52 can store the pixel block until the block combiner module 52 receives some or all of the pixel blocks in a video frame. After receiving one or more of the pixel blocks, the block combiner module 52 can combine the pixel blocks in a video frame and can output the video frame to frame buffer 51. The video frame can be stored in the frame 51 until it is displayed by the media display unit 14. In addition, the block combiner module 52 can output the video frame to a frame storage module 54. The video frames in the frame storage module 54

Petition 870180133578, of 9/24/2018, p. 36/169

30/69 can be used as frames of reference for the reconstruction of predictive and bi-predictive frames. In addition, the video frames in the frame storage module 54 can be residual frames that are used in the reconstruction of predictive and bi-predictive frames.

[0066] In order to reconstruct a predictive or bi-predictive frame, the decoding module 12 includes the reconstructed frame generation module 56. The reconstructed frame generation module 56 receives the decoded motion vectors from the entropy decoding module 44. In addition, the reconstructed frame generation module 56 retrieves the reference frames from the current frame of the frame storage module 54. The reconstructed frame generation module 56 then moves the macroblocks from their positions in the reference frames to the positions indicated by the motion vectors. A reconstructed picture results from the movement of the macroblocks in this way. After the reconstructed board generation module 56 generates the reconstructed board, the reconstructed board generation module 56 provides the reconstructed board to the predictive board generation module 58.

[0067] When the predictive board generation module 58 receives a temporary frame, the predictive board generation module 58 can retrieve a residual frame for the current frame from the frame storage module 54. After recovering the residual frame, the predictive frame generation module 58 can add corresponding color component values to each pixel of the residual frame and the reconstructed frame. A reconstructed video frame results from this addition. Then, the predictive frame generation module 58 can send this reconstructed frame to frame buffer 51 for eventual display on the media presentation unit 14.

Petition 870180133578, of 9/24/2018, p. 37/169

31/69 [0068] Figure 4 is a flowchart illustrating an exemplary operation of the coding module 8. Although the operation described in Figure 4 is described in sequential form, it should be noted that the operation can be performed in a chained manner.

[0069] Initially, the encoding module 8 receives a sequence of non-encoded video frames 60. For example, the encoding module 8 can receive a sequence of non-encoded frames in the form of pixel sets from the media source 4. When the encoding module 8 receives the sequence of non-encrypted frames, the frame control module 20 in the encoding module 8 can determine whether a current frame in the sequence of unencrypted frames should be encoded as an i-frame or as a predictive or bipredictive framework 62.

[0070] If frame control module 20 determines that the current frame should be encoded as an i-frame, block divider module 22 in encoding module 8 can divide the current frame into 64 pixel blocks. For example, the coding module 8 can divide the current frame into 2x2, 4x4, or 8x8 pixel blocks.

[0071] After dividing the current frame into pixel blocks, the component extraction module 24 can separate the pixel component values in each of the pixel blocks 66. As a result, there may be three blocks of pixel component values for each pixel block: a block of Y values to represent the brightness of the pixels, a block of Cb values to represent the blue chrominance of the pixels, and a block of Cr values to represent the red chrominance of the pixels.

Petition 870180133578, of 9/24/2018, p. 38/169

32/69 [0072] The direct transform module 26 in the coding module 8 can then generate a matrix of scaled coefficients for each of the matrix of pixel component values 68. The coefficients in these scaled coefficient matrices are approximations of values that would be produced by using an ideal two-dimensional direct discrete cosine transform in the respective matrices of the pixel component value matrices.

[0073] After the direct transform module 26 generates the staggered coefficient matrices for each of the pixel component matrices, the quantization module 28 in the coding module 8 can quantize the coefficients in each of the staggered coefficient matrices 70. Since the quantization module 28 has quantized the coefficients in each staggered coefficient matrix, the entropy coding module 30 can perform an entropy coding process in each of the quantized coefficient matrices 72. For example, the coding 8 can apply a Huffman coding scheme or an arithmetic coding scheme to each matrix of the quantized coefficients. The entropy coding process also compressed the data. However, entropy coding processes do not lead to loss of information. After carrying out the entropy encoding process on each matrix of quantized coefficients, the flow output module 38 in the encoding module 8 can add the encoded matrices of quantized coefficients to a bit stream for the video frame sequence 74. After the stream output module 38 adds the encoded arrays to the bit stream, the frame control module 20 can determine whether the current frame was the last video frame in the frame sequence 76. if the current frame is the last frame of the

Petition 870180133578, of 9/24/2018, p. 39/169

33/69 frame sequence (YES 76), encoding module 8 has finished encoding frame sequence 78. On the other hand, if the current frame is not the last frame in the frame sequence (NO 76), the coding module 8 can perform cycle feedback and determine whether a new current frame should be encoded as an i-frame 62.

[0074] If the current frame is not to be coded as an i-frame (NOT 62), motion estimation module 32 in coding module 8 can divide the current frame into a set of macroblocks 80. Then, the motion estimation module 32 can attempt to identify a source macroblock in one or more reference frames for each of the macroblocks in the current frame 82. motion estimation module 32 can then calculate a motion vector for each of the macroblocks in the current frame so that motion estimation module 32 would be able to identify a macroblock of origin 84. After motion estimation module 32 calculates motion vectors, the reconstructed frame generation module 34 uses motion vectors to generate a reconstructed frame by moving the macroblocks identified in the reference frames in the positions indicated by the motion vectors 86. The residual frame generation module 36 can then generate a residual frame for to the current frame by subtracting the pixel component values in the reconstructed frame from corresponding pixel component values in the current frame 88. After the residual frame generation module 36 generates the residual frame, the entropy coding module 30 can use an entropy coding scheme to encode the motion vectors for the current frame 90. In addition, spatial coding can be applied to the residual frame by applying steps 66 to 74 to the residual frame.

Petition 870180133578, of 9/24/2018, p. 40/169

34/69 [0075] Figure 5 is a flowchart illustrating an exemplary operation of the decoding module 12. Although the operation described in Figure 5 is described in sequential form, it should be noted that the operation can be performed in a chained manner.

[0076] Initially, the decoding module 12 receives an encoded video frame 100. After receiving the encoded video frame, the entropy decoding module 44 in the decoding module 12 can perform an entropy decoding process on data blocks within the encoded frame 102. The entropy decoding module 44 can perform an entropy decoding process that is equivalent to the entropy encoding process used to encode the frame. For example, if encoding module 8 uses Huffman encoding to encode the frame, entropy decoding module 44 uses Huffman decoding to decode the frame. As a result of the application of the entropy decoding process for each data block in the table, the entropy decoding module 44 produced a set of quantized coefficient matrices.

[0077] Next, the quantization module 46 in the decoding module 12 can quantize the coefficients in each of the matrices of quantized coefficients 104. After quantifying each coefficient in the matrices of quantized coefficients, the inverse transform module 48 in the decoding module 12 generates matrices of pixel component values 106. The pixel component values in a matrix of pixel component values are approximations of corresponding values that would be produced when transforming one of the matrices of the quantized coefficients using a two-dimensional inverse discrete cosine transform ideal.

Petition 870180133578, of 9/24/2018, p. 41/169

35/69 [0078] When the inverse transform module 48 computed an array of pixel component values for each of the coefficient arrays, the pixel reconstruction module 50 in decoding module 12 can combine appropriate arrays of component values pixel to create 108 pixel blocks. For example, decoding module 12 can combine a block of Y values with an associated block of Cr values and an associated block of Cb values to create a block of Y values. YCrCb pixels. After the pixel reconstruction module 50 creates the pixel blocks, the block combiner module 52 can recombine the pixel blocks in a 110 video frame.

[0079] Next, the frame control module 53 in the decoding module 12 can determine whether the current frame is an i-frame (112). If the current frame is an i-frame (SIM 112), the block combiner module 52 can send the video frame to frame buffer 51 in block 114. On the other hand, if the current frame is not a frame i (ie, the current frame is a predictive or bi-predictive frame) (NOT 112), entropy decoding module 44 uses an entropy decoding scheme to decode the motion vectors of the current frame 116. In The reconstructed frame generation module 56 then uses the decoded motion vectors and one or more reference frames in the frame storage module 54 to generate a reconstructed frame 118. The predictive frame generation module 58 can then use the frame reconstructed and the frame generated by the block combiner module 52 to generate a reconstructed frame 120.

[0080] Figure 6 is a block diagram illustrating exemplary details of the reverse transform module 48. As illustrated in the example in Figure 6, the module

Petition 870180133578, of 9/24/2018, p. 42/169

36/69 of reverse transform 48 may comprise an input module 14 0. The input module 140 may receive a coefficient matrix from the quantization module 46. For example, input module 140 may receive a pointer indicating a location in a device memory module 2 that stores the coefficient matrix. Alternatively, input module 140 may include internal data structures that store the coefficient matrix.

[0081] When the input module 140 receives a coefficient matrix, the input module 140 can provide the coefficient matrix to a scaling module 142 in the reverse transform module 48. The scaling module 142 can perform operations that require values that approximate the scaling coefficients in the matrix of coefficients by scaling factors in equivalent positions in a matrix of scaling factors. For example, the scaling module 142 can perform multiplication operations to obtain values that approximate the scaling coefficients in the coefficient matrix by scaling factors in equivalent positions in a scaling factor matrix. In another example, the scheduling module 142 can perform one or more addition and displacement operations. Each of these scaling factors can be an 8-bit integer value. When each of the scaling factors is an 8-bit integer, the dynamic range of values produced during the transform is reduced.

[0082] After the scaling module 142 has generated the scaled coefficient matrix, the coefficient polarization module 144 can have a polarized coefficient matrix by adding a midpoint bias value to the DC coefficient of the coefficient matrix

Petition 870180133578, of 9/24/2018, p. 43/169

37/69 staggered. As discussed above, the DC coefficient of the matrix is typically the coefficient in the upper left position of the matrix. Generally, the DC coefficient represents an average value of the other coefficients in the matrix.

[0083] In order to add an adaptive signal bias value to the DC coefficient on a 16-bit processor, the coefficient bias module 144 can use the following formula:

DC coefficient - DC + coefficient (l «(P + 2)).

In this formula, the term (l «(P + 2)) is added to provide the midpoint bias. P is a constant referring to the number of bits of the fixed point mantissa (that is, bits to the right of the base point) used in a transform applied by a 14 6 vector inverse transform module. The number 2 is added to P because the right shift module 148 can shift all coefficients by (P + 3) to the right, where the number '3' comes from the precision bits added when performing the transform. To elaborate this point, if a number x is generated by the displacement to the left of 1 by (P + ₂ ) and a number z is generated by displacement to the right of x by

(P + 3), then z = 1/2. (Contrary to the nominee, 2 ^{p + 2/2 p + 3} = 2 ° / 2 ¹ = 1/2). Like this, add (l «(P + 2)) to DC coefficient is equivalent to add (l «(P + 3)) / 2 to DC coefficient. [0084] After the module of polarization in

coefficient 144 generate the matrix of polarized coefficients, the vector inverse transform module 146 can generate a matrix of intermediate values by applying a scaled one-way fixed-point transform to each line vector of the matrix of polarized coefficients. THE

Petition 870180133578, of 9/24/2018, p. 44/169

38/69 Next, the vector inverse transform module 146 can compute a matrix of transformed coefficients by applying the fixed point scaled one-dimensional transform to each column vector of the matrix of intermediate values. An exemplary operation to apply the one-dimensional fixed point scaling transform to a scaled coefficient vector is shown in Figure 10B below.

[0085] After the vector inverse transform module 146 generates the matrix of transformed coefficients, the right shift module 148 can generate an adjusted coefficient matrix by shifting to the right of each of the coefficients in the coefficient matrix transformed by a number of positions equal to the number of bits added during application of the transform and during scaling. For example, if you apply the transform results to an additional 3 bits and scale the coefficients adds 10 additional bits, the right shift module 108 can shift each of the coefficients to the right by thirteen (3 + 10) positions.

[0086] After the right shift module 148 generates the matrix of adjusted coefficients, a cutting module 150 can generate a matrix of coefficients by cutting the coefficients in the matrix of adjusted coefficients in order to restrict the coefficients to a maximum range allowable pixel component values. For example, a typical pixel component value can range from -256 to 255. If the adjusted coefficient matrix included a coefficient equal to 270, the cutting module 150 would restrict that coefficient to the maximum allowable range by reducing the coefficient to 255. After the cutting module 150 finishes cutting the coefficients, these coefficients can represent pixel component values. When the

Petition 870180133578, of 9/24/2018, p. 45/169

39/69 cutting module 150 completes the cutting of the coefficients in the matrix, the cutting module 150 can provide the matrix of cutting coefficients to an output module 152.

[0087] When output module 152 receives an array of cut-off coefficients (which are now pixel component values), output module 152 can output the array of pixel component values to the pixel reconstruction module 50.

[0088] Figure 7 is a flowchart illustrating an exemplary operation of the reverse transform module 48. Initially, the input module 140 receives an array of coefficients (170). When input module 140 receives the coefficient matrix, the scaling module 142 can scale each value in the coefficient matrix (172). For example, the scaling module 142 can perform operations that multiply each coefficient in the coefficient matrix by values equivalently positioned in a matrix of scaling factors.

[0089] After scaling each coefficient in the coefficient matrix, the coefficient polarization module 144 can add a midpoint polarization value to the coefficient DC coefficient of the coefficient matrix (174). After the coefficient bias module 144 adds the bias value to the matrix's DC coefficient, the vector reverse transform module 146 can determine whether a line counter is less than a maximum line counter 176. Initially, the line counter can be set to zero. The maximum line counter can be a static value that is equal to the number of lines in the coefficient matrix. For example, if the coefficient matrix includes eight lines, the maximum line counter is equal to eight.

Petition 870180133578, of 9/24/2018, p. 46/169

40/69 [0090] If the line counter is smaller than the maximum line counter (YES 176), the vector inverse transform module 146 can compute a one-dimensional staggered fixed point transform into a line vector of the matrix of coefficients indicated by the line counter (178). When the vector inverse transform module 146 computes the transform into a line vector of the coefficient matrix, the vector inverse transform module 146 can replace the original coefficients in the line coefficient vector with a vector of intermediate coefficients. After the vector inverse transform module 146 computes the transform into a line vector of the coefficient matrix, the vector inverse transform module 146 can increment the line counter (180). The vector inverse transform module 146 can then return and again determine whether the line counter is less than the maximum line counter (176).

[0091] If the line counter is not less than (that is, greater than or equal to) the maximum line counter (NOT 176), the vector reverse transform module 146 can determine whether a line counter column is less than a maximum column counter (182). Initially, the column counter can be set to zero. The maximum column counter can be a static value that is equal to the number of columns in the coefficient matrix. For example, if the coefficient matrix includes eight columns, the maximum column counter is equal to eight.

[0092] If the column counter is less than the maximum column counter (SIM 182), the vector inverse transform module 146 can compute a one-dimensional transform in each column vector of the matrix of intermediate coefficients indicated by the counter

Petition 870180133578, of 9/24/2018, p. 47/169

41/69 column (184). When the inverse transform module 48 computes the transform into a column vector of intermediate coefficients, the inverse transform module 48 replaces the intermediate coefficients in the column vector with a vector of transformed coefficients.

[0093] After the vector inverse transform module 146 computes the transform into a column vector of the coefficient matrix, the vector inverse transform module 146 can increment the column counter (186). The vector inverse transform module 146 can then return again to determine whether the column counter is less than the maximum column counter (182).

[0094] If the column counter is not less than (that is, greater than or equal to) the maximum column counter (NOT 182), the right shift module 148 can shift each of the coefficients transformed in the matrix (188). When the right shift module 148 shifts a coefficient to the right, the right shift module 148 can shift the coefficient to the right by a number of positions. The result of the shift to the right of each of the second intermediate coefficients in the matrix is an array of adjusted values. After shifting module 148 to the right shifted each transformed coefficient to the right, cutting module 150 can cut the ones to ensure that they are within an appropriate range for pixel component values (190). For example, the cutting module 150 can cut the adjusted coefficients to ensure that the adjusted coefficients are within the range of -256 to 255. When the cutting module 150 finishes cutting the adjusted coefficients, the adjusted coefficients adjusted

Petition 870180133578, of 9/24/2018, p. 48/169

42/69 output module 152 can output the matrix resulting from pixel component values (192).

[0095] Figure 8 is a block diagram illustrating exemplary details of the direct transform module 26. As illustrated in the example in Figure 8, the direct transform module 26 comprises an input module 210 that receives an array of component values from pixel from the component extraction module 24. When the input module 210 receives an array of composite pixel values, the input module 210 can provide the array of pixel component values to a left shift module 212. The left shift module 212 can shift all pixel component values in the pixel component value array to the left by the number of mantissa bits used in values that a vector direct transform module 214 uses while performing the transform. direct minus the number of mantissa bits removed when performing the direct transform. For example, if ten bits of mantissa are used in values while performing the direct transform and three bits of mantissa are removed when performing the direct discrete cosine transform, the left shift module 212 can shift the pixel component values to the left in seven positions. In another embodiment, if three mantissa bits are used in values while performing the direct transform and three mantissa bits are removed when performing the direct transform, the left shift module 212 can shift the pixel component values to the left by zero position.

[0096] After the left shift module 212 shifts the pixel component values, the vector direct transform module 214 can apply

Petition 870180133578, of 9/24/2018, p. 49/169

43/69 a direct transform to each column vector in the matrix of pixel component values in order to produce an array of intermediate values. Then, the vector direct transform module 214 can apply the direct transform to each line vector in the matrix of intermediate values in order to produce a matrix of transformed coefficients. When the vector direct transform module 214 applies the direct transform to a vector, the vector direct transform module 214 can apply the direct transform described in Figure 11, below. Note that the transform described in Figure 11, below, is an inverse of the transform described in Figure 10A.

[0097] After the vector direct transform module 214 produces the transformed coefficient matrix, a scaling module 216 can apply scaling factors to each transformed coefficient in the transformed coefficient matrix. The scaling module 216 can apply reciprocal of the scaling factors used by the scaling module 142 in the reverse transform module 48. For example, if the scaling module 142 scales the coefficients by shifting the coefficients to the left in three positions, the module scaling 216 can scale the transformed values by shifting the transformed values to the right in three positions. In another example, if the scaling module 142 scales the coefficients by multiplying the coefficients by individual scaling factors in a matrix of scaling factors, the scaling module 216 can scale the transformed values by multiplying the transformed values by scaling factors in the scaling factor matrix and then shifting the resulting values to the right by a magnitude. In order to reduce rounding errors, the

Petition 870180133578, of 9/24/2018, p. 50/169

44/69 scaling module 216 can add a midpoint bias value to each of the transformed values after multiplying the transformed values by the scaling factors. For example, the scaling module 216 can use the following equation to generate an array of scaled coefficients:

F [v] [m] - (F '[v] [m] * S [ν] [w] + (1 «19) - ((F' [ν] [μ]> = θ)? 0: 1 )) »20 where v = 0..7,« = 0..7; where S [v] [u] is an entry in the matrix of scaling factors, F is the matrix of scaled coefficients, and F 'is the matrix of transformed coefficients.

[0098] After the scaling module 216 generates the scaled coefficient matrix, an output module 218 can output the coefficient matrix for the quantization module 28.

[0099] Figure 9 is a flow chart illustrating an exemplary operation of the direct transform module 26. Initially, the input module 210 receives an array of pixel component values (230). When input module 140 receives the array of pixel component values, the left shift module 212 can generate an array of coefficients adjusted by upward displacement of each pixel component value (232) to the values of the all shift module those in the matrix of For example, left 212 can shift coefficients to the left in ten positions. In this example, the left shift module 212 can shift all coefficients to the right by ten positions because the vector direct transform module 214 can use fixed point arithmetic in which numbers are encoded using ten bits in the fractional portion. Thus, by shifting the coefficients to the left in ten positions, the left shift module 212 converts

Petition 870180133578, of 9/24/2018, p. 51/169

45/69 effectively the values of pixel components in fixed point numbers with ten bits of mantissa.

[00100] After shifting each pixel component value in the adjusted value matrix to the left, the vector direct transform module 214 can determine whether a column counter is less than a maximum column counter (234). Initially, the column counter can be set to zero. The maximum column counter can be a static value that is equal to the number of columns in the adjusted coefficient matrix. For example, if the adjusted coefficient matrix includes eight columns, the maximum column counter is equal to eight.

[00101] If the column counter is smaller than the maximum column counter (SIM 234), the vector direct transform module 214 can compute a one-dimensional direct transform into a column vector indicated by the column counter (236). When the vector direct transform module 214 computes the direct transform into a column vector of the adjusted coefficient matrix, the vector direct transform module 214 replaces the original adjusted coefficients in the column vector with intermediate coefficients. After the vector direct transform module 214 computes the direct transform into a column vector of the adjusted coefficient matrix, the vector direct transform module 214 can increment the column counter (238). The vector direct transform module 214 can then return and again determine whether the column counter is less than the maximum column counter (234).

[00102] If the column counter is not less than (that is, greater than or equal to) the maximum column counter (NOT 234), the vector direct transform module 214 can determine whether a column counter line

Petition 870180133578, of 9/24/2018, p. 52/169

46/69 is less than a maximum line counter (240). Initially, the line counter can be set to zero. The maximum line counter can be a static value that is equal to the number of line vectors in the coefficient matrix. For example, if the coefficient matrix includes eight lines, the maximum line counter is equal to eight.

[00103] If the line counter is smaller than the maximum line counter (SIM 240), the vector direct transform module 214 can compute a one-dimensional discrete cosine transform into a line vector indicated by the line counter (242 ). Since the vector direct transform module 214 has already computed the direct transform in the line vectors in the matrix, the coefficient matrix now contains intermediate coefficients. When the vector direct transform module 214 computes the direct transform into a line vector of intermediate coefficients, the vector direct transform module 214 replaces the intermediate coefficients in the column vector with transformed coefficients.

[00104] After the vector direct transform module 214 computes the discrete cosine transform into a line vector of the coefficient matrix, the vector direct transform module 214 can increment the line counter (244). The vector direct transform module 214 can then return and again determine whether the line counter is less than the maximum line counter (240).

[00105] If the line counter is not less than (that is, greater than or equal to) the maximum line counter (NOT 240), the scaling module 216 can scale each coefficient transformed into the coefficient matrix processed (246). When the

Petition 870180133578, of 9/24/2018, p. 53/169

47/69 scaling 216 scaling a coefficient, the scaling module 216 can shift the coefficient to the right in a number of positions. After the scaling module 216 has scaled each of the transformed coefficients, the output module 218 can output the matrix resulting from scaled coefficients (248).

[00106] Figure 10A is a flow chart illustrating an exemplary one-dimensional transform 260. In Figure 10A, the values Xo, Xi, X2, X3, X4, X5, Xe and X7 represent input coefficients and the values xo, xi, X2, X3, X4, xs, X6 and X7 represent output values of transform 260. The value associated with a line after a circle that encloses a + symbol is the result of adding the values associated with the arrows that point to the circle. The value associated with a line after a circle that encloses an x symbol is the result of multiplying the coefficient positioned next to the circle and values associated with the lines that pass through the circles. The symbol next to an arrow represents a negation of the value associated with the arrow. For example, if the value 10 is associated with an arrow before a symbol, the value -10 is associated with an arrow after the symbol [00107] Transform 260 uses fourteen multiplications by seven singular irrational factors. For convenience, this disclosure refers to these singular irrational factors such as α, β, γ, δ, ε, ζ, and η. In transform 260, α = cos (3 ^ / 8), β = y / 2sen (3 π /%), γ - cos (ττ / 16), δ - sen (? R / 16), ε = cos (3tt / 16), ζ - if «(3Tz716), and η = 4ϊ. Note that α, β, γ, δ, ε, ζ, and η are all irrational numbers. As α, β, γ, δ, ε, ζ, and η are irrational numbers, tolerably the exact representations of α, β, γ, δ, ε, ζ, and η may require a relatively large number of bits.

Petition 870180133578, of 9/24/2018, p. 54/169

48/69

Consequently, compliance with the requirements of typical coding standards might not be possible when using fixed point numbers with relatively small numbers of mantissa bits to compute transform 260. Furthermore, such as α, β, γ, δ, ε, ζ , and η are irrational numbers, it might not be possible to reduce multiplications by a, β, γ, δ, ε, ζ, and η for sequences of displacement operations, addition operations, and / or subtraction operations.

[00108] Figure 10B is a flowchart illustrating an exemplary one-dimensional staggered transform 270. Transform 270 is a staggered version of transform 260. In transform 270, the value η was factored out of transform 260, a value (1 / Ψ) was factored values γ, δ, ε, and ζ of transform 260, and a value (l / φ) was factored out of values α and β of transform 260. When (1 / Ψ) and (l / φ) are decomposed of these coefficients, the following vector represents values by which X0-X7 are multiplied before transform 260:

<1, (l / í /), (l / ^ (V2 / í /) 1, (V2 / í /) (l / ^ (l / í /)>

The values α, β, γ, δ, ε, ζ, and η can change when (1 / Ψ) and (l / φ) are decomposed from α, β, γ, δ, ε, and ζ. For convenience purposes, the values α, β, γ, δ, ε, and ζ after (1 / Ψ) and (l / φ) are decomposed referred to as a ', β', γ ', δ', ε ', and ζ '. For example, when Ψ = 1.3552708058 and φ = 0.959700091, α '«133/256, β'« 321/256, γ '«679/512, δ'« 135/512, ε '«4605/4096 , and ζ '«1539/2048.

[00109] Values α ', β', γ ', δ', ε ', and ζ' can be selected in different ways such that they would approximate products of a ', β', γ ', δ', ε ' , and ζ 'by

Petition 870180133578, of 9/24/2018, p. 55/169

49/69 corresponding factors Ψ and φ. In general, the values of the factors Ψ and φ and the values a ', β', y ', δ', ε ', ζ' can be selected without changing the overall model of transform 270. Finding good values for these factors can represent a complex optimization problem that could produce multiple solutions with different balance of complexity / precision. For example, multiplications by some values of a ', β', y ', δ', ε ', and ζ' may be more convenient to calculate than others. Furthermore, the values of Ψ and φ can be important because they serve as the basis for scaling factors, and some scaling factors may be more convenient to calculate than others. For example, multiplications by a first set of values for a ', β', y ', δ', ε ', and ζ' can be approximated by a series of addition operations and displacement operations that are shorter than one series of addition operations and displacement operations for a different set of values for a ', β', y ', δ', ε ', and ζ'. For example, when Ψ = 1.3552708058 and φ =

0.959700091, y 'can be approximated as 679/512. Three shift operations and three addition operations may be required to approximate a multiplication by 679/512. Furthermore, when Ψ = 1.1810842 and φ = 0.801251953, y 'can be approximated as 18981/16384. Four shift operations and four addition operations can be required to approximate a multiplication by 18981/16384.

[00110] A matrix of scaling factors can be computed by multiplying the transposition of the vector <1, (l / ys), (l / φ), (fi / ys) 1, (yfè / ys) (l / φ) , (l / ys)> by the same vector. This results in an 8x8 array of values:

Petition 870180133578, of 9/24/2018, p. 56/169

50/69

F = (l / í /) * (l / ç?), G = (l / í /) * (V2 / í /), / f = (l / ç?) ² , I = (Υ / φ) * ^ 2 / ψ ·), and

J = (λ / 2 / ψ} * {^ 2! Ψ}.

[00111] As the scaling factors are multiplied with the coefficients before applying the transform, the scaling values A, B, C, D, E, F, G, Η, I, and J can be shifted to the left in P positions, where P is a constant referring to a number of bits of the fixed point mantissa (that is, bits to the right of the base point) used in a transform. This effectively converts the original coefficients into numbers of fixed points that have P bits of mantissa. In other words, each of the coefficients in the matrix is multiplied by 2 ^P. The resulting values can then be approximated by integer values. Thus, A, B, C, D, E, F, G, Η, I, and J in the above matrix can be replaced by A '& 2 ^P , B'2 ^P , C' * U <p * 2 ^p , D '«(Vw) * 2 ^P ,« (l / ç /) ² * 2 ^P , ^' «(ΐ / ί /) * (ΐ / ^) * 2 ^ρ , G '« (l / ^) * (V2 / ^) * 2 ^P , tf '«(l / p) ² * 2 ^p ,

7 '«(l / ^) * (V2 / _í /) * 2 ^p , and J'« (V2 / _Í /) * (V2 / _Í /) * 2 ^P.

[00112] For example, let Ψ = 1.3552708058 and φ = 0.959700091, and let P = 10. Given these values of

Ψ, φ, and P, the scaling factors yl '= 1024, 5' = 757,

Petition 870180133578, of 9/24/2018, p. 57/169

51/69

C = 1067, D '= 1071, £' = 560, F '= 789, <7 = 792, //' = 1112, / '= 1116, and ./' = 1120 can be selected. Thus, the following matrix of scaling factors can result:

1024 757 1067 1071 1024 1071 1067 757 757 560 789 792 757 792 789 560 1067 789 1112 1116 1067 1116 1112 789 1071 792 1116 1120 1071 1120 1116 792 1024 757 1067 1071 1024 1071 1067 757 1071 792 1116 1120 1071 1120 1116 792 1067 789 1112 1116 1067 1116 1112 789 757 560 789 792 757 792 789 560

[00113] The following tables summarize exemplary scaling factors and values of α ', β', γ ', δ', ε ', and ζ' that can be used with transform 270.

Table 1: Approximations for constant factors used in the

Algorithm A.

Factor Value Algorithms: x = x * [A, ..., J] »2; χ = χ * [α, γ, ε]; y = x * [(β, δ, ζ]; Complexity Times used Additions Unlock- camentos THE' 2048 (x * A ’) = (x« 9). 0 1 4 B' 1703 x2 = -x + (x «6), x3 = x2 + (x2« 3), x4 = (x + x3) «1, (x * B ') = (x3 + x4) »2; 4 4 8 Ç 2676 x2 = -x + (x «3), x3 = -x + (x2« 5), (x * C ') = x3 + (x3 «l); 3 3 8 D ' 2408 x2 = x + (x «5), x3 = x2 «l, x4 = x + x3, (x * D ') = x3 + (x4 «3); 3 3 8 AND' 1416 x2 = x «4, x3 = x2« 3, x4 = x3-x2, x5 = x + x4, (x * E ') = x3 + (x5 «l); 3 3 4

Petition 870180133578, of 9/24/2018, p. 58/169

52/69

Factor Value Algorithms: x = x * [A, ..., J] »2; x = x * [α, γ, ε]; y = x * [(β, δ, ζ]; Complexity Times used Additions Unlock- camentos F ’ 2225 x2 = x «5, x3 = x + (x2« 2) _r x4 = x3 «4, x5 = x2 + x3, (x * F ') = (x4 + x5)» 2; 3 4 8 G ' 2003 x2 = x «4, x3 = x2-x, x4 = -x3 + (x2« 5), (x * G ') = (x3 »2) + x4; 3 3 8 H' 3496 x2 = x «3, x3 = x2-x, x4 = x3« l, x5 = x2 + x4, (x * H ') = -x5 + (x4 «6); 3 3 4 I ' 3147 x2 = x «9, x3 = -x + (x« 4), x4 = x2 + x3, x5 = x2 + x4, (x * I ') = (x5 »2) + x4; 4 3 8 J ' 2832 x2 = x «2, x3 = x2-x, x4 = x3« 6, x5 = x2 + x4, (x * J ') = x5 + (x2' 7); 3 3 4 The' 53/12 8 (x * β ’) = χ, 3 3 2 β ’ 1 x2 = x »2, x3 = x + x2, x4 = x2 - x, (x * a ') = (x3 »5) (x4» 1); γ ’ 151/1 28 x2 = x - (x »4), 3 3 2 δ * 15/32 x3 = x2 + (x »7), (x * δ ’) = x2» 2; (x * γ ’) = y + x3; ε ' 1 (x * ε ’) = x 3 3 2 171/2 56 x2 = (x »3) - x; x3 = x + (xc »3); (x * ζ ’) = x3 - (x3» 2) ; Staggered complexity: of the transformed ID 44 18

Petition 870180133578, of 9/24/2018, p. 59/169

53/69

Factor Value Algorithms: x = x * [A, ..., J] »2; x = x * [α, γ, ε]; y = x * [(β, δ, ζ]; Complexity Times used Additions Unlock- camentos Total 2D scaling complexity: 196 200 Complexity of a transform Full 2D: 901 552

Table 2: Approximations for constant factors used in Algorithm B.

Factor Value Algorithms: x = x * [ . , J, Ot, Y, £] f Y = X * [β, δ, ζ] / Complexity Times used Additions Offsets THE' 1024 (χ * A ’) = x« 10; 0 1 4 B' 757 x2 = -x + (x «6), x3 = (x2« 2) _r x4 = (x3 «l), (x * B ') = x + x3, x = x4 + x5; 3 3 8 V 1067 x2 = -x + (x «6), x3 = -x + (x2« 2) _r (x * C ') = x3 «2, x = x2 + x4; 3 3 8 D ' 1071 x2 = x «6, x3 = x2-x, (x * D ') = x3 + (x3 «4); 2 2 8 AND' 560 x2 = x «5, x3 = x + x2, x4 = x3« 4, (x * E ') = x2 + x4; 2 2 4 F ’ 789 x2 = x «3, x3 = x2-x, x4 = x3 + (x2« 5), (x * F ') = x4 + (x4 «l); 3 3 8 G ' 792 x2 = x + (x «5), x3 = x2 «l, x4 = x2 + x3, (x * G ') = x4 «3; 2 3 8

Petition 870180133578, of 9/24/2018, p. 60/169

54/69

Factor Value Algorithms: x = x * [ y = x * [β, δ, ζ] / Complexity Times used Additions Offsets H' 1112 x2 = x «4, x3 = x2-x, x4 = x2 + x3, x5 = x3 + (x4« 2), (x * H ') = x5 «3; 3 3 4 I ' 1116 x2 = -x + (x «5), x3 = x2 «3, x4 = x2 + x3, (x * I ') = x4 «2; 2 3 8 J ' 1120 x2 = -x + (x «3), x3 = x2 + (x2« 2), (x * J ') = x3' 5; 2 3 4 The' 133/256 x2 = x »2, x3 = x + x2, x4 = x + (x3» 5), (x * a ') = x4 »l; (x * β ’) = x3 (-x2» 6); 3 4 2 β ’ 321/256 γ ’ 679/512 x2 = x »4, x3 = x» 9, x4 = x + x2, (x * δ ’) = x3 + (x4» 2); (x * γ ’) = x4 + (x * δ’); 3 3 2 δ * 135/512 ε ’ 4605/4096 x2 = x + (x »l), x3 = x2 + (x2» 9), - (x * ζ ’) = x3» l; (x * ε ’) = x2 (y» l); 3 4 2 ζ ’ 1539/2048 Complexity of transformed 44 22

Petition 870180133578, of 9/24/2018, p. 61/169

55/69

Factor Value Algorithms: x = x * [ γ = χ * [β, δ, ζ] / Complexity Times used Additions Offsets Stepped ID: Total 2D scaling complexity: 148 172 2D transform w / o scaling complexity: 704 352 Complexity of a complete 2D transform: 852 + 1 524 + 64

Table 3: Approximations for constant factors used in Algorithm C.

Factor Value Algorithms: x = x * [ y = x * [β, δ, ζ] / Complexity Times used Additions Offsets THE' 1024 (χ * A ’) = x« 10; 0 1 4 B' 1138 x2 = xl «l, x3 = x2« 3, x4 = x2 + x3, x5 = x3-x2, (x * B ’) = (x4-6) -x5; 3 3 8 Ç 1730 x2 = xl «l, x3 = xl + x2, x4 = x3« 3, x5 = x3 + x4, (x * C ') = x2 + (x5 «6); 3 3 8 D ' 1609 x2 = xl «3, x3 = x2« l, x4 = xl + x2, x5 = x3 + x4, (x * D ') = x4 + (x5 «6); 3 3 8 AND' 1264 x2 = xl «2, x3 = x2« 2, x4 = x2 + x3, x5 = x4 «6, (x * E ') = x5-x3; 2 3 4 F ’ 1922 x2 = xl «l, x3 = x2« 3, x4 = -xl + x3, x5 = x4 «7, 2 3 8

Petition 870180133578, of 9/24/2018, p. 62/169

56/69

Factor Value Algorithms: x = x * [ y = x * [β, δ, ζ] / Complexity Times used Additions Offsets (x * F ') = x2 + x5; G ’ 1788 x2 = xl «2, x3 = x2« l, x4 = x3-xl, x5 = x4 «8, (x * G ') = x5x2; 2 3 8 H' 2923 x2 = xl «l, x3 = xl + x2, x4 = (x3« 3) -xl, x5 = x4-x2, (x * H ') = (x4-7) -x5; 4 3 4 I ' 2718 x2 = xl «l, x3 = x2« l, x4 = xl + x3, x5 = x4 + (x4 «4), (x * I ') = (x5 '5) -x2; 3 4 8 J ' 2528 x2 = xl «2, x3 = xl + x2, x4 = x3« 4, x5 = x4-xl, (x * J ') = x5' 5; 2 3 4 The' 41/128 x2 = x + (x »5); x3 = x2 »2; (x * a ’) = x3 + (x» 4); (x * β ’) = x2-x3; 3 3 2 β ’ 99/128 γ ’ 113/128 x2 = (x »3) - (x» 7); x3 = x2- (x »ll); (x * δ ’) = x2 + (x3» l); (x * γ ’) = x-x2; 4 4 2 δ * 719/4096 ε ’ 1533/2048 x2 = (x »9) -x; (x * ζ ’) = x» l; (x * ε ’) = (x2 »2) -x2; 2 3 2 ζ ’ 1/2 Complexity of a scaled 1D transform: 44 20

Petition 870180133578, of 9/24/2018, p. 63/169

57/69

Factor Value Algorithms: x = x * [ y = x * [β, δ, ζ] / Complexity Times used Additions Offsets Total 2D scaling complexity: 160 192 Complexity of a complete 2D transform: 865 576

[00114] To illustrate how the IDCT 48 module can use the scaling factors, constant values, and algorithms provided in the tables above, consider Table 3 above. To apply an inverse discrete cosine transform using the values in Table 3, input module 140 in module IDCT 48 can receive an 8x8 coefficient matrix. Then, the scaling module 142 scales a coefficient at the position [0,0] of the matrix by a factor A; scales a coefficient at position [0.1] of the matrix by a factor B; scale a coefficient at position [0,2] of the matrix by a factor C; scales a coefficient at position [0.3] of the matrix by a factor D; schedules a coefficient at position [0.4] of the matrix by factor A; scales a coefficient at position [0.5] of the matrix by factor D; escalates a coefficient at position [0.6] of the matrix by factor C; scale a coefficient at position [0.7] of the matrix by factor B; escalates a coefficient at position [1.0] of the matrix by factor B; scales a coefficient at position [1,1] of the matrix by factor E; scales a coefficient at position [1,2] of the matrix by a factor F; scales a coefficient at position [1,3] of the matrix by a G factor; scales a coefficient at position [1,4] of the matrix by factor B; scale a coefficient at position [1.5] of the matrix by the G factor; scales a coefficient at position [1,6] of the matrix by the F factor; scales a coefficient at position [1,7] of the matrix by factor E; escalates a coefficient at position [2,0] of the matrix by factor C; schedule one

Petition 870180133578, of 9/24/2018, p. 64/169

58/69 coefficient in the position [2,1] of the matrix by the F factor; scales a coefficient at position [2,2] of the matrix by an H factor; scales a coefficient at position [2,3] of the matrix by a factor I; scales a coefficient at position [2,4] of the matrix by factor C; scale a coefficient at position [2,5] of the matrix by factor I; scales a coefficient at position [2,6] of the matrix by factor H; scales a coefficient at position [2,7] of the matrix by the F factor; scales a coefficient at position [3.0] of the matrix by factor D; scales a coefficient at position [3.1] of the matrix by the G factor; scales a coefficient at position [3,2] of the matrix by factor I; scales a coefficient at position [3,3] of the matrix by factor J; scales a coefficient at position [3,4] of the matrix by factor D; scales a coefficient at position [3,5] of the matrix by factor J; scales a coefficient at position [3,6] of the matrix by factor I; scales a coefficient at position [3.7] of the matrix by the G factor; scale a coefficient at position [4.0] of the matrix by factor A; scales a coefficient at position [4.1] of the matrix by factor B; scales a coefficient at position [4.2] of the matrix by factor C; scales a coefficient at position [4,3] of the matrix by factor D; scales a coefficient at position [4.4] of the matrix by factor A; scales a coefficient at position [4,5] of the matrix by factor D; scale a coefficient at position [4.6] of the matrix by factor C; scales a coefficient at position [4,7] of the matrix by factor B; scales a coefficient at position [5,0] of the matrix by factor D; scales a coefficient at position [5.1] of the matrix by the G factor; scales a coefficient at position [5.2] of the matrix by factor I; scales a coefficient at position [5.3] of the matrix by factor J; scales a coefficient at position [5.4] of the matrix by factor D; escalates a coefficient in the position

Petition 870180133578, of 9/24/2018, p. 65/169

59/69 [5.5] of the matrix by factor J; scales a coefficient at position [5.6] of the matrix by factor I; scales a coefficient at position [5,7] of the matrix by the G factor; schedules a coefficient at position [6.0] of the matrix by factor C; scales a coefficient at position [6.1] of the matrix by the F factor; scale a coefficient at position [6.2] of the matrix by an H factor; scales a coefficient at position [6.3] of the matrix by a factor I; scale a coefficient at position [6.4] of the matrix by factor C; scales a coefficient at position [6.5] of the matrix by factor I; scales a coefficient at position [6.6] of the matrix by factor H; scales a coefficient at position [6,7] of the matrix by the F factor; scale a coefficient at position [7.0] of the matrix by factor B; scales a coefficient at position [7.1] of the matrix by an E factor; scales a coefficient at position [7,2] of the matrix by an F factor; scales a coefficient at position [7.3] of the matrix by a G factor; scales a coefficient at position [7.4] of the matrix by factor B; scales a coefficient at position [7,5] of the matrix by the G factor; scale a coefficient at position [7.6] of the matrix by the F factor; and scale a coefficient at position [7,7] of the matrix by factor E. When scaling module 142 scales these coefficients, can scaling module 142 use the values A = 1024, 7? = 1138, C = 1730, D = 1609, E = 1264, F = 1922, G = 1788, H = 2923, / = 2718, and J = 2528, as specified in Table 6. These scaled coefficients constitute a matrix of coefficients staggered.

[00115] After the scaling module 142 scaling the coefficients, the inverse transform module

6 can apply a transform to each line vector of the scaled coefficient matrix to produce a matrix

Petition 870180133578, of 9/24/2018, p. 66/169

60/69 of intermediate coefficients and apply the transform to each column vector of the matrix of intermediate coefficients to produce a matrix of transformed coefficients. The inverse transform module 146 can apply the transform to a row vector or a column vector by: calculating an xo 'value by adding xo and X4; calculating an X4 'value by adding xo and -X4; calculating a value (X2 * a) by multiplying X2 by a value a; calculation of a value (Χ6 * β) by multiplying Xô by a β value; calculation of a value (Χ2 * β) by multiplying X2 by the value β; calculating a value (χδ * α) by multiplying Χδ by a value a; calculation of an X2 'value by adding (X2 * a) and - (Χ6 * β); calculation of an X6 ’value by adding (χδ * α) and (Χ2 * β); calculating an xo value by adding xo 'and xô'; calculating an X4 value by adding X4 'and X2'; calculating an X2 value by adding X4 'and -X2'; calculating a Χδ value by adding xo 'and -xô'; calculating an χγ 'value when adding xi' and -X7, calculating an xi 'value when adding xi and X7; calculating an X3 'value by multiplying X3 and square root of two; calculating an X5 'value by multiplying X5 and square root of two; calculating an X7 value by adding X7 'and X5'; calculating an X3 value by adding xi 'and -X3'; calculating an X5 value by adding X7 'and -X5'; calculating an xi value by adding X3 ’and xi 'by calculating a value (Χ7’ ’* ε) by multiplying X7 and an ε value; calculating a value (Χ7 ’’ * ζ) when multiplying X7 and a value ζ calculating a value (Χ3 ’’ * γ) when multiplying X3 and a γ value; calculating a value (Χ3 ’’ * δ) by multiplying X3 and a δ value; calculating a value (Χ5 ’’ * δ) by multiplying X5 and the value δ; calculating a value (Χ5 ’’ * γ) by multiplying X5 and the value γ; calculating a value (χι ’’ * ζ) by multiplying xi and the value ζ calculating a value (χι ’’ * ε)

Petition 870180133578, of 9/24/2018, p. 67/169

61/69 when multiplying xi and the value ε; calculating an xj value '''by adding (Χ7 * ε) and - (χι * ζ); calculating an X3 value '''when adding (Χ3 * γ) and - (Χ5 * δ); calculating a value x $ '''when adding (Χ5 ^π * γ) and (Χ3 * δ); calculating a value xi '''when adding (χι * ε) and (Χ7 * ζ); calculating an Xo value by adding xj '''andxo; calculating an Xi value when adding X4 and X5 '''; calculating an X2 value when adding X2 and X3 '''calculating an X3 value when adding xô and xj'''; calculating an X4 value when adding xô and -xi '''; calculating an X5 value when adding X2 and -X3 '''; calculating a Xô value when adding X4 and -X5 '''; calculating an X7 value when adding xo and -xi ''', where xo, xi, X2, X3, X4, X5, X6, X7 are coefficients in a row vector or in a column vector and Xo, Xi, X2, X3 λ X4, X5, Χβ, and X7 are output values of the transform. When performing these calculations, the inverse transform module 146 uses the fractional values α = 41/128, β = 99/128, γ = 1522/2048, δ = 1/2, ε = 113/128, and ζ = 719 / 4096, products by which they are computed by the algorithms without a multiplier as specified in Table 3.

[00116] The following code contains an exemplary C-language implementation of transform 270:

/ * Set the scaling factor values #define A 1024 üdefine B 1138 üdefine C 1730 üdefine D 1609 üdefine E 1264 üdefine F 1922 üdefine G 1788 üdefine H 2923 #define I 2718 #define J 2528 / * Define the matrix of escalation factors * / static int scale [8 * 8] = { A B C D, A, D, ç, B,

Petition 870180133578, of 9/24/2018, p. 68/169

62/69

Β, Ε,

C, F,

D, G, Α, Β, D, G, C, F, Β, Ε,};

F, G, Η, I, I, J, C, D, I, J, Η, I, F, G,

Β, G,

C, I,

D, J, Α, D, D, J, C, I, Β, G,

F, Ε, Η, F, I, D, C, Β, I, D, Η, F, F, Ε, void scaled_ld_idct (int *, int *); void idct (short * P) int block [8 * 8], block2 [8 * 8], i;

/ * Scale each of the input coefficients * / for (i = 0; i <64; i ++) block [i] = scale [i] * P [i];

/ * Add polarization for the DC coefficient * / block [0] + = 1 «12;

/ * Apply the inverse vector discrete cosine transform for each row and column of the block * / for (i = 0; i <8; i ++) scaled_ld_idct (block + i * 8, block2 + i); for (i = 0; i <8; i ++) scaled_ld_idct (block2 + i * 8, block + i);

/ * Shift each transformed coefficient to the right * / for (i = 0; i <64; i ++)

P [i] = block [i] »13;

} #define mul_l (y, z) {

int y2, y3;

y2 = (y »3) - (y» 7); y3 = y2 - (y »11);

Petition 870180133578, of 9/24/2018, p. 69/169

63/69 ζ = y2 + (y3 »1); y = y - y2;

} #define mul_2 (y, z) {

int y2; y2 = (y » 9) - y; z = y » 1; y = (y2 » 2) - y2;

} #define mul_3 (y, z) {

int y2, y3; y2 = y + (y »5); y3 = y2 »2; y = y3 + (y »4); z = y2 - y3;

} void scaled_ld_idct (int * in, int * out) {

int xO, xl, x2, x3, x4, x5, x6, x7, xa, xl = in [l]; x3 = in [3]; x5 = in [5]; x7 = in [7];

xb;

xa = xl + x7; xb = xl - x7;

xl = xa + x3; x3 = xa - x3; x7 = xb + x5;

Petition 870180133578, of 9/24/2018, p. 70/169

64/69

x5 = xb - x5; very 1 (x3, xa); very 1 (x5, xb); x3 = x3 - xb; x5 = x5 + shah; mui 2 (xl , xa) very 2 (x7 , xb) xl = xl + xb; x7 = x7 - shah; xO = in [ 0]; x2 = in [: 2] ; x4 = in [ 4]; x6 = in [ 6]; very 3 (x2, xa); very 3 (x6, xb); x2 = x2 - xb; x6 = x6 + shah;

xa = χθ + x4; xb = xO - x4;

xO = xa + x6; x6 = xa - x6; x4 = xb + x2; x2 = xb - x2; out [0 * 8] = xO + xl; out [l * 8] = x4 + x5; out [2 * 8] = x2 + x3; out [3 * 8] = x6 + x7;

Petition 870180133578, of 9/24/2018, p. 71/169

65/69

out [4 * 8] = x6 - x7; out [5 * 8] = x2 - x3; out [6 * 8] = x4 - x5; out [7 * 8] = xO - xl;

} void scaled_ld_fdct (int *, int *); void fdct (short * P) {

int block [8 * 8], block2 [8 * 8], i;

for (i = 0; i <64; i ++) block [i] = P [i] «7;

for (i = 0; i <8; i ++) scaled_ld_fdct (block + i, block2 + i * 8);

for (i = 0; i <8; i ++) scaled_ld_fdct (block2 + i, block + i * 8); for (i = 0; i <64; i ++)

P [i] = (block [i] * scale [i] + 0x7FFFF (block [i] »31))» 20;

} void scaled_ld_fdct (int * in, int * out) {

int xO, xl, x2, x3, x4, x5, x6, xO = in [0 * 8] + in [7 * 8] xl = in [0 * 8] - in [7 * 8] x4 = in [l * 8] + in [6 * 8] x5 = in [l * 8] - in [6 * 8] x2 = in [2 * 8] + in [5 * 8] x3 = in [2 * 8] - in [5 * 8] x6 = in [3 * 8] + in [4 * 8] x7 = in [3 * 8] - in [4 * 8] very 1 (x3 , xa);

Petition 870180133578, of 9/24/2018, p. 72/169

66/69 mul_l (x5, xb); x3 = x3 + xb; x5 = x5 - xa;

mul 2 (xl, xa); very 2 (x7, xb); xl = xl - xb; x7 = x7 + xa; xa = xl + x3; x3 = xl - x3; xb = x7 + x5; x5 = x7 - x5; xl = xa + xb; x7 = xa - xb; xa = xO + x6; x6 = xO - x6; xb = x4 + x2; x2 = x4 - x2; xO = xa + xb; x4 = xa - xb; mul 3 (x2, xa); mul 3 (x6, xb); x2 = xb + x2; x6 = x6 - xa; out [0] = xO; out [l] = xl; out [2] = x2; out [3] = x3; out [4] = x4; out [5] = x5;

Petition 870180133578, of 9/24/2018, p. 73/169

67/69 out [6] = x6;

out [7] = x7;

[00117] Figure 11 is a flowchart illustrating an exemplary transform 280 used by the direct transform module 214, in Figure 11, the values Xo, Xi, X2, X3, X4, X5, Χδ, and X7 represent output coefficients and the xo, xi, X2, X3, X4, X5, xô, and X7 values represent input values. The value associated with a line after a circle that encloses a + symbol is the result of adding the values associated with the arrows that point to the circle. The value associated with a line after a circle that encloses an x symbol is the result of multiplying the coefficient positioned next to the circle and values associated with the lines passing through the circles. The symbol next to an arrow represents a negation of the value associated with the arrow. For example, if the value 10 is associated with an arrow before a symbol, the value

-10 is associated with the arrow after the symbol In addition, it should be noted that the techniques described above to reduce rounding error using negative coefficients, and subtraction, can be used in the algorithm

190.

[00118] In transform 280, the output values Xo to X7 are multiplied by the factors 1, φ, φ, ψ, 42! Ψ, 42! Ψ and ψ, respectively. Note that these factors are the reciprocal of the decomposed values of α ’, β’, γ ’, δ’, ε ’, and ζ’ in transform 270.

[00119] The techniques described here can be implemented in hardware, software, firmware, or any combination thereof. Any features described as modules or components can be implemented in

Petition 870180133578, of 9/24/2018, p. 74/169

68/69 together in an integrated logic device, or separately as discrete but interoperable logic devices. If implemented in software, the techniques can be performed at least in part by a computer-readable medium comprising instructions that, when executed, perform one or more of the methods described above. The computer-readable medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise random access memory (RAM) such as dynamic, synchronous, random access memory (SDRAM), read memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read memory (EEPROM), flash memory, magnetic or optical data storage media, and the like. The techniques can additionally or alternatively be performed at least in part by a computer-readable communication medium that carries or communicates code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer.

[00120] The code can be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable port arrangements (FPGAs) , or other sets of equivalent integrated or discrete logic circuits. Consequently, the term processor, as used herein, can refer to any of the aforementioned structure or any other structure suitable for implementing the techniques described herein. In addition, in some respects, the functionality described here can be provided within software modules or dedicated hardware modules configured for

Petition 870180133578, of 9/24/2018, p. 75/169

69/69 encode and decode, or incorporated into a combined video encoder (CODEC).

[00121] Various embodiments of the invention have been described. These and other modalities are within the scope of the following claims.

Petition 870180133578, of 9/24/2018, p. 76/169

1/4

Claims (13)

1. Unpacking method comprising: stagger (172) each coefficient in a matrix 8x8 of coefficients encoded by one of a factor A, a factor B, a factor C, a factor D, a factor E, a factor F, a factor G, a factor H, a factor I, or a factor J in order to produce a staggered coefficient matrix, where / 1 = 1024, 5 = 1138, C = 1730, £) = 1609, £ = 1264, £ = 1922, G = 1788, H = 2923, I = 2718, and J = 2528; use (176 to 186) repeated applications of a one-dimensional staggered fixed-point transform to transform the staggered coefficient matrix into a transformed coefficient matrix; moving the transformed coefficients to the right (188) in the transformed coefficient matrix in order to produce an adjusted coefficient matrix; display an 8x8 pixel block, where each pixel in the 8x8 pixel block includes a pixel component value based on an adjusted coefficient in the adjusted coefficient matrix; the method characterized by the fact that it additionally comprises: adding a midpoint bias term to a DC coefficient of the matrix, where the midpoint bias value term is equal to 2 for the power of 12; and shift each coefficient in the matrix of transformed coefficients to thirteen positions to the right to generate a matrix of pixel component values. 2. Method, according to claim 1, characterized by the fact that repeatedly applying the one-dimensional staggered fixed-point transform comprises: of 09/24/2018, p. 146/169 2/4 apply (178) the one-dimensional staggered fixed point transform to each line vector of the staggered coefficient matrix to produce an intermediate coefficient matrix; and applying (184) the one-dimensional staggered fixed point transform to each column vector of the matrix of intermediate coefficients to produce a matrix of transformed coefficients. 3. Method, according to claim 1, characterized by the fact that displaying the 8x8 pixel block comprises displaying the 8x8 pixel block as part of a video frame in a video sequence. 4. Method, according to claim 1, characterized by the fact that displaying the 8x8 pixel block comprises displaying the 8x8 pixel block as part of a static image. 5. Method, according to claim 1, characterized by the fact that the method additionally comprises receiving a bit stream that includes the coefficient matrix. 6. Method, according to claim 1, characterized by the fact that the method additionally comprises: build the pixel block by incorporating pixel component values into the pixel component value matrix in pixels in the pixel block. 7. Method, according to claim 1, characterized by the fact that scaling each coefficient in the 8x8 coded coefficient matrix comprises scaling the coefficients in the coded coefficient matrix without using multiplication or division operations. 8. Unzip device comprising: of 09/24/2018, p. 147/169 3/4 mechanisms (142) for scaling coefficients in an 8x8 matrix of coefficients encoded by one of a factor A, a factor B, a factor C, a factor D, a factor E, a factor F, a factor G, a factor H, a factor I, or a factor J to produce a staggered coefficient matrix, where, 4 = 1024, 5 = 1138, C = 1730, £) = 1609, £ = 1264, F = 1922, G = 1788, H = 2923, I = 2718, and J = 2528; mechanisms (146) for using repeated applications of a one-dimensional staggered fixed-point transform to transform the staggered coefficient matrix into a transformed coefficient matrix; mechanisms (148) for displacing transformed coefficients in the matrix on the right transformed coefficients to produce an adjusted coefficient matrix; mechanisms (152) for outputting an 8x8 pixel block, where each pixel in the pixel block includes a pixel component value based on an adjusted coefficient in the adjusted coefficient matrix; the device characterized by the fact that it additionally comprises: mechanisms for adding a midpoint bias term to a DC coefficient, where the midpoint bias term is equal to 2 for the power of 12; and mechanisms for shifting each coefficient in the matrix of coefficients transformed into thirteen positions to generate a matrix of pixel component values. 9. Device, according to claim 8, characterized by the fact that the device comprises mechanisms to receive a media file that includes data from which the matrix of the coefficients of 09/24/2018, p. 148/169 4/4 encoded is derived. 10. Device according to claim 8, characterized by the fact that the device additionally: understands 5 mechanisms to build the block of pixels to incorporate pixel component values in the matrix of pixel component values in pixels in the block of pixels. 11. Device according to claim 8, 10 characterized by the fact that the device additionally comprises mechanisms for displaying the pixel block. 12. Device, according to claim 8, characterized by the fact that the mechanisms to scale the coded coefficients scale the coefficients 15 encoded without using multiplication or division operations. 13. Computer-readable medium characterized by the fact that it contains the method of any one of claims 1 to 7. Petition 870180133578, of 9/24/2018, p. 149/169 1/12 DEVICE FIGURE 1 Petition 870180133578, of 9/24/2018, p. 157/169 12/2 COMPUTING DIPOSITIVE 2 FIGURE Petition 870180133578, of 9/24/2018, p. 158/169 12/3 FIGURE 3 Petition 870180133578, of 9/24/2018, p. 159/169 4/12 FIGURE 4 Petition 870180133578, of 9/24/2018, p. 160/169 5/12 FIGURE 5 Petition 870180133578, of 9/24/2018, p. 161/169 6/12 FIGURE 6 Petition 870180133578, of 9/24/2018, p. 162/169 7/12 FIGURE 7 Petition 870180133578, of 9/24/2018, p. 163/169 12/8 FIGURE 8 Petition 870180133578, of 9/24/2018, p. 164/169 9/12 FIGURE 9 Petition 870180133578, of 9/24/2018, p. 165/169 12/10 260 FIGURE 10A Petition 870180133578, of 9/24/2018, p. 166/169 12/11 270 FIGURE 10B Petition 870180133578, of 9/24/2018, p. 167/169 12/12 280 FIGURE 11 Petition 870180133578, of 9/24/2018, p. 168/169