US20220159286A1 - Low-latency consumption of an encoded video bitstream - Google Patents
Low-latency consumption of an encoded video bitstream Download PDFInfo
- Publication number
- US20220159286A1 US20220159286A1 US17/587,497 US202217587497A US2022159286A1 US 20220159286 A1 US20220159286 A1 US 20220159286A1 US 202217587497 A US202217587497 A US 202217587497A US 2022159286 A1 US2022159286 A1 US 2022159286A1
- Authority
- US
- United States
- Prior art keywords
- frame
- bitstream
- buffer
- video encoder
- encoding
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 239000000872 buffer Substances 0.000 claims abstract description 96
- 238000000034 method Methods 0.000 claims abstract description 43
- 230000015654 memory Effects 0.000 claims description 21
- 230000004044 response Effects 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 24
- 238000013459 approach Methods 0.000 description 9
- 230000005540 biological transmission Effects 0.000 description 6
- 230000006835 compression Effects 0.000 description 6
- 238000007906 compression Methods 0.000 description 6
- 238000012545 processing Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 4
- 230000007246 mechanism Effects 0.000 description 3
- 230000002093 peripheral effect Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 241000699670 Mus sp. Species 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
- H04N19/423—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation characterised by memory arrangements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
- H04N19/436—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation using parallelised computational arrangements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/46—Embedding additional information in the video signal during the compression process
- H04N19/463—Embedding additional information in the video signal during the compression process by compressing encoding parameters before transmission
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/593—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving spatial prediction techniques
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/70—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/146—Data rate or code amount at the encoder output
- H04N19/152—Data rate or code amount at the encoder output by measuring the fullness of the transmission buffer
Definitions
- H.264 is a video compression standard, proposed by the Joint Video Team (JVT).
- JVT Joint Video Team
- Many of today's multimedia-enabled digital devices incorporate digital video codec's (hardware and/or software configured to encode and/or decode data) that conform to the H.264 standard.
- the High Efficiency Video Coding (HEVC) standard is another video compression standard which followed H.264.
- a hardware accelerator e.g., video encoder
- the encoded bitstream is typically written to memory before being consumed by another logical unit (e.g., for network transmission).
- Video consumption (e.g., streaming) of each encoded video frame is typically initiated after each frame is completely encoded.
- a video encoder typically waits until an entire frame has been encoded to confirm that all bitstream memory write operations have been completed, confirm how many bits have been written, and provide a way for the bitstream consumer to determine the number of bits that were written and the location of the bitstream on the storage device.
- a disadvantage of this approach is that it means consumption of the encoded frame can only begin when the entire frame has been encoded. This prevents frame encoding from being pipelined with other actions such as network transmission, resulting in increased latency.
- FIG. 1 is a block diagram of one implementation of a computing system.
- FIG. 2 is a timing diagram of one implementation of the operations for an encoder and a bitstream consumer.
- FIG. 3 is a timing diagram of another implementation of the operations for an encoder and a bitstream consumer.
- FIG. 4 is a block diagram of another implementation of a computing system.
- FIG. 5 is a block diagram of one implementation of a video encoder coupled to a consumer module.
- FIG. 6 is a generalized flow diagram illustrating one implementation of a method for enabling on-the-fly access to an encoded video bitstream of a given video frame while the given video frame is being encoded.
- FIG. 7 is a generalized flow diagram illustrating one implementation of a method for a consumer module accessing an encoded video bitstream in real-time as the bitstream is being encoded.
- FIG. 8 is a generalized flow diagram illustrating one implementation of a method for controlling the granularity of bitstream write pointer updates.
- FIG. 9 is a generalized flow diagram illustrating one implementation of a method for a video encoder performing prediction across chunks being consumed.
- FIG. 10 illustrates a timing diagram of one implementation of real-time consumption of chunks of an encoded video bitstream.
- a system includes at least one or more processors coupled to a memory device.
- the one or more processors include circuitry which executes at least a video encoder and a consumer module.
- the consumer module is any of various types of logical units which access and consume (e.g., send over a network) the encoded video bitstream in real-time as the video bitstream is encoded by the video encoder. Rather than waiting until an entire frame or slice has been encoded by the video encoder, the consumer module starts to process the encoded bitstream mid-frame and/or mid-slice.
- slice-encoding is typically codec-specific, and any of various types of slice encodings are used in various embodiments.
- the techniques described herein for on-the-fly consumption of an encoded video bitstream are applicable to full frame encoding or any codec-specific sub-frame encoding (e.g., H-264 slice encoding)
- the video encoder in order to facilitate the consumption of the encoded bitstream prior to completing the encoding of the entire frame, the video encoder periodically updates a buffer write pointer with an indication of the amount of encoded data written to a video buffer.
- a granularity at which the video encoder updates the buffer write pointer is programmable.
- the consumer module programs the granularity at which the video encoder updates the buffer write pointer.
- the video encoder is configurable to update the buffer write pointer after writing a particular number of bits of encoded data to the buffer (e.g., 512 bits, 1024 bits, etc.).
- the video encoder updates the buffer write pointer and sets the frame completion flag to indicate encoding of the frame is complete.
- the chunk has a different size than the above mentioned granularity (i.e., the chunk size is variable).
- the consumer module monitors updates to the buffer write pointer. After detecting an update to the buffer write pointer, the consumer module reads from the bitstream buffer up to a location indicated by the buffer write pointer.
- the buffer write pointer also includes a frame completion flag to indicate whether or not frame encoding has completed for the current flame.
- the frame completion flag is set to true (e.g., 1) by the video encoder when frame encoding is completed for the current frame and the buffer write pointer has the final frame bitcount for the current frame.
- the frame completion flag is reset to false (e.g., 0) by the consumer module after the consumer module has retrieved all of the encoded data from the video buffer for the current frame.
- the frame completion flag provides a way for the consumer module to determine that the buffer write pointer will no longer be updated for the current frame, allowing the consumer module to read any remaining encoded bitstream data even if the amount of data is less than the specified granularity.
- frame completion synchronization is implemented separately from buffer write pointer updates.
- computing system 100 includes at least processors 105 A-N, input/output (I/O) interfaces 120 , bus 125 , memory controller(s) 130 , network interface 135 , and memory device(s) 140 .
- processors 105 A-N are representative of any number of processors which are included in system 100 .
- processor 105 A is a general purpose processor, such as a central processing unit (CPU).
- processor 105 N is a data parallel processor with a highly parallel architecture.
- Data parallel processors include graphics processing units (GPUs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and so forth.
- processors 105 A-N include circuitry to implement a video encoder and a consumer module. The consumer module accesses and consumes an encoded bitstream of a video frame in real-time while the video encoder is encoding the video frame.
- Memory controller(s) 130 are representative of any number and type of memory controllers accessible by processors 105 A-N and I/O devices (not shown) coupled to I/O interfaces 120 . Memory controller(s) 130 are coupled to any number and type of memory devices(s) 140 . Memory device(s) 140 are representative of any number and type of memory devices. For example, the type of memory in memory device(s) 140 includes Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), NAND Flash memory, NOR flash memory, Ferroelectric Random Access Memory (FeRAM), or others.
- DRAM Dynamic Random Access Memory
- SRAM Static Random Access Memory
- NAND Flash memory NAND Flash memory
- NOR flash memory NOR flash memory
- I/O interfaces 120 are representative of any number and type of I/O interfaces (e.g., peripheral component interconnect (PCI) bus, PCI-Extended (PCI-X), PCIE (PCI Express) bus, gigabit Ethernet (GBE) bus, universal serial bus (USB)).
- PCI peripheral component interconnect
- PCI-X PCI-Extended
- PCIE PCI Express
- GEE gigabit Ethernet
- USB universal serial bus
- peripheral devices include (but are not limited to) displays, keyboards, mice, printers, scanners, joysticks or other types of game controllers, media recording devices, external storage devices, network interface cards, and so forth.
- Network interface 135 is used to receive and send network messages across a network.
- computing system 100 is a computer, laptop, mobile device, game console, server, streaming device, wearable device, or any of various other types of computing systems or devices. It is noted that the number of components of computing system 100 varies from implementation to implementation. For example, there can be more or fewer of each component than the number shown in FIG. 1 . It is also noted that computing system 100 can include other components not shown in FIG. 1 . Additionally, in other implementations, computing system 100 can be structured in other ways than shown in FIG. 1 .
- an encoder 205 encodes individual frames 0 - 2 of a video stream and makes each frame available to bitstream consumer 210 at the end of each frame. For example, encoder 205 confirms bitstream memory write operations have completed, confirms how many bits have been written to the bitstream buffer, and then provides a way for the bitstream consumer to determine the size and location of the encoded bitstream in memory. It is noted that bitstream consumer 210 can also be referred to as a “consumer module”. Encoder 205 encodes the video stream in accordance with any of various coding standards.
- encoder 205 encodes a bitstream to be compliant with the H.264 video compression standard. In another implementation, encoder 205 encodes a bitstream to be compliant with the High Efficiency Video Coding (HEVC) standard. In other implementations, encoder 205 encodes a bitstream to be compliant with other standards.
- HEVC High Efficiency Video Coding
- bitstream consumer 210 is only able to start consuming a given frame after encoder 205 has encoded the entire frame. For example, only after frame 0 has been completely encoded by encoder 205 is bitstream consumer 210 able to start consuming the encoded bitstream generated for frame 0 . The same sequence of events occurs for frames 1 - 2 . This results in a delay from the time when encoder 205 starts encoding a frame and to the time when bitstream consumer 210 is able to start consuming the encoded frame.
- FIG. 3 a timing diagram 300 of another implementation of the operations for an encoder 305 and a bitstream consumer 310 is shown.
- encoder 305 encodes slices of a frame and then bitstream consumer 310 is able to consume an individual slice after encoder 305 has finished encoding the given slice. This helps to reduce the latency of operations as compared to the implementation illustrated in timing diagram 200 (of FIG. 2 ).
- encoder 305 encodes slice 0 of frame 0 and then makes the location and size of encoded slice 0 available to bitstream consumer 310 .
- bitstream consumer 310 is able to consume encoded slice 0 while encoder 305 starts to encode slice 1 of frame 0 .
- a drawback of the slice-based approach is the slice-boundary effect which potentially introduces visual artifacts that degrade the user experience.
- the slice-based approach reduces compression efficiency since some coding standards (e.g., H.264) stipulate that prediction cannot be performed across slices. In such cases, each slice of a frame is encoded independent of other slices of the frame.
- the slice-based approach results in extra slice header overhead as each slice will include a header at the beginning of the slice. The negative effects of the slice-based approach are amplified as the number of slices per frame increases.
- processors 405 A-N are representative of any type and number of processors (e.g., CPU, GPU, DSP, FPGA, ASIC).
- processor 405 A includes circuitry to execute video encoder 410 for encoding a video bitstream.
- the circuitry of processor 405 A executes software instructions to implement the functionality of video encoder 410 .
- the circuitry of processor 405 A includes control logic to implement the functionality of video encoder 410 .
- any combination of hardware and/or software of processor 405 A implements the functionality of video encoder 410 .
- processor 405 N includes circuitry to execute consumer module 415 for accessing and processing an encoded video bitstream. In one implementation, the circuitry of processor 405 N executes software instructions to implement the functionality of consumer module 415 . In another implementation, the circuitry of processor 405 N includes control logic to implement the functionality of consumer module 415 . In other implementations, any combination of hardware and/or software of processor 405 N implements the functionality of consumer module 415 . In another implementation, a single processor 405 includes the hardware and/or software to implement both video encoder 410 and consumer module 415 .
- multiple processors 405 A-N include the hardware and/or software to implement video encoder 410 and/or multiple processors 405 A-N include the hardware and/or software to implement consumer module 415 .
- system 400 includes other components and/or is organized in other suitable manners.
- Video encoder 410 includes circuitry for encoding the frames of a video sequence into an encoded video bitstream.
- video encoder 410 stores an encoded bitstream in bitstream buffer 445 within memory device(s) 435 .
- Bitstream buffer 445 is representative of any number and size of buffers and memory device(s) 435 are representative of any number and type of memory devices.
- Memory device(s) 435 are located in any suitable location within system 400 .
- memory device(s) 435 are external to processor 405 A, internal to processor 405 A, implemented as one or more cache(s) of processor 405 A, or otherwise.
- video encoder 410 When video encoder 410 writes a portion of encoded data to bitstream buffer 445 , video encoder 410 updates the value of buffer write pointer 450 to indicate how much data (e.g., how many bits) has been written to bitstream buffer 445 .
- a data ready flag 460 is used to indicate that encoded data has been written to the buffer 445 and is ready for consumption.
- an update to the buffer write pointer 450 is detected and serves to indicate that encoded data has been written to the buffer 445 and is ready for consumption.
- the frequency at which video encoder 410 updates buffer write pointer 450 is programmable.
- consumer module 415 programs the frequency at which video encoder 410 updates buffer write pointer 450 .
- consumer module 415 specifies an update granularity, with the update granularity defined as the number of bits which triggers an update to buffer write pointer 450 .
- video encoder 410 writes a number of bits to bitstream buffer 445 and the number of bits is greater than or equal to the update granularity, then video encoder 410 updates buffer write pointer 450 . Otherwise, if the number of bits written to bitstream buffer 445 is less than the update granularity, then video encoder 410 does not update buffer write pointer 450 .
- a chunk refers to one or more bits.
- a chunk refers to a particular number of bytes of data, while in other implementations a chunk refers to a number of bits that is not evenly divisible into bytes. All such implementations are possible and are contemplated.
- consumer module 415 consumes chunk sizes that match the maximum size of a network transmission packet payload.
- the maximum size of a network transmission packet payload is 1024 bytes, and so consequently, consumer module 415 programs the update granularity to be equal to 1024 bytes (i.e., the chunk size is 8192 bits). In other implementations, the update granularity is set equal to other numbers of bits.
- video encoder 410 When an entire frame has been encoded, video encoder 410 ensures that the bitstream write pointer 450 contains the final bitcount of the entire frame bitstream, even if this final bitcount is less than the granularity programmed by consumer module 415 . To handle the issue of the final bitcount, video encoder 410 sets the frame completion flag when the entire frame has been encoded. Setting the frame completion flag 455 notifies consumer module 415 that the frame encoding is complete and the bitstream write pointer 450 has the final frame bitcount. In various implementations, the consumer module 415 polls the frame completion flag 455 to determine if frame encoding is complete. In other implementations, a signal or other indication is generated to notify the consumer module 415 that data is ready for consumption.
- consumer module 415 is informed that bitstream write pointer 450 will no longer be updated for the frame so consumer module 415 is able to read any remaining bitstream bits even if the number of remaining bitstream bits is less than the specified granularity.
- the frame completion flag 455 is reset by consumer module 415 after consumer module 415 has retrieved all of the bits of bitstream buffer 445 for the current frame.
- bitstream write pointer 450 is determined by the consumer module 415 (e.g., a specific location in GPU memory). In one implementation, a separate bitstream write pointer 450 location is specified for each frame. This prevents consumer module 415 and video encoder 410 from having to synchronize their actions to maintain the state of a single bitstream write pointer 450 if the single bitstream write pointer 450 is used for multiple frames. In some implementations, updates to bitstream write pointer 450 are performed and communicated to the consumer module 415 by other entities besides the video encoder 410 .
- System 500 includes at least video encoder 510 , bitstream buffer 515 , bitstream write pointer (BWP) and frame completion flag 520 , and consumer module 525 .
- Bitstream buffer 515 is representative of any type of buffer or collection of storage elements for storing an encoded video bitstream. Also, bitstream buffer 515 is located within any of various memory devices or memory subsystems. For example, in one implementation, bitstream buffer 515 is stored in a GPU memory. In other implementations, bitstream buffer 515 is stored in other locations.
- An example routine implemented by video encoder 510 and consumer module 525 is shown in FIG. 5 . It is noted that this routine is merely indicative of one implementation. In other implementations, other routines which differ from the example shown in FIG. 5 are possible and are contemplated.
- the example routine begins in step 505 A with the consumer module 525 resetting BWP and frame completion flag 520 at the end of a given frame and before the start of the next frame.
- the video encoder 510 sends write requests with encoded data to bitstream buffer 515 in step 505 B.
- the amount of data in each write request varies depending on the implementation.
- the bitstream buffer 515 sends write request completion acknowledgments (“Acks”) to the video encoder 510 when the data of the write request has been written to the bitstream buffer 515 .
- data for a given frame is encoded as a whole in the sense that each portion of frame data that is encoded has a dependency on one or more other portions frame data.
- This is in contrast to prior art approaches in which slices of a frame are encoded independent of one another.
- encoding slices independent of one another results in boundary artifacts due to the lack of prediction across slice boundaries.
- video encoder 510 updates the value of the BWP in step 505 D to indicate how much encoded data was written to the bitstream buffer 515 .
- circuitry associated with the buffer 515 updates the BWP.
- the video encoder 510 updates the frame completion flag.
- the frame completion flag 455 is a single bit and video encoder 510 sets the frame completion flag to a value of one when the end of a frame has been reached.
- Consumer module 525 reads the BWP and frame completion flag 520 in step 505 E to determine if encoded data is ready for consumption, if encoding of an entire frame is complete, and how far consumer module 525 is able to read from bitstream buffer 515 (how much data is ready for consumption). Then, based on the value of the BWP, consumer module 525 reads a portion of the encoded bitstream from bitstream buffer 515 in step 505 F. In some implementations, a separate data ready flag is used to indicate that encoded data has been written to the buffer 515 and is ready for consumption. For example, by polling the data ready flag the consumer module 525 determines that data is ready for consumption.
- the video encoder 510 sets the data ready flag on receipt of a write request Ack. Upon reading data from the buffer, the consumer module 525 resets the data ready flag. In such a case, the frame completion flag serves to indicate whether or not the data ready for consumption represents the final portion of a frame. It is noted that consumer module 525 is able to read from bitstream buffer 515 and retrieve encoded data for a given frame while video encoder 510 is still encoding the given frame. Also, consumer module 525 does not have to wait until an entire frame or slice has been encoded by the video encoder 510 before reading encoded data from the bitstream buffer 515 . This helps to reduce the latency experienced by consumer module 525 when consuming encoded data from bitstream buffer 515 and also reduces the overall latency of the consumer application.
- FIG. 6 one implementation of a method 600 for enabling on-the-fly access to an encoded video bitstream of a given video frame while the given video frame is being encoded is shown.
- steps in this implementation and those of FIGS. 7-8 are shown in sequential order. However, it is noted that in various implementations of the described methods, one or more of the elements described are performed concurrently, in a different order than shown, or are omitted entirely. Other additional elements are also performed as desired. Any of the various systems or apparatuses described herein are configured to implement method 600 .
- a video encoder starts encoding a given frame of a video sequence (block 605 ). As the video encoder encodes portions of the given frame, the video encoder sends write requests with encoded data to a bitstream buffer (block 610 ). Additionally, the video encoder tracks the completion of write requests of encoded data to the bitstream buffer (block 615 ). If the completion of a write request has been acknowledged (conditional block 620 , “yes” leg), then the video encoder updates a bitstream write pointer with an indication of the amount of data written to the bitstream buffer (block 625 ). Depending on the implementation, the indication specifies an amount of data, a memory address which contains the last portion of data written to the bitstream buffer, or otherwise.
- the update to the bitstream write pointer is performed as an atomic operation to prevent simultaneous access of the bitstream write pointer by both the video encoder and the consumer module. If no acknowledgments have been received for pending write requests (conditional block 620 , “no” leg), then method 600 returns to block 610 .
- the video encoder determines if the entire given frame has been encoded and written to the bitstream buffer (conditional block 630 ). If the entire frame has been encoded and written to the bitstream buffer (conditional block 630 , “yes” leg), then the video encoder sets a frame completion flag (block 635 ). After block 635 , the video encoder waits for the frame completion flag to be cleared (block 640 ), and then method 600 returns to block 605 . In one implementation, the consumer module clears the frame completion flag. In another implementation, another entity, different from the consumer module, clears the frame completion flag.
- method 600 returns to block 610 . It is noted that after the video encoder updates the bitstream write pointer following the completion of a write request, the video encoder does not insert a header (e.g., slice header) into the next portion of data that is encoded and written to the bitstream write buffer. Additionally, the video encoder is able to perform prediction (e.g., intra-frame coding prediction) across the boundary between the last portion of data that was encoded and written to the bitstream writer buffer and the next portion of data that is being encoded.
- a header e.g., slice header
- various encoding techniques encode pixel data using information (coded and/or uncoded) from other pixels in the frame (e.g., neighboring or other pixels).
- information coded and/or uncoded
- the encoded data that crosses boundaries provides a more consistent result with a reduction in artifacts.
- Various such approaches are known to those skilled in the art and all such approaches are contemplated.
- method 700 for a consumer module to access an encoded video bitstream in real-time as the bitstream is being encoded is shown. It is noted that in one implementation, method 700 is performed by the consumer module in parallel with a video encoder performing method 600 .
- a consumer module monitors a bitstream write pointer (block 705 ). For example, in some implementations the consumer module polls the bitstream write pointer to determine if an update has been received. In other implementations, a flag (e.g., a data ready flag) is used to indicate data has been received and is ready for consumption.
- a flag e.g., a data ready flag
- the consumer module performs an atomic read in block 705 to prevent simultaneous access by the video encoder and the consumer module to the bitstream write pointer. If an update to the bitstream write pointer is detected (or the data ready flag is set) (conditional block 710 , “yes” leg), then the consumer module retrieves encoded data from the bitstream buffer up to the location indicated by the bitstream write pointer (block 715 ). It is noted that the consumer module retrieves encoded data from the bitstream buffer for a given frame while the video encoder is still in the process of encoding the given frame. Then, the consumer module processes the encoded data that has been retrieved from the bitstream buffer (block 720 ).
- the consumer module performs any type of processing (e.g., streaming the encoded data over a network) depending on the implementation. If the bitstream write pointer has not been updated (conditional block 710 , “no” leg), then method 700 jumps to block 725 .
- any type of processing e.g., streaming the encoded data over a network
- the consumer module monitors a frame completion flag (block 725 ). If the frame completion flag is set (conditional block 730 , “yes” leg), then the consumer module retrieves a final amount of encoded data from the bitstream buffer even if the final amount is less than a specified granularity (block 735 ). Next, the consumer module clears the frame completion flag (block 740 ). Alternatively, in another implementation, another hardware or software module clears the frame completion flag in block 740 . After block 740 , method 700 returns to block 705 . If the frame completion flag is not set (conditional block 730 , “no” leg), then method 700 returns to block 705 .
- a consumer module programs a granularity setting at which updates to a bitstream writer pointer will be performed (block 805 ). For example, in one implementation, the consumer module programs the granularity setting to be equal to the maximum size of the network transmission packet payload. In other implementations, the consumer module programs the value of the granularity setting based on other factors.
- a video encoder reads the granularity setting as programmed by the consumer module (block 810 ). Then, the video encoder tracks the completion of write requests to the bitstream buffer during the encoding of a video frame (block 815 ). Next, the video encoder determines, based on the completed write requests, if the amount of encoded data (e.g., number of bits) written to the bitstream buffer is greater than or equal to the granularity setting (conditional block 820 ).
- the amount of encoded data e.g., number of bits
- the video encoder If the amount of encoded data written to the bitstream buffer is greater than or equal to the granularity setting (conditional block 820 , “yes” leg), then the video encoder updates a bitstream write pointer with an indication of the amount of encoded data written to the bitstream buffer (block 825 ). Also, the video encoder resets a count used for tracking the amount of encoded data written to the bitstream buffer (block 830 ). After block 830 , method 800 returns to block 815 . Otherwise, if the amount of encoded data written to the bitstream buffer is less than the granularity setting (conditional block 820 , “no” leg), then the video encoder waits for more data to be written to the bitstream buffer (block 835 ). After block 835 , method 800 returns to block 815 .
- a video encoder encodes a given chunk of a video frame (block 905 ).
- a consumer module starts consuming the given chunk of the video frame (block 910 ).
- the video encoder updates a buffer write pointer to let the consumer module know that there is an encoded chunk in a buffer which the consumer module is able to start consuming.
- the video encoder starts encoding a subsequent chunk using intra-frame prediction (block 915 ). It is noted that blocks 910 and 915 are performed simultaneously in one implementation.
- the video encoder finishes encoding the subsequent chunk (block 920 ).
- block 925 the consumer modules starts consuming the subsequent chunk. It is noted that blocks 925 and 915 are performed simultaneously in one implementation. If there are more chunks to encode (conditional block 930 , “yes” leg), then method 900 returns to block 915 . If there are no more chunks to encode (conditional block 930 , “no” leg, then method 900 ends.
- FIG. 10 a timing diagram 1000 of one implementation of real-time consumption of chunks of an encoded video bitstream is shown.
- bitstream consumer 1010 is able to consume a given chunk of the frame while encoder 1005 is encoding the next chunk of the frame. This helps to reduce the latency of operations as compared to the implementations illustrated in timing diagrams 200 (of FIG. 2 ) and 300 (of FIG. 3 ).
- encoder 1005 encodes chunks of frame 0 and then updates a write pointer after each chunk to indicate how much encoded data was written to a buffer.
- Consumer 1010 is then able to access and consume each chunk of frame 0 from the buffer while encoder 1005 is encoding a subsequent chunk. This pattern continues for frames 1 - 2 and subsequent frames.
- the implementation illustrated in timing diagram 1000 helps to minimize latency by allowing the consumption of the encoded frames 0 - 2 to be pipelined with the encoding of frames 0 - 2 .
- bitstream consumer 1010 is able to consume chunks of a frame at the same time that encoder 1005 is encoding subsequent chunks of the frame.
- program instructions of a software application are used to implement the methods and/or mechanisms described herein.
- program instructions executable by a general or special purpose processor are contemplated.
- such program instructions are represented by a high level programming language.
- the program instructions are compiled from a high level programming language to a binary, intermediate, or other form.
- program instructions are written that describe the behavior or design of hardware.
- Such program instructions are represented by a high-level programming language, such as C.
- a hardware design language such as Verilog is used.
- the program instructions are stored on any of a variety of non-transitory computer readable storage mediums. The storage medium is accessible by a computing system during use to provide the program instructions to the computing system for program execution.
- a computing system includes at least one or more memories and one or more processors configured to execute program instructions.
Abstract
Systems, apparatuses, and methods for reducing latency when consuming an encoded video bitstream in real-time are disclosed. A video encoder encodes a video bitstream and writes chunks of the encoded bitstream to a bitstream buffer. Prior to the encoder completing the encoding of an entire frame, or an entire slice of a frame, a consumer module consumes encoded chunks of the bitstream. In one implementation, to enable pipelining of the consumption with the encoding, the encoder updates a buffer write pointer with an indication of the amount of data that has been written to the bitstream buffer. The consumer module retrieves encoded data from the bitstream buffer up to the location indicated by the buffer write pointer. In this way, the consumer module is able to access and consume encoded video data prior to the encoder finishing encoding an entire frame or an entire slice of the frame.
Description
- This application is a continuation of U.S. patent application Ser. No. 15/965,281, now U.S. Pat. No. 11,252,429, entitled “LOW-LATENCY CONSUMPTION OF AN ENCODED VIDEO BITSTREAM”, filed Apr. 27, 2018, the entirety of which is incorporated herein by reference.
- The bandwidth requirements of digital video streaming continue to grow with time. Various applications benefit from video compression which requires less storage space for archived video information and/or less bandwidth for the transmission of the video information. Accordingly, various techniques to improve the quality and accessibility of the digital video have being developed. An example of such a technique is described H.264 which is a video compression standard, proposed by the Joint Video Team (JVT). Many of today's multimedia-enabled digital devices incorporate digital video codec's (hardware and/or software configured to encode and/or decode data) that conform to the H.264 standard. The High Efficiency Video Coding (HEVC) standard is another video compression standard which followed H.264.
- To prepare video for streaming, storage, or additional processing, a hardware accelerator (e.g., video encoder) outputs an encoded bitstream for each video frame. The encoded bitstream is typically written to memory before being consumed by another logical unit (e.g., for network transmission). Video consumption (e.g., streaming) of each encoded video frame is typically initiated after each frame is completely encoded. In other words, a video encoder typically waits until an entire frame has been encoded to confirm that all bitstream memory write operations have been completed, confirm how many bits have been written, and provide a way for the bitstream consumer to determine the number of bits that were written and the location of the bitstream on the storage device. A disadvantage of this approach is that it means consumption of the encoded frame can only begin when the entire frame has been encoded. This prevents frame encoding from being pipelined with other actions such as network transmission, resulting in increased latency.
- The advantages of the methods and mechanisms described herein may be better understood by referring to the following description in conjunction with the accompanying drawings, in which:
-
FIG. 1 is a block diagram of one implementation of a computing system. -
FIG. 2 is a timing diagram of one implementation of the operations for an encoder and a bitstream consumer. -
FIG. 3 is a timing diagram of another implementation of the operations for an encoder and a bitstream consumer. -
FIG. 4 is a block diagram of another implementation of a computing system. -
FIG. 5 is a block diagram of one implementation of a video encoder coupled to a consumer module. -
FIG. 6 is a generalized flow diagram illustrating one implementation of a method for enabling on-the-fly access to an encoded video bitstream of a given video frame while the given video frame is being encoded. -
FIG. 7 is a generalized flow diagram illustrating one implementation of a method for a consumer module accessing an encoded video bitstream in real-time as the bitstream is being encoded. -
FIG. 8 is a generalized flow diagram illustrating one implementation of a method for controlling the granularity of bitstream write pointer updates. -
FIG. 9 is a generalized flow diagram illustrating one implementation of a method for a video encoder performing prediction across chunks being consumed. -
FIG. 10 illustrates a timing diagram of one implementation of real-time consumption of chunks of an encoded video bitstream. - In the following description, numerous specific details are set forth to provide a thorough understanding of the methods and mechanisms presented herein. However, one having ordinary skill in the art should recognize that the various implementations may be practiced without these specific details. In some instances, well-known structures, components, signals, computer program instructions, and techniques have not been shown in detail to avoid obscuring the approaches described herein. It will be appreciated that for simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, the dimensions of some of the elements may be exaggerated relative to other elements.
- Various systems, apparatuses, methods, and computer-readable mediums for enabling low-latency consumption of an encoded video bitstream prior to an entire frame or slice being encoded are disclosed herein. A system includes at least one or more processors coupled to a memory device. The one or more processors include circuitry which executes at least a video encoder and a consumer module. The consumer module is any of various types of logical units which access and consume (e.g., send over a network) the encoded video bitstream in real-time as the video bitstream is encoded by the video encoder. Rather than waiting until an entire frame or slice has been encoded by the video encoder, the consumer module starts to process the encoded bitstream mid-frame and/or mid-slice. It is noted that slice-encoding is typically codec-specific, and any of various types of slice encodings are used in various embodiments. The techniques described herein for on-the-fly consumption of an encoded video bitstream are applicable to full frame encoding or any codec-specific sub-frame encoding (e.g., H-264 slice encoding)
- In one implementation, in order to facilitate the consumption of the encoded bitstream prior to completing the encoding of the entire frame, the video encoder periodically updates a buffer write pointer with an indication of the amount of encoded data written to a video buffer. In one implementation, a granularity at which the video encoder updates the buffer write pointer is programmable. In one implementation, the consumer module programs the granularity at which the video encoder updates the buffer write pointer. In other words, the video encoder is configurable to update the buffer write pointer after writing a particular number of bits of encoded data to the buffer (e.g., 512 bits, 1024 bits, etc.). In the event a last portion of a frame encoded results in fewer than the granularity, the video encoder updates the buffer write pointer and sets the frame completion flag to indicate encoding of the frame is complete. In this case, the chunk has a different size than the above mentioned granularity (i.e., the chunk size is variable).
- In parallel with the video encoder encoding the video bitstream, the consumer module monitors updates to the buffer write pointer. After detecting an update to the buffer write pointer, the consumer module reads from the bitstream buffer up to a location indicated by the buffer write pointer. In addition to the bitcount, the buffer write pointer also includes a frame completion flag to indicate whether or not frame encoding has completed for the current flame. In one implementation, the frame completion flag is set to true (e.g., 1) by the video encoder when frame encoding is completed for the current frame and the buffer write pointer has the final frame bitcount for the current frame. In this implementation, the frame completion flag is reset to false (e.g., 0) by the consumer module after the consumer module has retrieved all of the encoded data from the video buffer for the current frame. The frame completion flag provides a way for the consumer module to determine that the buffer write pointer will no longer be updated for the current frame, allowing the consumer module to read any remaining encoded bitstream data even if the amount of data is less than the specified granularity. In one implementation, frame completion synchronization is implemented separately from buffer write pointer updates.
- Referring now to
FIG. 1 , a block diagram of one implementation of acomputing system 100 is shown. In one implementation,computing system 100 includes at leastprocessors 105A-N, input/output (I/O)interfaces 120,bus 125, memory controller(s) 130,network interface 135, and memory device(s) 140. In other implementations,computing system 100 includes other components and/orcomputing system 100 is arranged differently.Processors 105A-N are representative of any number of processors which are included insystem 100. - In one implementation,
processor 105A is a general purpose processor, such as a central processing unit (CPU). In one implementation,processor 105N is a data parallel processor with a highly parallel architecture. Data parallel processors include graphics processing units (GPUs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and so forth. One or more ofprocessors 105A-N include circuitry to implement a video encoder and a consumer module. The consumer module accesses and consumes an encoded bitstream of a video frame in real-time while the video encoder is encoding the video frame. - Memory controller(s) 130 are representative of any number and type of memory controllers accessible by
processors 105A-N and I/O devices (not shown) coupled to I/O interfaces 120. Memory controller(s) 130 are coupled to any number and type of memory devices(s) 140. Memory device(s) 140 are representative of any number and type of memory devices. For example, the type of memory in memory device(s) 140 includes Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), NAND Flash memory, NOR flash memory, Ferroelectric Random Access Memory (FeRAM), or others. - I/O interfaces 120 are representative of any number and type of I/O interfaces (e.g., peripheral component interconnect (PCI) bus, PCI-Extended (PCI-X), PCIE (PCI Express) bus, gigabit Ethernet (GBE) bus, universal serial bus (USB)). Various types of peripheral devices (not shown) are coupled to I/O interfaces 120. Such peripheral devices include (but are not limited to) displays, keyboards, mice, printers, scanners, joysticks or other types of game controllers, media recording devices, external storage devices, network interface cards, and so forth.
Network interface 135 is used to receive and send network messages across a network. - In various implementations,
computing system 100 is a computer, laptop, mobile device, game console, server, streaming device, wearable device, or any of various other types of computing systems or devices. It is noted that the number of components ofcomputing system 100 varies from implementation to implementation. For example, there can be more or fewer of each component than the number shown inFIG. 1 . It is also noted thatcomputing system 100 can include other components not shown inFIG. 1 . Additionally, in other implementations,computing system 100 can be structured in other ways than shown inFIG. 1 . - Turning now to
FIG. 2 , a timing diagram 200 of one implementation of the operations for anencoder 205 and abitstream consumer 210. In one implementation, anencoder 205 encodes individual frames 0-2 of a video stream and makes each frame available tobitstream consumer 210 at the end of each frame. For example,encoder 205 confirms bitstream memory write operations have completed, confirms how many bits have been written to the bitstream buffer, and then provides a way for the bitstream consumer to determine the size and location of the encoded bitstream in memory. It is noted thatbitstream consumer 210 can also be referred to as a “consumer module”.Encoder 205 encodes the video stream in accordance with any of various coding standards. For example, in one implementation,encoder 205 encodes a bitstream to be compliant with the H.264 video compression standard. In another implementation,encoder 205 encodes a bitstream to be compliant with the High Efficiency Video Coding (HEVC) standard. In other implementations,encoder 205 encodes a bitstream to be compliant with other standards. - Accordingly, as shown in timing diagram 200,
bitstream consumer 210 is only able to start consuming a given frame afterencoder 205 has encoded the entire frame. For example, only afterframe 0 has been completely encoded byencoder 205 isbitstream consumer 210 able to start consuming the encoded bitstream generated forframe 0. The same sequence of events occurs for frames 1-2. This results in a delay from the time whenencoder 205 starts encoding a frame and to the time whenbitstream consumer 210 is able to start consuming the encoded frame. - Referring now to
FIG. 3 , a timing diagram 300 of another implementation of the operations for anencoder 305 and abitstream consumer 310 is shown. In the implementation illustrated in timing diagram 300,encoder 305 encodes slices of a frame and thenbitstream consumer 310 is able to consume an individual slice afterencoder 305 has finished encoding the given slice. This helps to reduce the latency of operations as compared to the implementation illustrated in timing diagram 200 (ofFIG. 2 ). For example,encoder 305 encodesslice 0 offrame 0 and then makes the location and size of encodedslice 0 available tobitstream consumer 310. Then,bitstream consumer 310 is able to consume encodedslice 0 whileencoder 305 starts to encodeslice 1 offrame 0. This pattern continues for slices 1-2 offrame 0 and the subsequent slices offrame 0. As the content and structure of slices are typically defined by a compression standard, codecs used to generate such slices conform to the corresponding standard. As used herein, slices are also referred to herein as “codec-level slices”. - While the implementation illustrated in timing diagram 300 results in reduced latency compared to the implementation illustrated in timing diagram 200, a drawback of the slice-based approach is the slice-boundary effect which potentially introduces visual artifacts that degrade the user experience. Furthermore, the slice-based approach reduces compression efficiency since some coding standards (e.g., H.264) stipulate that prediction cannot be performed across slices. In such cases, each slice of a frame is encoded independent of other slices of the frame. In addition, the slice-based approach results in extra slice header overhead as each slice will include a header at the beginning of the slice. The negative effects of the slice-based approach are amplified as the number of slices per frame increases.
- Turning now to
FIG. 4 , a block diagram of another implementation of acomputing system 400 is shown. In one implementation,computer system 400 includesprocessors 405A-N, I/O interfaces 420,bus 425, memory controller(s) 430, memory device(s) 435, andnetwork interface 440.Processors 405A-N are representative of any type and number of processors (e.g., CPU, GPU, DSP, FPGA, ASIC). In one implementation,processor 405A includes circuitry to executevideo encoder 410 for encoding a video bitstream. In one implementation, the circuitry ofprocessor 405A executes software instructions to implement the functionality ofvideo encoder 410. In another implementation, the circuitry ofprocessor 405A includes control logic to implement the functionality ofvideo encoder 410. In other implementations, any combination of hardware and/or software ofprocessor 405A implements the functionality ofvideo encoder 410. - In one implementation,
processor 405N includes circuitry to executeconsumer module 415 for accessing and processing an encoded video bitstream. In one implementation, the circuitry ofprocessor 405N executes software instructions to implement the functionality ofconsumer module 415. In another implementation, the circuitry ofprocessor 405N includes control logic to implement the functionality ofconsumer module 415. In other implementations, any combination of hardware and/or software ofprocessor 405N implements the functionality ofconsumer module 415. In another implementation, a single processor 405 includes the hardware and/or software to implement bothvideo encoder 410 andconsumer module 415. In a further implementation,multiple processors 405A-N include the hardware and/or software to implementvideo encoder 410 and/ormultiple processors 405A-N include the hardware and/or software to implementconsumer module 415. It should be understood that in other implementations,system 400 includes other components and/or is organized in other suitable manners. -
Video encoder 410 includes circuitry for encoding the frames of a video sequence into an encoded video bitstream. In one implementation,video encoder 410 stores an encoded bitstream inbitstream buffer 445 within memory device(s) 435.Bitstream buffer 445 is representative of any number and size of buffers and memory device(s) 435 are representative of any number and type of memory devices. Memory device(s) 435 are located in any suitable location withinsystem 400. For example, in various embodiments, memory device(s) 435 are external toprocessor 405A, internal toprocessor 405A, implemented as one or more cache(s) ofprocessor 405A, or otherwise. Whenvideo encoder 410 writes a portion of encoded data tobitstream buffer 445,video encoder 410 updates the value ofbuffer write pointer 450 to indicate how much data (e.g., how many bits) has been written tobitstream buffer 445. In some implementations, a dataready flag 460 is used to indicate that encoded data has been written to thebuffer 445 and is ready for consumption. In other implementations, an update to thebuffer write pointer 450 is detected and serves to indicate that encoded data has been written to thebuffer 445 and is ready for consumption. - In one implementation, the frequency at which
video encoder 410 updatesbuffer write pointer 450 is programmable. For example, in one implementation,consumer module 415 programs the frequency at whichvideo encoder 410 updatesbuffer write pointer 450. In other words,consumer module 415 specifies an update granularity, with the update granularity defined as the number of bits which triggers an update to bufferwrite pointer 450. Whenvideo encoder 410 writes a number of bits tobitstream buffer 445 and the number of bits is greater than or equal to the update granularity, thenvideo encoder 410 updatesbuffer write pointer 450. Otherwise, if the number of bits written tobitstream buffer 445 is less than the update granularity, thenvideo encoder 410 does not updatebuffer write pointer 450. This helps avoid unnecessary memory transactions for updatingbuffer write pointer 450 if theconsumer module 415 only consumes a particular size of bitstream chunk at a time. As used herein, a “chunk” refers to one or more bits. In some implementations a chunk refers to a particular number of bytes of data, while in other implementations a chunk refers to a number of bits that is not evenly divisible into bytes. All such implementations are possible and are contemplated. For example, in one implementation,consumer module 415 consumes chunk sizes that match the maximum size of a network transmission packet payload. In one implementation, the maximum size of a network transmission packet payload is 1024 bytes, and so consequently,consumer module 415 programs the update granularity to be equal to 1024 bytes (i.e., the chunk size is 8192 bits). In other implementations, the update granularity is set equal to other numbers of bits. - When an entire frame has been encoded,
video encoder 410 ensures that thebitstream write pointer 450 contains the final bitcount of the entire frame bitstream, even if this final bitcount is less than the granularity programmed byconsumer module 415. To handle the issue of the final bitcount,video encoder 410 sets the frame completion flag when the entire frame has been encoded. Setting theframe completion flag 455 notifiesconsumer module 415 that the frame encoding is complete and thebitstream write pointer 450 has the final frame bitcount. In various implementations, theconsumer module 415 polls theframe completion flag 455 to determine if frame encoding is complete. In other implementations, a signal or other indication is generated to notify theconsumer module 415 that data is ready for consumption. In this way,consumer module 415 is informed thatbitstream write pointer 450 will no longer be updated for the frame soconsumer module 415 is able to read any remaining bitstream bits even if the number of remaining bitstream bits is less than the specified granularity. Theframe completion flag 455 is reset byconsumer module 415 afterconsumer module 415 has retrieved all of the bits ofbitstream buffer 445 for the current frame. - In one implementation, the storage location of
bitstream write pointer 450 is determined by the consumer module 415 (e.g., a specific location in GPU memory). In one implementation, a separatebitstream write pointer 450 location is specified for each frame. This preventsconsumer module 415 andvideo encoder 410 from having to synchronize their actions to maintain the state of a singlebitstream write pointer 450 if the singlebitstream write pointer 450 is used for multiple frames. In some implementations, updates tobitstream write pointer 450 are performed and communicated to theconsumer module 415 by other entities besides thevideo encoder 410. - Referring now to
FIG. 5 , a block diagram of one implementation of acomputing system 500 having a video encoder coupled to a consumer module is shown.System 500 includes atleast video encoder 510,bitstream buffer 515, bitstream write pointer (BWP) andframe completion flag 520, andconsumer module 525.Bitstream buffer 515 is representative of any type of buffer or collection of storage elements for storing an encoded video bitstream. Also,bitstream buffer 515 is located within any of various memory devices or memory subsystems. For example, in one implementation,bitstream buffer 515 is stored in a GPU memory. In other implementations,bitstream buffer 515 is stored in other locations. An example routine implemented byvideo encoder 510 andconsumer module 525 is shown inFIG. 5 . It is noted that this routine is merely indicative of one implementation. In other implementations, other routines which differ from the example shown inFIG. 5 are possible and are contemplated. - In one implementation, the example routine begins in
step 505A with theconsumer module 525 resetting BWP andframe completion flag 520 at the end of a given frame and before the start of the next frame. As thevideo encoder 510 starts encoding the next frame, thevideo encoder 510 sends write requests with encoded data tobitstream buffer 515 instep 505B. The amount of data in each write request varies depending on the implementation. In response to thevideo encoder 510 sending write requests tobitstream buffer 515, thebitstream buffer 515 sends write request completion acknowledgments (“Acks”) to thevideo encoder 510 when the data of the write request has been written to thebitstream buffer 515. In various implementations, data for a given frame is encoded as a whole in the sense that each portion of frame data that is encoded has a dependency on one or more other portions frame data. This is in contrast to prior art approaches in which slices of a frame are encoded independent of one another. As noted above, encoding slices independent of one another results in boundary artifacts due to the lack of prediction across slice boundaries. By encoding the entire frame data as a whole, even though portions of encoded frame data are readied for consumption during the encoding process, such boundary artifacts are eliminated. - In response to receiving a write request completion Ack in step 505C,
video encoder 510 updates the value of the BWP in step 505D to indicate how much encoded data was written to thebitstream buffer 515. Alternatively, in some implementations, circuitry associated with thebuffer 515 updates the BWP. Also, when thevideo encoder 510 has completed encoding an entire frame, thevideo encoder 510 updates the frame completion flag. In one implementation, theframe completion flag 455 is a single bit andvideo encoder 510 sets the frame completion flag to a value of one when the end of a frame has been reached.Consumer module 525 reads the BWP andframe completion flag 520 in step 505E to determine if encoded data is ready for consumption, if encoding of an entire frame is complete, and howfar consumer module 525 is able to read from bitstream buffer 515 (how much data is ready for consumption). Then, based on the value of the BWP,consumer module 525 reads a portion of the encoded bitstream frombitstream buffer 515 instep 505F. In some implementations, a separate data ready flag is used to indicate that encoded data has been written to thebuffer 515 and is ready for consumption. For example, by polling the data ready flag theconsumer module 525 determines that data is ready for consumption. For example, thevideo encoder 510 sets the data ready flag on receipt of a write request Ack. Upon reading data from the buffer, theconsumer module 525 resets the data ready flag. In such a case, the frame completion flag serves to indicate whether or not the data ready for consumption represents the final portion of a frame. It is noted thatconsumer module 525 is able to read frombitstream buffer 515 and retrieve encoded data for a given frame whilevideo encoder 510 is still encoding the given frame. Also,consumer module 525 does not have to wait until an entire frame or slice has been encoded by thevideo encoder 510 before reading encoded data from thebitstream buffer 515. This helps to reduce the latency experienced byconsumer module 525 when consuming encoded data frombitstream buffer 515 and also reduces the overall latency of the consumer application. - Turning now to
FIG. 6 , one implementation of amethod 600 for enabling on-the-fly access to an encoded video bitstream of a given video frame while the given video frame is being encoded is shown. For purposes of discussion, the steps in this implementation and those ofFIGS. 7-8 are shown in sequential order. However, it is noted that in various implementations of the described methods, one or more of the elements described are performed concurrently, in a different order than shown, or are omitted entirely. Other additional elements are also performed as desired. Any of the various systems or apparatuses described herein are configured to implementmethod 600. - A video encoder starts encoding a given frame of a video sequence (block 605). As the video encoder encodes portions of the given frame, the video encoder sends write requests with encoded data to a bitstream buffer (block 610). Additionally, the video encoder tracks the completion of write requests of encoded data to the bitstream buffer (block 615). If the completion of a write request has been acknowledged (
conditional block 620, “yes” leg), then the video encoder updates a bitstream write pointer with an indication of the amount of data written to the bitstream buffer (block 625). Depending on the implementation, the indication specifies an amount of data, a memory address which contains the last portion of data written to the bitstream buffer, or otherwise. It is noted that in one implementation, the update to the bitstream write pointer is performed as an atomic operation to prevent simultaneous access of the bitstream write pointer by both the video encoder and the consumer module. If no acknowledgments have been received for pending write requests (conditional block 620, “no” leg), thenmethod 600 returns to block 610. - After
block 625, the video encoder determines if the entire given frame has been encoded and written to the bitstream buffer (conditional block 630). If the entire frame has been encoded and written to the bitstream buffer (conditional block 630, “yes” leg), then the video encoder sets a frame completion flag (block 635). Afterblock 635, the video encoder waits for the frame completion flag to be cleared (block 640), and thenmethod 600 returns to block 605. In one implementation, the consumer module clears the frame completion flag. In another implementation, another entity, different from the consumer module, clears the frame completion flag. If the entire frame has not been encoded and written to the bitstream buffer (conditional block 630, “no” leg), thenmethod 600 returns to block 610. It is noted that after the video encoder updates the bitstream write pointer following the completion of a write request, the video encoder does not insert a header (e.g., slice header) into the next portion of data that is encoded and written to the bitstream write buffer. Additionally, the video encoder is able to perform prediction (e.g., intra-frame coding prediction) across the boundary between the last portion of data that was encoded and written to the bitstream writer buffer and the next portion of data that is being encoded. For example, various encoding techniques encode pixel data using information (coded and/or uncoded) from other pixels in the frame (e.g., neighboring or other pixels). In this manner, the encoded data that crosses boundaries (slice, chunk, or otherwise) provides a more consistent result with a reduction in artifacts. Various such approaches are known to those skilled in the art and all such approaches are contemplated. - Referring now to
FIG. 7 , one implementation of amethod 700 for a consumer module to access an encoded video bitstream in real-time as the bitstream is being encoded is shown. It is noted that in one implementation,method 700 is performed by the consumer module in parallel with a videoencoder performing method 600. A consumer module monitors a bitstream write pointer (block 705). For example, in some implementations the consumer module polls the bitstream write pointer to determine if an update has been received. In other implementations, a flag (e.g., a data ready flag) is used to indicate data has been received and is ready for consumption. It is noted that in one implementation, the consumer module performs an atomic read inblock 705 to prevent simultaneous access by the video encoder and the consumer module to the bitstream write pointer. If an update to the bitstream write pointer is detected (or the data ready flag is set) (conditional block 710, “yes” leg), then the consumer module retrieves encoded data from the bitstream buffer up to the location indicated by the bitstream write pointer (block 715). It is noted that the consumer module retrieves encoded data from the bitstream buffer for a given frame while the video encoder is still in the process of encoding the given frame. Then, the consumer module processes the encoded data that has been retrieved from the bitstream buffer (block 720). The consumer module performs any type of processing (e.g., streaming the encoded data over a network) depending on the implementation. If the bitstream write pointer has not been updated (conditional block 710, “no” leg), thenmethod 700 jumps to block 725. - After
block 720, the consumer module monitors a frame completion flag (block 725). If the frame completion flag is set (conditional block 730, “yes” leg), then the consumer module retrieves a final amount of encoded data from the bitstream buffer even if the final amount is less than a specified granularity (block 735). Next, the consumer module clears the frame completion flag (block 740). Alternatively, in another implementation, another hardware or software module clears the frame completion flag inblock 740. Afterblock 740,method 700 returns to block 705. If the frame completion flag is not set (conditional block 730, “no” leg), thenmethod 700 returns to block 705. - Turning now to
FIG. 8 , one implementation of amethod 800 for controlling the granularity of bitstream write pointer updates is shown. A consumer module programs a granularity setting at which updates to a bitstream writer pointer will be performed (block 805). For example, in one implementation, the consumer module programs the granularity setting to be equal to the maximum size of the network transmission packet payload. In other implementations, the consumer module programs the value of the granularity setting based on other factors. Next, a video encoder reads the granularity setting as programmed by the consumer module (block 810). Then, the video encoder tracks the completion of write requests to the bitstream buffer during the encoding of a video frame (block 815). Next, the video encoder determines, based on the completed write requests, if the amount of encoded data (e.g., number of bits) written to the bitstream buffer is greater than or equal to the granularity setting (conditional block 820). - If the amount of encoded data written to the bitstream buffer is greater than or equal to the granularity setting (
conditional block 820, “yes” leg), then the video encoder updates a bitstream write pointer with an indication of the amount of encoded data written to the bitstream buffer (block 825). Also, the video encoder resets a count used for tracking the amount of encoded data written to the bitstream buffer (block 830). Afterblock 830,method 800 returns to block 815. Otherwise, if the amount of encoded data written to the bitstream buffer is less than the granularity setting (conditional block 820, “no” leg), then the video encoder waits for more data to be written to the bitstream buffer (block 835). Afterblock 835,method 800 returns to block 815. - Referring now to
FIG. 9 , one implementation of amethod 900 for a video encoder performing prediction across chunks being consumed is shown. A video encoder encodes a given chunk of a video frame (block 905). Next, a consumer module starts consuming the given chunk of the video frame (block 910). In one implementation, the video encoder updates a buffer write pointer to let the consumer module know that there is an encoded chunk in a buffer which the consumer module is able to start consuming. Then, the video encoder starts encoding a subsequent chunk using intra-frame prediction (block 915). It is noted thatblocks blocks conditional block 930, “yes” leg), thenmethod 900 returns to block 915. If there are no more chunks to encode (conditional block 930, “no” leg, thenmethod 900 ends. - Turning now to
FIG. 10 , a timing diagram 1000 of one implementation of real-time consumption of chunks of an encoded video bitstream is shown. In the implementation illustrated in timing diagram 1000, whileencoder 1005 is encoding frames 0-2,bitstream consumer 1010 is able to consume a given chunk of the frame whileencoder 1005 is encoding the next chunk of the frame. This helps to reduce the latency of operations as compared to the implementations illustrated in timing diagrams 200 (ofFIG. 2 ) and 300 (ofFIG. 3 ). For example, in one implementation,encoder 1005 encodes chunks offrame 0 and then updates a write pointer after each chunk to indicate how much encoded data was written to a buffer.Consumer 1010 is then able to access and consume each chunk offrame 0 from the buffer whileencoder 1005 is encoding a subsequent chunk. This pattern continues for frames 1-2 and subsequent frames. The implementation illustrated in timing diagram 1000 helps to minimize latency by allowing the consumption of the encoded frames 0-2 to be pipelined with the encoding of frames 0-2. In this implementation,bitstream consumer 1010 is able to consume chunks of a frame at the same time that encoder 1005 is encoding subsequent chunks of the frame. - In various implementations, program instructions of a software application are used to implement the methods and/or mechanisms described herein. For example, program instructions executable by a general or special purpose processor are contemplated. In various implementations, such program instructions are represented by a high level programming language. In other implementations, the program instructions are compiled from a high level programming language to a binary, intermediate, or other form. Alternatively, program instructions are written that describe the behavior or design of hardware. Such program instructions are represented by a high-level programming language, such as C. Alternatively, a hardware design language (HDL) such as Verilog is used. In various implementations, the program instructions are stored on any of a variety of non-transitory computer readable storage mediums. The storage medium is accessible by a computing system during use to provide the program instructions to the computing system for program execution. Generally speaking, such a computing system includes at least one or more memories and one or more processors configured to execute program instructions.
- It should be emphasized that the above-described implementations are only non-limiting examples of implementations. Numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to embrace all such variations and modifications.
Claims (20)
1. A system comprising:
a video encoder comprising circuitry configured to:
encode a first portion of a given video frame to generate a first chunk of encoded data, the first portion comprising less than an entirety of the given video frame;
write the first chunk of encoded data to a buffer; and
provide an indication that the first chunk of encoded data is ready for consumption; and
a consumer module comprising circuitry configured to retrieve the first chunk of encoded data from the buffer prior to the video encoder completing encoding of the entirety of the given video frame, in response to detecting said indication.
2. The system as recited in claim 1 , wherein the circuitry of the video encoder is further configured to provide said indication prior to encoding the entirety of the given video frame.
3. The system as recited in claim 1 , wherein the circuitry of the video encoder is further configured to encode all chunks of the given video frame using intra-frame coding prediction.
4. The system as recited in claim 1 , wherein the circuitry of the video encoder is further configured to provide an indication as to how much data is included in the first chunk.
5. The system as recited in claim 1 , wherein a granularity at which the video encoder provides said indication.
6. The system as recited in claim 4 , wherein:
the circuitry of the video encoder is configured to set a frame completion flag when encoding of the given video frame is complete; and
the circuitry of the consumer module is configured to retrieve a final amount of encoded data from the buffer when the final amount is less than the granularity, responsive to detecting the frame completion flag is set.
7. The system as recited in claim 6 , wherein the circuitry of the consumer module is configured to clear the frame completion flag responsive to retrieving the final amount of encoded data from the buffer.
8. A method comprising:
encoding, by circuitry of a video encoder, a first portion of a given video frame to generate a first chunk of encoded data, the first portion comprising less than an entirety of the given video frame;
writing, by the circuitry of the video encoder, the first chunk of encoded data to a buffer;
providing, by the circuitry of the video encoder, an indication that the first chunk of encoded data is ready for consumption;
retrieving, by circuitry of a consumer module, the first chunk of encoded data from the buffer prior to the video encoder completing encoding of the entirety of the given video frame, in response to detecting said indication.
9. The method as recited in claim 8 , further comprising providing, by the circuitry of the video encoder, said indication prior to encoding the entirety of the given video frame.
10. The method as recited in claim 8 , further comprising encoding all chunks of the given video frame using intra-frame coding prediction.
11. The method as recited in claim 8 , further comprising providing an indication as to how much data is included in the first chunk.
12. The method as recited in claim 11 , further comprising programming, by the circuitry of the consumer module, a granularity at which the video encoder provides said indication.
13. The method as recited in claim 11 , further comprising:
setting, by the circuitry of the video encoder, a frame completion flag when encoding of the given video frame is complete; and
retrieving, by the circuitry of the consumer module, a final amount of encoded data from the buffer when the final amount is less than the granularity, responsive to detecting the frame completion flag is set.
14. The method as recited in claim 13 , further comprising clearing, by the circuitry of the consumer module, the frame completion flag responsive to retrieving the final amount of encoded data from the buffer.
15. An apparatus comprising:
a memory;
a buffer; and
one or more processors coupled to the memory;
wherein the apparatus is configured to:
read given video frame data from the memory;
encode a first portion of the given video frame to generate a first chunk of encoded data, the first portion comprising less than an entirety of the given video frame;
write the first chunk of encoded data to the buffer; and
provide an indication that the first chunk of encoded data is ready for consumption; and
retrieve the first chunk of encoded data from the buffer prior to the video encoder completing encoding of the entirety of the given video frame, in response to detecting said indication.
16. The apparatus as recited in claim 15 , wherein the apparatus is further configured to provide said indication prior to encoding the entirety of the given video frame.
17. The apparatus as recited in claim 15 , wherein the apparatus is further configured to encode all chunks of the given video frame using intra-frame coding prediction.
18. The apparatus as recited in claim 17 , wherein the apparatus is further configured to provide an indication as to how much data is included in the first chunk.
19. The apparatus as recited in claim 17 , wherein the apparatus is further configured to:
set a frame completion flag when the encoding of the given video frame is complete; and
retrieve a final amount of encoded data from the buffer when the final amount is less than the granularity, responsive to the frame completion flag being set.
20. The apparatus as recited in claim 19 , wherein the apparatus is further configured to clear the frame completion flag responsive to retrieving the final amount of encoded data from the buffer.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/587,497 US20220159286A1 (en) | 2018-04-27 | 2022-01-28 | Low-latency consumption of an encoded video bitstream |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/965,281 US11252429B2 (en) | 2018-04-27 | 2018-04-27 | Low-latency consumption of an encoded video bitstream |
US17/587,497 US20220159286A1 (en) | 2018-04-27 | 2022-01-28 | Low-latency consumption of an encoded video bitstream |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/965,281 Continuation US11252429B2 (en) | 2018-04-27 | 2018-04-27 | Low-latency consumption of an encoded video bitstream |
Publications (1)
Publication Number | Publication Date |
---|---|
US20220159286A1 true US20220159286A1 (en) | 2022-05-19 |
Family
ID=68291381
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/965,281 Active US11252429B2 (en) | 2018-04-27 | 2018-04-27 | Low-latency consumption of an encoded video bitstream |
US17/587,497 Pending US20220159286A1 (en) | 2018-04-27 | 2022-01-28 | Low-latency consumption of an encoded video bitstream |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/965,281 Active US11252429B2 (en) | 2018-04-27 | 2018-04-27 | Low-latency consumption of an encoded video bitstream |
Country Status (6)
Country | Link |
---|---|
US (2) | US11252429B2 (en) |
EP (1) | EP3785438A4 (en) |
JP (2) | JP7336459B2 (en) |
KR (2) | KR102546236B1 (en) |
CN (2) | CN117201812A (en) |
WO (1) | WO2019207368A1 (en) |
Families Citing this family (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11076158B2 (en) * | 2019-09-09 | 2021-07-27 | Facebook Technologies, Llc | Systems and methods for reducing WiFi latency using transmit opportunity and duration |
US11843772B2 (en) | 2019-12-06 | 2023-12-12 | Ati Technologies Ulc | Video encode pre-analysis bit budgeting based on context and features |
US11070829B2 (en) | 2019-12-16 | 2021-07-20 | Ati Technologies Ulc | Reducing latency in wireless virtual and augmented reality systems |
US11714620B1 (en) | 2022-01-14 | 2023-08-01 | Triad National Security, Llc | Decoupling loop dependencies using buffers to enable pipelining of loops |
KR102632507B1 (en) * | 2023-06-20 | 2024-01-31 | 쿠팡 주식회사 | Method and system for processing video for streaming service |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5592227A (en) * | 1994-09-15 | 1997-01-07 | Vcom, Inc. | Method and apparatus for compressing a digital signal using vector quantization |
US6535251B1 (en) * | 1999-10-26 | 2003-03-18 | Sharplabs Of America, Inc. | Video encoder and method for adjusting quantization step in real time |
US6654539B1 (en) * | 1998-10-26 | 2003-11-25 | Sony Corporation | Trick playback of digital video data |
US20110194439A1 (en) * | 2010-02-09 | 2011-08-11 | Canon Kabushiki Kaisha | Method and device for computing the available space in a packet for data stream transport |
US20140176586A1 (en) * | 2012-12-21 | 2014-06-26 | Qualcomm Incorporated | Multi-mode memory access techniques for performing graphics processing unit-based memory transfer operations |
US20150195379A1 (en) * | 2014-01-03 | 2015-07-09 | Yahoo! Inc. | Systems and methods for content delivery |
US20150281705A1 (en) * | 2014-03-27 | 2015-10-01 | Arris Enterprises, Inc. | Method and apparatus for encoding rate control in advanced coding schemes |
Family Cites Families (21)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5768533A (en) | 1995-09-01 | 1998-06-16 | National Semiconductor Corporation | Video coding using segmented frames and retransmission to overcome channel errors |
US6067287A (en) | 1997-09-15 | 2000-05-23 | Accton Technology | Neural fuzzy connection admission controller and method in a node of an asynchronous transfer mode (ATM) communication network |
US7535485B2 (en) | 2000-08-15 | 2009-05-19 | Polycom, Inc. | Delay reduction for transmission and processing of video data |
US7885337B2 (en) | 2004-08-23 | 2011-02-08 | Qualcomm Incorporated | Efficient video slicing |
US8131660B2 (en) | 2008-04-08 | 2012-03-06 | Microsoft Corporation | Reconfigurable hardware accelerator for boolean satisfiability solver |
JP5750899B2 (en) * | 2011-01-17 | 2015-07-22 | 株式会社リコー | Image forming apparatus and control method thereof |
BR122020007529B1 (en) * | 2012-01-20 | 2021-09-21 | Ge Video Compression, Llc | CODING CONCEPT ALLOWING PARALLEL PROCESSING, TRANSPORT DEMULTIPLEXER AND VIDEO BITE FLOW |
JP6214235B2 (en) * | 2012-07-02 | 2017-10-18 | キヤノン株式会社 | File generation method, file generation apparatus, and program |
PL4002850T3 (en) * | 2012-08-09 | 2023-05-08 | Sun Patent Trust | Image coding method and apparatus |
US9510008B2 (en) | 2013-11-25 | 2016-11-29 | Entropic Communications, Llc | Video decoder memory bandwidth compression |
KR101895176B1 (en) * | 2014-06-18 | 2018-09-04 | 텔레폰악티에볼라겟엘엠에릭슨(펍) | Dependent random access point pictures |
US9800898B2 (en) | 2014-10-06 | 2017-10-24 | Microsoft Technology Licensing, Llc | Syntax structures indicating completion of coded regions |
US10417555B2 (en) | 2015-05-29 | 2019-09-17 | Samsung Electronics Co., Ltd. | Data-optimized neural network traversal |
US20160379109A1 (en) | 2015-06-29 | 2016-12-29 | Microsoft Technology Licensing, Llc | Convolutional neural networks on hardware accelerators |
US10003811B2 (en) | 2015-09-01 | 2018-06-19 | Microsoft Technology Licensing, Llc | Parallel processing of a video frame |
WO2017095948A1 (en) | 2015-11-30 | 2017-06-08 | Pilot Ai Labs, Inc. | Improved general object detection using neural networks |
EP3531700B1 (en) * | 2016-10-18 | 2022-04-06 | Panasonic Intellectual Property Management Co., Ltd. | Image coding method, transmission method and image coding device |
US11562115B2 (en) | 2017-01-04 | 2023-01-24 | Stmicroelectronics S.R.L. | Configurable accelerator framework including a stream switch having a plurality of unidirectional stream links |
US10582250B2 (en) | 2017-07-24 | 2020-03-03 | Advanced Micro Devices, Inc. | Integrated video codec and inference engine |
US11227214B2 (en) | 2017-11-14 | 2022-01-18 | Advanced Micro Devices, Inc. | Memory bandwidth reduction techniques for low power convolutional neural network inference applications |
US11948073B2 (en) | 2018-04-20 | 2024-04-02 | Advanced Micro Devices, Inc. | Machine learning inference engine scalability |
-
2018
- 2018-04-27 US US15/965,281 patent/US11252429B2/en active Active
-
2019
- 2019-02-26 CN CN202310984323.8A patent/CN117201812A/en active Pending
- 2019-02-26 CN CN201980028611.8A patent/CN112055969B/en active Active
- 2019-02-26 KR KR1020207034305A patent/KR102546236B1/en active IP Right Grant
- 2019-02-26 JP JP2020560303A patent/JP7336459B2/en active Active
- 2019-02-26 EP EP19793999.4A patent/EP3785438A4/en active Pending
- 2019-02-26 WO PCT/IB2019/051542 patent/WO2019207368A1/en active Application Filing
- 2019-02-26 KR KR1020237020467A patent/KR20230097207A/en not_active Application Discontinuation
-
2022
- 2022-01-28 US US17/587,497 patent/US20220159286A1/en active Pending
-
2023
- 2023-08-21 JP JP2023134181A patent/JP2023156488A/en active Pending
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5592227A (en) * | 1994-09-15 | 1997-01-07 | Vcom, Inc. | Method and apparatus for compressing a digital signal using vector quantization |
US6654539B1 (en) * | 1998-10-26 | 2003-11-25 | Sony Corporation | Trick playback of digital video data |
US6535251B1 (en) * | 1999-10-26 | 2003-03-18 | Sharplabs Of America, Inc. | Video encoder and method for adjusting quantization step in real time |
US20110194439A1 (en) * | 2010-02-09 | 2011-08-11 | Canon Kabushiki Kaisha | Method and device for computing the available space in a packet for data stream transport |
US20140176586A1 (en) * | 2012-12-21 | 2014-06-26 | Qualcomm Incorporated | Multi-mode memory access techniques for performing graphics processing unit-based memory transfer operations |
US20150195379A1 (en) * | 2014-01-03 | 2015-07-09 | Yahoo! Inc. | Systems and methods for content delivery |
US20150281705A1 (en) * | 2014-03-27 | 2015-10-01 | Arris Enterprises, Inc. | Method and apparatus for encoding rate control in advanced coding schemes |
Non-Patent Citations (1)
Title |
---|
Hoffman, D., Fernando, G., Goyal, V., and M. Civanlar, "RTP Payload Format for MPEG1/MPEG2 Video", RFC 2250, DOI 10.17487/RFC2250, January 1998, <https://www.rfc-editor.org/info/rfc2250> (Year: 1998) * |
Also Published As
Publication number | Publication date |
---|---|
US11252429B2 (en) | 2022-02-15 |
KR20230097207A (en) | 2023-06-30 |
CN112055969B (en) | 2023-08-29 |
CN117201812A (en) | 2023-12-08 |
KR20210002677A (en) | 2021-01-08 |
EP3785438A1 (en) | 2021-03-03 |
JP2023156488A (en) | 2023-10-24 |
US20190335189A1 (en) | 2019-10-31 |
CN112055969A (en) | 2020-12-08 |
KR102546236B1 (en) | 2023-06-21 |
JP2021522729A (en) | 2021-08-30 |
JP7336459B2 (en) | 2023-08-31 |
WO2019207368A1 (en) | 2019-10-31 |
EP3785438A4 (en) | 2022-06-08 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20220159286A1 (en) | Low-latency consumption of an encoded video bitstream | |
JP4991816B2 (en) | Image processing device | |
KR102596932B1 (en) | GPU parallel Huffman decoding | |
US11175854B2 (en) | Data processing systems | |
US10032246B2 (en) | Approach to caching decoded texture data with variable dimensions | |
US10116943B2 (en) | Adaptive video compression for latency control | |
US10276125B2 (en) | Method of and apparatus for controlling overrun when writing data from a display controller to memory | |
US11093323B2 (en) | Performant inline ECC architecture for DRAM controller | |
US20160335734A1 (en) | Memory subsystem synchronization primitives | |
KR102619668B1 (en) | Apparatus and method of using a slice update map | |
US10448020B2 (en) | Intelligent MSI-X interrupts for video analytics and encoding | |
US20150229921A1 (en) | Intra searches using inaccurate neighboring pixel data | |
US20150070372A1 (en) | Image processing apparatus and a method of storing encoded data blocks generated by such an image processing apparatus | |
US9986159B2 (en) | Technique for reducing the power consumption for a video encoder engine | |
US11418212B2 (en) | Parallel decoding techniques | |
US10558606B1 (en) | Reliable voltage scaled links for compressed data | |
US20230206380A1 (en) | Optimizing partial writes to compressed blocks | |
CN109886862B (en) | Data processing method and device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |