TW202029121A - Motion estimation through input perturbation - Google Patents

Abstract

The present disclosure relates to methods and devices for motion estimation which may include a GPU. In one aspect, the GPU may generate at least one first motion vector in a first subset of a frame, the first motion vector providing a first motion estimation for image data in the first subset of the frame. The GPU may also perturb the image data. Also, the GPU may generate at least one second motion vector based on the perturbed image data, the second motion vector providing a second motion estimation for the image data. Moreover, the GPU may compare the first motion vector and the second motion vector. Further, the GPU may determine at least one third motion vector for the motion estimation of the image data based on the comparison between the first motion vector and the second motion vector.

Description

Momentum estimation via input disturbance

The present invention generally relates to processing systems, and more particularly, to one or more technologies used for graphics processing in processing systems.

Computing devices often use graphics processing units (GPUs) to accelerate the presentation of graphics data for display. Such computing devices may include, for example, computer workstations, mobile phones such as so-called smart phones, embedded systems, personal computers, tablet computers, and video game consoles. The GPU executes a graphics processing pipeline, which includes a plurality of processing stages that operate together to execute graphics processing commands and output frames. The central processing unit (CPU) can control the operation of the GPU by issuing one or more graphics processing commands to the GPU. Modern CPUs are usually capable of executing multiple applications at the same time, and each application may need to utilize the GPU during execution. Devices that provide content for visual presentation on displays generally include a graphics processing unit (GPU).

Generally, the GPU of the device is configured to execute each program in the graphics processing pipeline. However, with the advent of wireless communication and content streaming (for example, game content or any other content rendered using GPU), there is a need for distributed graphics processing. For example, it has emerged that the processing performed by the GPU of a first device (for example, a client device such as a game console, virtual reality device, or any other device) is offloaded to a second device (for example, a server such as hosting a mobile game). Server).

The following content presents a simplified overview of one or more aspects to provide a basic understanding of these aspects. This summary is not an extensive overview of all the aspects covered, and neither intends to identify the key or decisive elements of all aspects, nor does it intend to delimit any or all aspects. Its sole purpose is to present some concepts in one or more aspects in a simplified form as a prelude to more specific implementations presented later.

In one aspect of the present invention, a method, a computer-readable medium, and a first device are provided. The device may be a GPU. In one aspect, the GPU may generate at least one first motion vector in a first subset of a frame, and the first motion vector provides a first motion vector for the image data in the first subset of the frame. A momentum estimate. The GPU may also disturb the image data in the first subset of the frame. In addition, the GPU can generate at least one second motion vector based on the disturbed image data, the second motion vector providing a second momentum estimate for the image data in the first subset of the frame. In addition, the GPU can compare the first motion vector with the second motion vector. In addition, the GPU may determine at least one third motion vector for the momentum estimation of the image data in the first subset of the frame based on the comparison between the first motion vector and the second motion vector .

The details of one or more examples of the present invention are set forth in the accompanying drawings and the description below. Other features, objectives and advantages of the present invention will be apparent from the description and drawings and the scope of the patent application.

This application claims the rights of U.S. Non-Provisional Application No. 16/215,547 filed on December 10, 2018, and the entire content of the application is incorporated herein by reference.

Hereinafter, various aspects of the system, device, computer program product and method are described more fully with reference to the accompanying drawings. However, the present invention may be embodied in many different forms, and it should not be construed as being limited to any specific structure or function presented throughout the present invention. The fact is that these aspects are provided so that the present invention will be thorough and complete, and will fully convey the scope of the present invention to those skilled in the art. Based on the teachings herein, those familiar with the art should understand that the scope of the present invention is intended to cover the systems, devices, and computer programs disclosed herein that are implemented independently of or in combination with other aspects of the present invention Any aspect of products and methods. For example, any number of aspects described herein can be used to implement the device or any number of aspects described herein can be used to practice a method. In addition, the scope of the present invention is intended to cover the use of other structures, functions, or structures and functionalities other than the various aspects of the present invention described herein or different from the various aspects of the present invention described herein To practice this device or method. Any aspect disclosed herein can be embodied by one or more elements within the scope of the patent application.

Although various aspects are described herein, many variations and permutations of these aspects are within the scope of the present invention. Although some potential benefits and advantages of the aspects of the present invention are mentioned, the scope of the present invention is not intended to be limited to specific benefits, uses, or goals. In fact, the aspects of the present invention are intended to be widely applicable to different wireless technologies, system configurations, networks, and transmission protocols, some of which are illustrated in the figures and in the following description with examples. The embodiments and drawings only illustrate the present invention rather than limit the present invention. The scope of the present invention is defined by the scope of the attached patent application and its equivalents.

Refer to various devices and methods to present several aspects. These devices and methods are described in the following detailed description and illustrated in the accompanying drawings through various blocks, components, circuits, programs, algorithms, etc. (collectively referred to as "components"). These components can be implemented using electronic hardware, computer software, or any combination thereof. Whether these components are implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system.

By way of example, an element or any part of an element or any combination of elements may be implemented as a "processing system" that includes one or more processors (which may also be referred to as processing units). Examples of processors include microprocessors, microcontrollers, graphics processing units (GPU), general-purpose GPUs (GPGPU), central processing units (CPU), application processors, digital signal processors (DSP), and reduced instruction set computing (RISC) processor, system-on-chip (SoC), baseband processor, special application integrated circuit (ASIC), field programmable gate array (FPGA), programmable logic device (PLD), state machine, gate Control logic, discrete hardware circuits, and other suitable hardware configured to perform the various functionalities described throughout this invention. One or more processors in the processing system can execute software. Software will be broadly interpreted as meaning instructions, instruction sets, code, code segments, code, programs, subprograms, software components, applications, software applications, packaged software, routines, subroutines, objects, Execution code, execution thread, program, function, etc., regardless of what it is called software, firmware, middleware, microcode, hardware description language or other. The term application can refer to software. As described herein, one or more technologies may refer to applications (ie, software) that are configured to perform one or more functions. In such instances, the application program can be stored in memory (for example, on-chip memory of the processor, system memory, or any other memory). The hardware described herein, such as a processor, can be configured to execute application programs. For example, an application program can be described as including code that, when executed by hardware, causes the hardware to perform one or more of the techniques described herein. As an example, hardware can access code from memory and execute code accessed from memory to perform one or more of the techniques described in this article. In some instances, components are identified in the present invention. In such instances, the component may be hardware, software, or a combination thereof. Components can be individual components or subcomponents of a single component.

Therefore, in one or more examples described herein, the described functions can be implemented by hardware, software or any combination thereof. If implemented in software, the functions can be stored on a computer-readable medium or encoded as one or more instructions or program codes on the computer-readable medium. Computer-readable media include computer storage media. The storage medium can be any available medium that can be accessed by a computer. By way of example but not limitation, such computer-readable media may include random-access memory (RAM), read-only memory (ROM), and electrically erasable programmable ROM (electrically erasable programmable ROM, EEPROM), optical disk storage, magnetic disk storage, other magnetic storage devices, combinations of the foregoing types of computer-readable media, or any other media that can be used to store computer-executable codes in the form of instructions or A data structure that can be accessed by a computer.

Generally speaking, the present invention describes the use of distributed graphics processing pipelines across multiple devices, improved graphics content coding, and/or reduction of processing units (ie, configured to perform any of one or more of the technologies described herein). A processing unit, such as a graphics processing unit (GPU)) that is loaded with technology. For example, this invention describes graphics processing techniques used in communication systems. Other example benefits are described throughout this disclosure.

As used herein, the term "code writer" can generally refer to an encoder and/or decoder. For example, the reference to the "content writer" may include a reference to a content encoder and/or a content decoder. Similarly, as used herein, the term "coding" can generally refer to encoding and/or decoding. As used herein, the terms "encoding" and "compression" are used interchangeably. Similarly, the terms "decoding" and "decompression" are used interchangeably.

As used herein, examples of the term "content" can refer to the terms "video", "graphic content", "image" and vice versa. This holds true whether these terms are used as adjectives, nouns or other parts of speech. For example, the reference to "Content Writer" can include references to "Video Code Writer", "Graphic Content Writer" or "Image Code Writer"; and to "Video Code Writer", " The reference to "Graphic Content Code Writer" or "Image Code Writer" may include the reference to "Content Code Writer". As another example, a reference to a processing unit that provides content to a content writer may include a reference to a processing unit that provides graphics content to a video encoder. In some instances, as used herein, the term "graphic content" may refer to content generated by one or more programs of the graphics processing pipeline. In some examples, as used herein, the term "graphic content" may refer to content generated by a processing unit configured to perform graphics processing. In some instances, as used herein, the term "graphic content" may refer to content generated by a graphics processing unit.

As used herein, examples of the term "content" can refer to graphic content or display content. In some examples, as used herein, the term "graphic content" may refer to content generated by a processing unit configured to perform graphics processing. For example, the term "graphic content" may refer to content generated by one or more programs of the graphics processing pipeline. In some instances, as used herein, the term "graphic content" may refer to content generated by a graphics processing unit. In some instances, as used herein, the term "display content" may refer to content generated by a processing unit configured to perform display processing. In some instances, as used herein, the term “display content” may refer to content generated by the display processing unit. The graphic content can be processed to become display content. For example, the graphics processing unit may output graphics content such as a frame to a buffer (which may be referred to as a frame buffer). The display processing unit can read graphics content such as one or more frames from the buffer, and perform one or more display processing techniques on it to generate display content. For example, the display processing unit can be configured to perform synthesis on one or more presented layers to generate a frame. As another example, the display processing unit may be configured to synthesize, fuse, or otherwise combine two or more layers together into a single frame. The display processing unit may be configured to perform scaling (for example, scaling up or scaling down). In some examples, the frame may refer to a layer. In other examples, the frame may refer to two or more layers that have been fused together to form the frame (ie, the frame includes two or more layers, and may subsequently be fused to include two or more Frames on two layers).

As referenced herein, the first component (e.g., processing unit) may provide content such as graphical content to the second component (e.g., content writer). In some examples, the first component can provide content to the second component by storing the content in a memory accessible by the second component. In such instances, the second component can be configured to read the content stored in the memory by the first component. In other examples, the first component may provide content to the second component without any intermediate components (for example, no memory or another component). In such instances, the first component may be described as providing content directly to the second component. For example, the first component can output content to the second component, and the second component can be configured to store the content received from the first component in a memory such as a buffer.

FIG. 1 is a block diagram illustrating an example content generation and coding system 100 configured to implement one or more techniques of the present invention. The content generation and coding system 100 includes a source device 102 and a destination device 104. According to the techniques described herein, the source device 102 can be configured to use the content encoder 108 to encode the graphical content generated by the processing unit 106 before transmission to the destination device 104. The content encoder 108 can be configured to output a bit stream with a bit rate. The processing unit 106 may be configured to control and/or influence the bit rate of the content encoder 108 based on how the processing unit 106 generates graphical content.

The source device 102 may include one or more components (or circuits) for performing various functions described herein. The destination device 104 may include one or more components (or circuits) for performing various functions described herein. In some examples, one or more components of the source device 102 may be system-on-chip (SOC) components. Similarly, in some examples, one or more components of the destination device 104 may be components of the SOC.

The source device 102 may include one or more components configured to perform one or more techniques of the present invention. In the example shown, the source device 102 may include a processing unit 106, a content encoder 108, a system memory 110, and a communication interface 112. The processing unit 106 may include an internal memory 109. The processing unit 106 may be configured to perform graphics processing, such as in the graphics processing pipeline 107-1. The content encoder 108 may include an internal memory 111.

The memory outside the processing unit 106 and the content encoder 108 (such as the system memory 110) can be accessed through the processing unit 106 and the content encoder 108. For example, the processing unit 106 and the content encoder 108 can be configured to read from and/or write to external memory (such as the system memory 110). The processing unit 106 and the content encoder 108 can be communicatively coupled to the system memory 110 via a bus. In some examples, the processing unit 106 and the content encoder 108 may be communicatively coupled to each other via a bus or different connections.

The content encoder 108 can be configured to receive graphical content from any source such as the system memory 110 and/or the processing unit 106. The system memory 110 can be configured to store graphics content generated by the processing unit 106. For example, the processing unit 106 can be configured to store graphic content in the system memory 110. The content encoder 108 may be configured to receive graphic content in the form of pixel data (for example, from the system memory 110 and/or the processing unit 106). In addition, the content encoder 108 can be configured to receive the pixel data of the graphic content generated by the processing unit 106. For example, the content encoder 108 may be configured to receive the value of each component (eg, each color component) of one or more pixels of the graphic content. As an example, pixels in the red (R), green (G), and blue (B) (RGB) color spaces may include the first value of the red component, the second value of the green component, and the third value of the blue component .

The internal memory 109, the system memory 110, and/or the internal memory 111 may include one or more volatile or non-volatile memory or storage devices. In some examples, the internal memory 109, the system memory 110, and/or the internal memory 111 may include random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), erasable and programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic data media or optical storage media or any other type of memory.

According to some examples, the internal memory 109, the system memory 110, and/or the internal memory 111 may be non-transitory storage media. The term "non-transitory" may indicate that the storage medium is not embodied by a carrier wave or a propagated signal. However, the term "non-transitory" should not be interpreted as meaning that the internal memory 109, the system memory 110, and/or the internal memory 111 are not removable or their contents are static. As an example, the system memory 110 can be removed from the source device 102 and moved to another device. As another example, the system memory 110 may not be removable from the source device 102.

The processing unit 106 may be a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose GPU (GPGPU), or any other processing unit that can be configured to perform graphics processing. In some examples, the processing unit 106 may be integrated into the motherboard of the source device 102. In some examples, the processing unit 106 may exist on a graphics card installed in a port in the motherboard of the source device 102, or may be incorporated in a peripheral device configured to interoperate with the source device 102 in other ways.

The processing unit 106 may include one or more processors, such as one or more microprocessors, application-specific integrated circuits (ASIC), field programmable gate array (FPGA), arithmetic logic unit (ALU), digital signal processing DSP, discrete logic, software, hardware, firmware, other equivalent integrated or discrete logic circuits, or any combination thereof. If the technical part is implemented in software, the processing unit 106 can store the instructions for the software in a suitable non-transitory computer-readable storage medium (for example, the internal memory 109), and can use one or more processors Execute instructions in the hardware to execute the technology of the present invention. Any one of the above content (including hardware, software, a combination of hardware and software, etc.) can be regarded as one or more processors.

The content encoder 108 may be any processing unit configured to perform content encoding. In some examples, the content encoder 108 may be integrated into the motherboard of the source device 102. The content encoder 108 may include one or more processors, such as one or more microprocessors, application-specific integrated circuits (ASIC), field programmable gate array (FPGA), arithmetic logic unit (ALU), digital signal Processor (DSP), discrete logic, software, hardware, firmware, other equivalent integrated or discrete logic circuits, or any combination thereof. If the technology is implemented in software, the content encoder 108 can store instructions for the software in a suitable non-transitory computer-readable storage medium (for example, the internal memory 111), and can use one or more processors Execute instructions in the hardware to execute the technology of the present invention. Any one of the above content (including hardware, software, a combination of hardware and software, etc.) can be regarded as one or more processors.

The communication interface 112 may include a receiver 114 and a transmitter 116. The sink 114 can be configured to perform any of the receiving functions described herein with respect to the source device 102. For example, the receiver 114 may be configured to receive information from the destination device 104, which may include a request for content. In some instances, in response to receiving a request for content, the source device 102 may be configured to perform one or more of the techniques described herein, such as generating or otherwise generating graphical content for delivery to the destination device 104. The transmitter 116 can be configured to perform any of the transmission functions described herein with respect to the source device 102. For example, the transmitter 116 may be configured to transmit the encoded content to the destination device 104, such as the encoded graphic content generated by the processing unit 106 and the content encoder 108 (ie, the graphic content is generated by the processing unit 106). , The content encoder 108 receives the graphic content as input to generate or otherwise generate encoded graphic content). The receiver 114 and the transmitter 116 can be combined into a transceiver 118. In such instances, the transceiver 118 may be configured to perform any of the receiving and/or transmitting functions described herein with respect to the source device 102.

The destination device 104 may include one or more components configured to perform one or more techniques of the present invention. In the example shown, the destination device 104 may include a processing unit 120, a content decoder 122, a system memory 124, a communication interface 126, and one or more displays 131. The reference to the display 131 may refer to one or more displays 131. For example, the display 131 may include a single display or a plurality of displays. The display 131 may include a first display and a second display. The first display may be a left-eye display, and the second display may be a right-eye display. In some examples, the first display and the second display may receive different frames for presentation thereon. In other examples, the first display and the second display may receive the same frame for presentation thereon.

The processing unit 120 may include an internal memory 121. The processing unit 120 may be configured to perform graphics processing, such as in the graphics processing pipeline 107-2. The content decoder 122 may include an internal memory 123. In some examples, the destination device 104 may include a display processor, such as a display processor 127, to perform one or more frames on one or more frames generated by the processing unit 120 before being presented by the one or more displays 131 Display processing technology. The display processor 127 may be configured to perform display processing. For example, the display processor 127 may be configured to perform one or more display processing techniques on one or more frames generated by the processing unit 120. One or more displays 131 may be configured to display content generated using decoded content. For example, the display processor 127 may be configured to process one or more frames generated by the processing unit 120, wherein the one or more frames are coded by the processing unit 120 by using the source device 102 The content is derived from the decoded content. Subsequently, the display processor 127 may be configured to perform display processing on one or more frames generated by the processing unit 120. One or more displays 131 may be configured to display or otherwise present the frames processed by the display processor 127. In some examples, the one or more display devices may include one or more of the following: liquid crystal display (LCD), plasma display, organic light emitting diode (OLED) display, projection display device, augmented reality display Device, virtual reality display device, head mounted display or any other type of display device.

The memory outside the processing unit 120 and the content decoder 122 (such as the system memory 124) can be accessed through the processing unit 120 and the content decoder 122. For example, the processing unit 120 and the content decoder 122 can be configured to read from and/or write to the external memory 124 from an external memory such as the system memory 124. The processing unit 120 and the content decoder 122 may be communicatively coupled to the system memory 124 via a bus. In some examples, the processing unit 120 and the content decoder 122 may be communicatively coupled to each other via a bus or through different connections.

The content decoder 122 can be configured to receive graphical content from any source such as the system memory 124 and/or the communication interface 126. The system memory 124 may be configured to store received coded graphic content such as coded graphic content received from the source device 102. The content decoder 122 may be configured to receive encoded graphic content in the form of encoded pixel data (eg, from the system memory 124 and/or the communication interface 126). The content decoder 122 can be configured to decode encoded graphic content.

The internal memory 121, the system memory 124, and/or the internal memory 123 may include one or more volatile or non-volatile memory or storage devices. In some examples, the internal memory 121, the system memory 124 and/or the internal memory 123 may include random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), erasable and programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic data media or optical storage media or any other type of memory.

According to some examples, the internal memory 121, the system memory 124, and/or the internal memory 123 may be non-transitory storage media. The term "non-transitory" may indicate that the storage medium is not embodied by carrier waves or propagated signals. However, the term "non-transitory" should not be interpreted as meaning that the internal memory 121, the system memory 124, and/or the internal memory 123 are not removable or their contents are static. As an example, the system memory 124 can be removed from the destination device 104 and moved to another device. As another example, the system memory 124 may not be removable from the destination device 104.

The processing unit 120 may be a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose GPU (GPGPU), or any other processing unit that can be configured to perform graphics processing. In some examples, the processing unit 120 may be integrated into the motherboard of the destination device 104. In some instances, the processing unit 120 may exist on a graphics card installed in a port in the motherboard of the destination device 104, or may be incorporated in a peripheral device configured to interoperate with the destination device 104 in other ways .

The processing unit 120 may include one or more processors, such as one or more microprocessors, application-specific integrated circuits (ASIC), field programmable gate array (FPGA), arithmetic logic unit (ALU), digital signal processing DSP, discrete logic, software, hardware, firmware, other equivalent integrated or discrete logic circuits, or any combination thereof. If the technology is implemented in software, the processing unit 120 can store the instructions for the software in a suitable non-transitory computer-readable storage medium (for example, the internal memory 121), and can use one or more processors Execute instructions in the hardware to execute the technology of the present invention. Any one of the above (including hardware, software, a combination of hardware and software, etc.) can be regarded as one or more processors.

The content decoder 122 may be any processing unit configured to perform content decoding. In some examples, the content decoder 122 may be integrated into the motherboard of the destination device 104. The content decoder 122 may include one or more processors, such as one or more microprocessors, application-specific integrated circuits (ASIC), field programmable gate array (FPGA), arithmetic logic unit (ALU), digital signal Processor (DSP), discrete logic, software, hardware, firmware, other equivalent integrated or discrete logic circuits, or any combination thereof. If the technology is implemented in software, the content decoder 122 can store instructions for the software in a suitable non-transitory computer-readable storage medium (for example, internal memory 123), and can use one or more processors Execute instructions in the hardware to execute the technology of the present invention. Any one of the above content (including hardware, software, a combination of hardware and software, etc.) can be regarded as one or more processors.

The communication interface 126 may include a receiver 128 and a transmitter 130. The receiver 128 can be configured to perform any of the receiving functions described herein with respect to the destination device 104. For example, the receiver 128 may be configured to receive information from the source device 102, and the information may include encoded content, such as the encoded content generated by the processing unit 106 and the content encoder 108 of the source device 102 or otherwise Graphic content (that is, the graphic content is generated by the processing unit 106, and the content encoder 108 receives the graphic content as input to generate or otherwise generate encoded graphic content). As another example, the receiver 114 may be configured to receive location information from the destination device 104, which location information may be encoded or unencoded (ie, unencoded). In addition, the receiver 128 can be configured to receive location information from the source device 102. In some examples, the destination device 104 may be configured to decode the encoded graphical content received from the source device 102 according to the techniques described herein. For example, the content decoder 122 may be configured to decode encoded graphic content, thereby generating or otherwise generating decoded graphic content. The processing unit 120 may be configured to use the decoded graphics content to generate or otherwise generate one or more frames for presentation on the one or more displays 131. The transmitter 130 can be configured to perform any of the transmission functions described herein with respect to the destination device 104. For example, the transmitter 130 may be configured to transmit information to the source device 102, and the information may include a request for content. The receiver 128 and the transmitter 130 can be combined into a transceiver 132. In such instances, the transceiver 132 may be configured to perform any of the receiving functions and/or transmission functions described herein with respect to the destination device 104.

The content encoder 108 and the content decoder 122 of the content generation and coding system 100 represent examples of computing components (for example, processing units), which can be configured to execute according to the various examples described in the present invention. One or more technologies for encoding content and decoding content. In some examples, the content encoder 108 and the content decoder 122 may be configured to operate according to content coding standards such as video coding standards, display streaming compression standards, or image compression standards.

As shown in Figure 1, the source device 102 can be configured to generate encoded content. Therefore, the source device 102 may be referred to as a content encoding device or a content encoding device. The destination device 104 can be configured to decode the encoded content generated by the source device 102. Therefore, the destination device 104 may be referred to as a content decoding device or a content decoding device. In some examples, as shown, the source device 102 and the destination device 104 may be independent devices. In other examples, the source device 102 and the destination device 104 may be on the same computing device or part of the same computing device. In either instance, the graphics processing pipeline can be distributed between two devices. For example, a single graphics processing pipeline may include multiple graphics processing. The graphics processing pipeline 107-1 may include one or more of a plurality of graphics processing. Similarly, the graphics processing pipeline 107-2 may include one or more of a plurality of graphics processing. In this regard, the graphics processing pipeline 107-1 is connected in series or otherwise follows the graphics processing pipeline 107-2 to form a complete graphics processing pipeline. In addition, the graphics processing pipeline 107-1 may be a partial graphics processing pipeline, and the graphics processing pipeline 107-2 may be a partial graphics processing pipeline, and when combined, a distributed graphics processing pipeline is generated.

1 again, in some aspects, the graphics processing pipeline 107-2 may include a generating component configured to generate at least one first motion vector in the first subset of the frame, the first motion vector providing Used for the first momentum estimation of the image data in the first subset of the frame. The graphics processing pipeline 107-2 may also include a disturbance component configured to disturb the image data in the first subset of the frame. In addition, the generating component can be configured to generate at least one second motion vector based on the disturbed image data in the first subset of the frame, and the at least one second motion vector provides the image data for the first subset of the frame Second momentum estimate. The graphics processing pipeline 107-2 may also include a comparison component configured to compare the first motion vector with the second motion vector. In addition, the graphics processing pipeline 107-2 may include a determination component 198 configured to determine the momentum estimate for the image data in the first subset of the frame based on the comparison between the first motion vector and the second motion vector At least one third motion vector. By distributing the graphics processing pipeline between the source device 102 and the destination device 104, in some instances, the destination device may be able to render graphical content that cannot be rendered in other ways; and therefore, may not render. Other example benefits are described throughout this disclosure.

As described herein, a device such as source device 102 and/or destination device 104 can refer to any device, apparatus, or system configured to perform one or more of the techniques described herein. For example, the device can be a server, base station, user equipment, client device, station, access point, computer (for example, personal computer, desktop computer, laptop computer, tablet computer, computer workstation or Mainframe computers), final products, devices, phones, smart phones, servers, video game platforms or consoles, handheld devices (for example, portable video game devices or personal digital assistants (PDAs)), wearable computing devices ( Such as smart watches, augmented reality devices or virtual reality devices), non-wearable devices, augmented reality devices, virtual reality devices, displays (for example, display devices), televisions, TV boxes, intermediate network devices, digital media playback Device, video streaming device, content streaming device, on-board computer, any mobile device, any device configured to produce graphical content, or any device configured to perform one or more of the technologies described in this article Device.

The source device 102 can be configured to communicate with the destination device 104. For example, the destination device 104 may be configured to receive encoded content from the source device 102. In some examples, the coupled communication between the source device 102 and the destination device 104 is shown as a link 134. The link 134 may include any type of media or device capable of moving the encoded content from the source device 102 to the destination device 104.

In the example of FIG. 1, the link 134 may include a communication medium to enable the source device 102 to transmit the encoded content to the destination device 104 in real time. The encoded content can be modulated according to a communication standard such as a wireless communication protocol and transmitted to the destination device 14. The communication medium may include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network (such as a local area network, a wide area network, or a global network such as the Internet). The communication medium may include routers, switches, base stations, or any other equipment that can be used to facilitate communication from the source device 102 to the destination device 104. In other examples, the link 134 may be a point-to-point connection between the source device 102 and the destination device 104, such as a wired or wireless display link connection (eg, HDMI link, display port link, MIPI DSI link, or via The encoded content may traverse another link from the source device 102 to the destination device 104 via it).

In another example, the link 134 may include a storage medium configured to store the encoded content generated by the source device 102. In this example, the destination device 104 can be configured to access the storage medium. The storage media may include various locally accessible data storage media, such as Blu-ray Disc, DVD, CD-ROM, flash memory, or other suitable digital storage media for storing encoded content.

In another example, the link 134 may include a server or another intermediate storage device configured to store the encoded content generated by the source device 102. In this example, the destination device 104 may be configured to access encoded content stored in a server or other intermediate storage device. The server may be a type of server capable of storing encoded content and transmitting the encoded content to the destination device 104.

The devices described herein can be configured to communicate with each other, such as source device 102 and destination device 104. Communication may include the transmission and/or reception of information. Information can be carried in one or more messages. As an example, the first device communicating with the second device may be described as being communicatively coupled to the second device or otherwise coupled to the second device. For example, the client device and the server can be coupled in a communication manner. As another example, the server may be communicatively coupled to a plurality of client devices. As another example, any device described herein that is configured to perform one or more techniques of the present invention may be communicatively coupled to one or more other devices configured to perform one or more techniques of the present invention. In some examples, when coupled in a communicative manner, the two devices can actively transmit or receive information, or can be configured to transmit or receive information. If not communicatively coupled, any two devices may be configured to be communicatively coupled to each other, such as according to one or more communication protocols that comply with one or more communication standards. The reference to "any two devices" does not mean that only two devices can be configured to be communicatively coupled to each other; rather, any two devices include more than two devices. For example, the first device may be communicatively coupled with the second device, and the first device may be communicatively coupled with the third device. In such instances, the first device may be a server.

1, the source device 102 may be described as being communicatively coupled to the destination device 104. In some examples, the term "communicatively coupled" may refer to a communication connection that may be direct or indirect. In some examples, the link 134 may represent the communication coupled between the source device 102 and the destination device 104. The communication connection can be wired and/or wireless. Wired connections can refer to conductive paths, traces, or physical media (excluding wireless physical media) through which information can travel. The conductive path can refer to any conductor of any length, such as a conductive pad, a conductive via, a conductive plane, a conductive trace, or any conductive medium. A direct communication connection may refer to a connection between two components that are communicatively coupled without an intermediate component. Indirect communication connection may refer to a connection in which at least one intermediate component resides between two components that are communicatively coupled. Two devices that are communicatively coupled can communicate with each other via one or more different types of networks (for example, wireless networks and/or wired networks) according to one or more communication protocols. In some examples, two devices that are communicatively coupled can be associated with each other through an association process. In other examples, two devices that are communicatively coupled can communicate with each other without associated processing. For example, a device such as the source device 102 may be configured to unicast, broadcast, multicast, or otherwise transmit information (eg, encoded content) to one or more other devices (eg, including destination devices) 104 one or more destination devices). The destination device 104 in this example can be described as being communicatively coupled to each of one or more other devices. In some instances, the communication connection can enable the transmission and/or reception of information. For example, according to the technology of the present invention, a first device communicatively coupled to a second device can be configured to transmit information to and/or receive information from the second device. Similarly, according to the technology of the present invention, the second device in this example can be configured to transmit information to and/or receive information from the first device. In some examples, the term "communicatively coupled" may refer to a temporary, intermittent, or permanent communication connection.

Any of the devices described herein (such as source device 102 and destination device 104) can be configured to operate according to one or more communication protocols. For example, the source device 102 may be configured to communicate with the destination device 104 using one or more communication protocols (eg, receive information from the destination device 104 and/or transmit information to the destination device 104). In such instances, the source device 102 may be described as communicating with the destination device 104 via a connection. The connection may conform to or be based on a communication protocol in other ways. Similarly, the destination device 104 can be configured to communicate with the source device 102 using one or more communication protocols (eg, receive information from the source device 102 and/or transmit information to the source device 102). In such instances, the destination device 104 may be described as communicating with the source device 102 via a connection. The connection may conform to or be based on a communication protocol in other ways.

As used herein, the term "communication protocol" can refer to any communication protocol, such as a communication protocol compatible with a communication standard. As used herein, the term "communication standard" may include any communication standard such as a wireless communication standard and/or a wired communication standard. The wireless communication standard may correspond to a wireless network. As an example, the communication standard may include any wireless communication standard corresponding to wireless personal area network (WPAN) standards such as Bluetooth (eg, IEEE 802.15), Bluetooth Low Energy (BLE) (eg, IEEE 802.15.4) . As another example, the communication standard may include any wireless local area network (WLAN) standard such as Wi-Fi (for example, any 802.11 standard, such as 802.11a, 802.11b, 802.11c, 802.11n, or 802.11ax). Wireless communication standards. As another example, the communication standard may include any wireless communication standard corresponding to a wireless wide area network (WWAN) standard such as 3G, 4G, 4G LTE, or 5G.

With reference to Figure 1, the content encoder 108 may be configured to encode graphical content. In some examples, the content encoder 108 may be configured to encode graphical content into one or more video frames. When the content encoder 108 encodes the content, the content encoder 108 may generate a bitstream. The bit stream may have a bit rate such as bit/time unit, where the time unit is any time unit such as seconds or minutes. The bit stream may include a sequence of bits forming a coded representation of the graphic content and related data. In order to generate a bit stream, the content encoder 108 may be configured to perform an encoding operation on pixel data, such as pixel data corresponding to a shadow texture map. For example, when the content encoder 108 performs an encoding operation on the image data provided as input to the content encoder 108 (for example, one or more blocks of the shadow texture map), the content encoder 108 can generate a series of writing codes Images and related information. Related information may include a set of coding parameters such as quantization parameters (QP).

Momentum estimation is the process of analyzing multiple two-dimensional (2D) images and generating a motion vector describing the movement of an area from one image to another. Essentially, momentum estimation generates motion vectors that describe how objects move within certain parts of the image. Motion vectors have various uses, including video compression, post-processing effects (such as motion blur), and frame extrapolation or interpolation. In order to reduce the rendering workload on the GPU, virtual reality (VR) or augmented reality (AR) systems can use momentum estimation to extrapolate frames from previously rendered content. In this way, this allows the GPU to render the frame at a reduced rate, and the extrapolated frame will be displayed to the user where the content is presented. Momentum estimation may be applicable because in VR or AR systems, there are stronger drivers that reduce rendering workload, such as in GPUs. The present invention can reduce the rendering workload by rendering fewer frames on the GPU and using momentum estimation to fill the gaps in the image or motion vector. In addition, although momentum estimation can be performed on the video content, the momentum estimation described herein can utilize the presentation content presented in real time on the GPU.

In some cases, the repeated patterns in the input image may be difficult to process for momentum estimation. For example, presentation content has more frequent repeating patterns than other content (for example, light content) because presentation content uses texture mapping with repeating patterns. In some aspects, momentum estimation techniques have problems with repeating patterns because these techniques try to match objects that move from one frame to another. Because the presentation content can accurately repeat the pattern in some cases, there can be erroneous motion matching, where the momentum estimation incorrectly attempts to match the motion. For example, due to the repeating pattern, the momentum estimation can skip the motion cycle, resulting in incorrect mapping of the next element and generating incorrect momentum estimation. In addition, since many areas of the image can be matched equally, some systems may have difficulty identifying motion correctly. Wrong motion vectors can be generated in these areas of incorrect momentum estimation, which can lead to serious errors in use cases based on accurate motion recognition. Since presentation content can contain repeated structures and free use of patterns, as presentation content momentum estimation use cases increase, incorrect momentum estimation may become an increasingly serious problem.

Some aspects of the present invention can provide a method for identifying false motion vectors. By correctly identifying the wrong motion vector, it can be removed to generate a more accurate momentum estimate. In some aspects, the momentum estimation according to the present invention may be in addition to or on top of the general momentum estimation process. Therefore, some aspects of the present invention can execute the momentum estimation process multiple times (for example, twice). In these cases, momentum estimation can be performed via input disturbances, which can lead to an overall improvement in momentum estimation. This momentum estimation with input perturbation can be performed twice, where it can be performed once for the original input image, and the perturbed version of the input image can be performed a second time. In some aspects, by sending the input image and generating the resulting motion vector, the first pass can be an undisturbed or strict pass. The second pass can be a perturbation pass, where the motion vector of the input image is perturbed in some ways. When all passes are completed, the obtained motion vectors can be compared to determine, for example, real motion and invalid motion caused by repeated patterns. In some cases, the first pass and the second pass can be executed simultaneously or in parallel. The vector obtained from the match between the original pass and the disturbance pass is identified as valid, and the unmatched vector is discarded as invalid.

The present invention can perform input perturbation in a variety of ways. In some aspects, the present invention can introduce sufficiently new irregularities into the image so that it can disturb the momentum estimation. For example, by introducing new irregularities into the motion vector, the momentum estimation can affect the motion vector in certain areas of the input image (such as the repeated pattern area). In some cases, this irregularity can be the δ or difference value of the motion vector. In such cases, there should not be too much delta or difference, so as not to lose track of the real object's movement. Therefore, the present invention can find a balance between adding enough noise to disturb the input image without affecting the real motion. As mentioned above, the present invention can compare, for example, two passes of no disturbance pass and disturbance pass. Areas where momentum estimation is not aligned between two passes can be identified as uncorrelated or erroneous motion vectors. In some instances, there may be too much disturbance, so that the momentum estimate changes greatly to the extent that it affects the actual motion. In fact, too much disturbance may cause the motion in some input image areas to not actually follow the real motion, but will produce some artifacts of false repeating patterns.

Figure 2 illustrates an example of momentum estimation 200 according to the present invention. FIG. 2 shows that the content block 202 and the RGB perturbation texture block 204 can both be the input of the color space conversion passes 206 and 208. As shown in FIG. 2, color space conversions 206 and 208 can lead to momentum estimates 210 and 212, which can lead to delta or disparity analysis 214, and can subsequently lead to motion vector calculation 216. The color space conversion 206/208 in FIG. 2 can consider the aforementioned disturbances when converting into brightness (Y), first chromaticity (U), and second chromaticity (V) (YUV) values. The conversion from RGB to YUV can be an effective way to perturb the GPU. As shown in Figure 2, some aspects may first sample the presentation content, then perturb the RGB values and then perform the YUV conversion based on these equivalent values. Some aspects of the present invention may not pay attention to what type of YUV conversion is used. For example, the type of YUV conversion may depend on the YUV being followed. In other aspects, the noise is calculated by multiplying pixels by pixels, and this conversion can be done in addition to the color space conversion. In addition, an effective way to perform perturbation can be to input noise as another texture.

In some aspects, the perturbation may allow the GPU to determine the motion vector and then present subsequent frames. Effective perturbation can modify the input enough to disrupt false regions or feature matching due to the repeated pattern, but it may not affect the input within the range of impairing the recognition of real motion. In some aspects of the present invention, the momentum estimation path of the VR or AR framework for performing frame extrapolation can be implemented on some platforms such as the SDM845 platform. As mentioned above, the method used to perform perturbation can be color space conversion. The presentation content can usually be in the RGB color space, and the momentum estimation can be in the YUV color space. Therefore, before the present invention performs momentum estimation, color space conversion can be performed. For example, the color space conversion can be performed on the GPU during the first or bumpless pass. The invention can also sample textures that contain the value of how the input will be disturbed. Therefore, the present invention can perform this modification during the color space conversion, and then continue to perform momentum estimation.

In some cases, the aforementioned perturbation can be achieved by transferring additional noise texture to the existing RGB to YUV conversion, for example, by using a conversion shader that applies a certain amount of uniformly distributed random RGB values. In some aspects, the magnitude can be added to each color channel of the input image before the color space conversion. In some aspects, within the color space conversion, noise addition can be performed, and then the color space conversion can be performed on the modified value. The conversion shader can apply several different amplitude values (for example, 5% value). By determining when the noise can eliminate too many legal vectors, the magnitude can be determined in advance or through experiments. In fact, this value can be flexible because the same value may not be applicable to every image.

In some aspects, not only can a constant noise magnitude be applied, but the magnitude can also be changed based on the variance of the input image. For example, if there is an input image with a high contrast area in a specific area, the present invention can apply a higher degree of noise texture. In other examples, if there are images with low variance and softer features, the present invention can apply a lower level of noise in the perturbation pass. When the amount of noise is changed based on the variance of the image, some aspects of the present disclosure can obtain improved results. For example, if the present invention determines the variance of the region and uses the variance to determine the amount of noise to be applied, an improved momentum estimate can be obtained. Therefore, by measuring the variance of each local area of the image and using the variance to determine how much noise to apply, the present invention can improve the ability to detect false motion vectors and reduce the possibility of accidentally discarding real motion. In fact, the present invention can obtain more effective results with more or less noise based on the actual input image.

As mentioned above, the amount of variance imposed by the present invention can be content-dependent. If too much noise is applied, it can disturb the feature recognition of actual features. In some instances, depending on the content of the image, the noise level of 10% may be too high. For example, by applying too much noise in certain areas, the result may be too many false motion vectors in that area. When the noise starts to interfere with the tracking of the actual object, there may also be some legal motion vectors identified as errors. Therefore, if too much disturbance is applied, the actual object tracking can be disrupted. For example, applying too much noise may lead to mismatches in some areas of the input image with legal movement, for example, the δ or the difference may be too high, which may lead to the wrong conclusion that the legal motion vector is the wrong vector. In short, if the amount of noise is too high and modified too much, the amount of noise may be too different from the actual vector, and the noise will overwhelm the real features. Therefore, the present invention can evaluate the input image in order to correctly change the amount of disturbance to be applied. In fact, some aspects of the present invention can adjust the amount of noise or disturbance applied to the motion vector based on the specific content of the input image.

3A and 3B respectively illustrate images 300 and 310 on which momentum estimation is performed. Because momentum estimation can be performed on images 300 or 310, these images can be referred to as motion estimation or momentum estimation images. As shown in FIG. 3A, the image 300 includes a repeating background 302 and an inserted texture image 304. In some aspects, the repeating background 302 may be static. The inserted texture image 304 may move in a number of different directions, for example, across the direction of the repeated background 302. Image 300 illustrates an example of momentum estimation according to the present invention, which can be executed on a hardware platform such as the SDM845 platform.

In some aspects, by processing the input image or inserting the texture image 304 and obtaining the resulting motion vector, the image 300 may include strict or undisturbed passes. In these situations, the "X" in the repeating background 302 may appear to be shifting to the right or left because it will match the other "X"s to the right or left. In other aspects, the image 300 may include perturbation passes, where the motion vector of the input image or the inserted texture image 304 is perturbed in some way, and causes the perturbed image. In these aspects, the input image or the inserted texture image 304 may be moved slightly to the right or left to generate a disturbance image, making it unlikely that it will get the same "X" match in the repeated background 302. For example, when perturbing the image 304 to generate a perturbed image, there may be incorrect "X" matches that differ from strict or undisturbed passes.

FIG. 3B shows an image 310 of the result of the disturbance. For example, the image 310 may be the result of applying a disturbance to the image 300 in FIG. 3A. In FIG. 3B, the disturbance in the image 310 is shown by the repeating background 312 and the interpolated texture image 314 that appears gray. In contrast, FIG. 3A shows that the repeated background 302 and the inserted texture image 304 appear black before the disturbance. In fact, during the perturbation times, the entire image 300 may be perturbed, which may cause the entire image 310 to be perturbed, for example, appearing in gray instead of black. The result of perturbation to the image can be reflected in many ways (such as image change or color fade). For example, compared with the image 300, the image 310 may be faded. In some aspects, the result of the disturbance can be applied to the entire image. In other aspects, the result of the disturbance can be applied to a specific part of the image.

In some aspects, when the value of "X" in the image is disturbed or shaken by disturbances, the background "X" is more likely to move in different directions and not match the previous pass. In this way, the present invention can compare the δ or difference between the strict pass and the disturbance pass. Essentially, the present invention can compare the common motion vectors between strict passes and disturbance passes. These common motion vectors may correspond to real object motion. The present invention can also perform more than two passes, so that in addition to the strict pass and the disturbance pass, there may be multiple passes. In addition, when more than two passes are executed, multiple passes can be executed in parallel. Although performing more than two passes can provide a better estimate, the indirect cost can limit the number of passes that can be performed.

As mentioned above, some examples of the present invention may not present every frame. For example, the GPU according to the present invention can render every other frame and still use the motion vector of the rendered content. In some aspects, due to the requirements of VR rendering, the GPU may be limited in resources. Therefore, not rendering each frame can save GPU resources, and a specific frame rate can be achieved by offloading the rendering work from the GPU. As mentioned previously, the present invention can use the motion vector and the resulting momentum estimate as an alternative or alternative for presenting each frame. In this way, the present invention can save the power and performance of the GPU. In addition, this may allow the present invention to render higher quality frames on the GPU.

In some aspects of the invention, when momentum estimation is performed, the result may be a single vector solution. In such cases, a threshold value can be set for the deviation between no disturbance or strict pass and disturbance pass to determine whether a certain vector is legal or wrong. As mentioned above, this can be referred to as calculating δ or difference, and the value of δ or difference can be set to any preferred value. In an example of using δ to calculate momentum estimation, if the difference between the passes of a vector is less than δ, the vector can be considered as a motion vector from the pass without disturbance. In another example, if the difference between passes of a certain vector is greater than δ, the motion can be considered as zero. The aspect of the present invention can also compare neighboring vectors and determine its motion. Some aspects of the present invention can also assume that, compared to having incorrect estimates, it is preferable to not have any motion estimation. In essence, it can be assumed that no momentum estimate is better than incorrect estimates.

The present invention can provide a number of different methods for applying disturbances to momentum estimation. For example, as mentioned above, the variance may include the local image variance calculated from the input frame data. In some aspects, the variance can be generated as an additional data stream during momentum estimation. Therefore, the variance data of the perturbed momentum estimates can be taken from the undisturbed momentum estimates. In addition, the variance data of the perturbation momentum estimates can be estimated by using the variance data of the previous frame. In some aspects, this can avoid losing parallelism on the momentum estimation passes of the current frame.

Some aspects of the present invention can provide a coloring device that applies the aforementioned noise disturbance. For example, the present invention can use the following code for this type of colorizer: #version 300 es #extension GL_EXT_YUV_target: require uniform sampler2DArray texArray; uniform sampler2D noiseTexture; uniform float perturbationFactor; in vec3 tc; layout (yuv) out vec3 fragColor; void main(void) { vec3 color = texture(texArray, tc).xyz; fragColor = rgb_2_yuv(color + (texture(noiseTexture, tc.xy).xyz * perturbationFactor), itu_601); }

In the above example code, texArray can be the input image, noiseTexture can be the noise perturbation texture, perturbationFactor can be the scale factor of the amount of perturbation applied, vec3 tc can be the texture coordinates of the input image, and fragColor can be the color of the output fragment.

The present invention can also provide a color shader that applies noise disturbance in consideration of local variance. For example, the present invention can use the following code for this type of colorizer: #version 300 es #extension GL_EXT_YUV_target: require uniform sampler2DArray texArray; uniform sampler2D noiseTexture; uniform sampler2D varianceTexture; uniform vec2 varianceSampleRange; uniform vec2 varianceSampleSpread; uniform float perturbationFactor; in vec3 tc; layout (yuv) out vec3 fragColor; void main(void) { vec3 color = texture(texArray, tc).xyz; float varianceFactor = 0.0; for (int i = -1 * varianceSampleRange.x; i ＜ varianceSampleRange.x, i++) { for (int j = -1 * varianceSampleRange.y; j ＜ varianceSampleRange.y, j++) { vec2 sampleLocation; sampleLocation.x = tc.x + (varianceSampleSpread.x * i); sampleLocation.y = tc.y + (varianceSampleSpread.y * j); varianceFactor += texture(varianceTexture, sampleLocation.xy).x; } } varianceFactor = varianceFactor/((varianceSampleRange.x*2)* (varianceSampleRange.y* 2)); fragColor = rgb_2_yuv(color + (texture(noiseTexture, tc.xy).xyz* perturbationFactor * varianceFactor), itu_601); }

In the above example code, texArray can be the input image, noiseTexture can be the noise disturbance texture, varianceTexture can be the variance data texture, varianceSampleRange can be the size of the area of the variance texture to be sampled, and varianceSampleSpread can be the difference between the sampling positions of the variance texture Distance, perturbationFactor can be the scale factor of the applied perturbation, vec3 tc can be the texture coordinates of the input image, and fragmentColor can be the color of the output fragment.

The present invention can also process an array of two motion vectors to detect deviations. For example, the resulting motion vectors from the aforementioned two passes can be compared and used. In some aspects, the threshold may be a use case specific tuning parameter set based on the use case tolerance for incorrect momentum estimation. As mentioned above, it is better not to generate momentum estimates than to generate incorrect momentum estimates in the use case using prototype VR. Therefore, in some aspects of the present invention, the threshold may be set to a minimum value such as 0.001. The present invention can use the following code to process an array of two motion vectors to detect deviations: for (int loopY = 0; loopY ＜ arrayHeight; loopY++) { for (int loopX = 0; loopX ＜ arrayWidth; loopX++) { float Xa = motionVectorsUnperturbed[(loopY * arrayWidth) + loopX].xMagnitude; float Xb = motionVectorsPerturbed[(loopY * arrayWidth) + loopX].xMagnitude; float Ya = motionVectorsUnperturbed[(loopY * arrayWidth) + loopX].yMagnitude; float Yb = motionVectorsPerturbed[(loopY * arrayWidth) + loopX].yMagnitude; if (((fabs(Xa)-fabs(Xb)) ＞ threshold) || ((fabs(Ya)-fabs(Yb)) ＞ threshold)) { finalMotionVectors[(loopY * arrayWidth) + loopX].xMagnitude = 0; finalMotionVectors[(loopY * arrayWidth) + loopX].yMagnitude = 0; } else { finalMotionVectors[(loopY * arrayWidth) + loopX].xMagnitude = motionVectorsUnperturbed[(loopY * arrayWidth) + loopX].xMagnitude; finalMotionVectors[(loopY * arrayWidth) + loopX].yMagnitude = motionVectorsUnperturbed[(loopY * arrayWidth) + loopX].yMagnitude; } } }

Figure 4A illustrates another example of momentum estimation 400 according to the present invention. As shown in FIG. 4A, the momentum estimation 400 includes a frame 402, a repeated background 404, an inserted texture image 410, at least one first motion vector 412, at least one second motion vector 414, at least one first motion vector 412, and at least one The difference 416 between the second motion vectors 414. As shown in FIG. 4A, in some aspects, at least one first motion vector 412 may be generated in the first frame subset of the frame 402. As shown in FIG. 4A, in some aspects, the first subset of frames may be positioned above the inserted texture image 410 in a portion of the repeated background 404. In other aspects, the first subset of frames can be positioned in another part of the frame 402. At least one first motion vector 412 can provide a first momentum estimate for the image data in the first subset of frames. In some aspects, the image data in the first frame subset may be disturbed. In addition, at least one second motion vector 414 can be generated based on the disturbed image data in the first frame subset. At least one second motion vector 414 can provide a second momentum estimate for the image data in the first subset of frames. In addition, at least one first motion vector 412 and at least one second motion vector 414 may be compared. In some aspects, comparing at least one first motion vector 412 with at least one second motion vector 414 may include determining a difference 416 between at least one first motion vector 412 and at least one second motion vector 414. The difference 416 can be determined to be less than or greater than the threshold value. In some aspects, the above comparison can be expressed as a function, for example, the formula: f(v ₁ ,v ₂ )=|v ₁ -v ₂ |<threshold value.

In some aspects, the at least one third motion vector used for the momentum estimation of the image data in the first frame subset may be determined based on the comparison between the at least one first motion vector 412 and the at least one second motion vector 414. In addition, when the difference 416 is less than the threshold value, at least one third motion vector may be set as at least one first motion vector 412. The difference 416 can also be referred to as delta analysis. As mentioned herein, in some aspects, the difference 416 may be an absolute value. Based on the above, at least one third vector can be expressed as the following formula: v ₃ =v ₁ , when |v ₁ -v ₂ |<the threshold value. In some aspects, when the difference 416 is greater than or equal to the threshold, at least one third motion vector may be set to have a zero motion value. Based on this, at least one third vector can also be expressed as a formula: v ₃ =0, when |v ₁ -v ₂ |≥the threshold.

In some cases, the second difference between the at least one third motion vector and one or more neighboring vectors surrounding or immediately adjacent to the at least one third motion vector may be determined. In such cases, as shown in FIG. 4A, one or more neighboring vectors next to or surrounding the third motion vector may be used to determine the second difference between the third motion vector and the neighboring vector. In addition, if the difference is greater than the threshold and/or the second difference is less than the threshold, at least one third motion vector may be set based on one or more neighboring vectors. In addition, the aforementioned threshold value can be based on the momentum estimation tolerance of the image data in the first frame subset.

Figure 4B illustrates another example of momentum estimation 450 according to the present invention. As shown in FIG. 4B, the momentum estimation 450 includes a frame 452, a repeated background 454, an inserted texture image 460, at least one first motion vector 462, at least one second motion vector 464, at least one first motion vector 462, and at least one The difference 466 between the second motion vector 464 and at least one fourth motion vector 472. In Figure 4B, as described in conjunction with Figure 4A, at least one first motion vector 462 can be generated in the first frame subset of frame 452, and at least one first motion vector 462 can be provided for the first frame subset. The first momentum estimate of the image data. As shown in FIG. 4B, in some aspects, the first subset of frames may be positioned above the inserted texture image 460 in a portion of the repeated background 454. In other aspects, the first subset of frames may be positioned in another part of frame 452. In some cases, the image data in the first frame subset may be disturbed. In addition, at least one second motion vector 464 can be generated based on the disturbed image data in the first subset of frames, where at least one second motion vector 464 can provide a second momentum estimate for the image data in the first subset of frames. In addition, at least one first motion vector 462 and at least one second motion vector 464 may be compared. In some aspects, comparing at least one first motion vector 462 with at least one second motion vector 464 may include determining a difference 466 between at least one first motion vector 462 and at least one second motion vector 464. The difference 466 can be determined to be less than or greater than the threshold. In some aspects, the at least one third motion vector used for the momentum estimation of the image data in the first frame subset may be determined based on the comparison between the at least one first motion vector 462 and the at least one second motion vector 464.

As shown in FIG. 4B, at least one fourth motion vector 472 can be determined to be used for the momentum estimation of the second frame subset of frame 452. As shown in FIG. 4B, in some aspects, the second subset of frames may be positioned below the first subset of frames in a portion of the repeating background 454 and above the texture image 460 inserted. In other aspects, the second subset of frames may be positioned in another part of frame 452. In some aspects, when the difference 466 is greater than or equal to the threshold, at least one fourth motion vector 472 can be determined. As illustrated in Figure 4B, in some aspects, the second subset of frames may be different from the first subset of frames. In addition, at least one third motion vector may be determined based on at least one first motion vector 462, at least one second motion vector 464, and at least one fourth motion vector 472.

In some aspects, perturbing the image data in the first frame subset may include modifying the magnitude of the red (R), green (G), and blue (B) (RGB) values of the image data to, for example, a value of m. In some cases, m may not be equal to zero. In addition, m may be less than or equal to 5% and greater than or equal to -5%. In addition, the image data in the first frame subset can be disturbed by a disturbance amount, and the disturbance amount can be adjusted based on the local variance of the RGB values of the image data. The local variance of the RGB values of the image data can also be based on the first momentum estimate for the image data in the first subset of frames. In addition, the local variance of the RGB value of the image data is estimated based on the previous variance of the RGB value. In other aspects, the image data in the first frame subset is disturbed by a disturbance, and the disturbance is based on the luminance (Y), first chromaticity (U), and second chromaticity (V) of the image data (YUV ) Value of local variance. In addition, image data from RGB image data can be converted into YUV image data, where the image data can be disturbed before the RGB image data is converted into YUV image data.

5A and 5B illustrate examples of momentum estimation 500 and 550 according to the present invention. As shown in FIGS. 5A and 5B, momentum estimate 500 and momentum estimate 550 include corresponding motion vectors to assist momentum estimation. More specifically, Figure 5A illustrates the momentum estimate 500 without perturbation. Therefore, the momentum estimate 500 may include incorrect motion regions 502 and 504. Figure 5B shows the momentum estimate 550 including the perturbation pass. Thus, the momentum estimate 550 may not include erroneous motion regions because these motion vectors have been identified as incorrect and removed.

Figure 6 illustrates an example flowchart 600 of an example method in accordance with one or more techniques of the present invention. The method can be executed by a device or a GPU. The method can assist the GPU to process momentum estimation.

At 602, as described in conjunction with the examples in FIGS. 2, 3, 4A, and 5B, the GPU may generate at least one first motion vector in the first subset of the frame. For example, the first motion vector may provide a first momentum estimate for the image data in the first subset of the frame. At 604, as described in conjunction with the examples in FIGS. 2, 3, 4A, and 5B, the GPU may disturb the image data. In some aspects, the GPU can disturb all image data in the frame. In other aspects, the GPU can disturb the image data in the first subset of the frame. In other aspects, the GPU may disturb the image data outside the first subset of the frame. In addition, at 606, as described in conjunction with the examples in FIGS. 2, 3, 4A, and 5B, the GPU may generate at least one second motion vector based on the disturbance image data. In some aspects, the second motion vector may provide a second momentum estimate for the image data. At 608, as described in connection with the examples in FIGS. 2, 3, 4A, and 5B, the GPU may compare the first motion vector with the second motion vector. In addition, at 610, as described in conjunction with the examples in FIG. 2, FIG. 3, FIG. 4A, FIG. 4B, and FIG. 5B, the GPU may determine the image data based on the comparison between the first motion vector and the second motion vector The momentum estimates at least one third motion vector.

In some aspects, as described in conjunction with the example in FIG. 2, when comparing the first motion vector and the second motion vector, the GPU may determine the difference between the first motion vector and the second motion vector. The difference can also be called delta analysis. In some cases, the difference may be an absolute value. In addition, as further described in conjunction with the example in FIG. 2, the GPU can determine whether the difference is less than the threshold. In addition, as described in conjunction with the examples in FIG. 2, FIG. 3, FIG. 4A, FIG. 4B, and FIG. As described in conjunction with the example in FIG. 2, when the difference is greater than the threshold, the GPU can also determine at least one fourth motion vector for momentum estimation of the second subset of the frame. In addition, the second subset of frames can be different from the first subset of frames. In some aspects, as described in conjunction with the examples in FIGS. 2, 3, 4A, 4B, and 5B, the third motion vector may be based on the determined first motion vector, second motion vector, and fourth motion Vector and judge.

In addition, in some aspects, as described in conjunction with the examples in FIGS. 2, 3, 4A, 4B, and 5B, when the difference is greater than the threshold, the third motion vector can be set to have a zero motion value . In some aspects, as described in conjunction with the examples in FIG. 2, FIG. 3, FIG. 4A, FIG. 4B, and FIG. 5B, the GPU may determine whether the third motion vector is between one or more adjacent vectors around the third motion vector. The second difference. In these aspects, when the difference is greater than the threshold and the second difference is less than the threshold, the third motion vector may be set based on one or more neighboring vectors.

In other aspects, as described in conjunction with the examples in FIG. 2, FIG. 3, FIG. 4A, FIG. 4B, and FIG. 5B, the aforementioned threshold value may also be based on the momentum estimation tolerance of the image data in the first subset of the frame. In addition, as described in conjunction with the examples in FIGS. 2, 3, 4A, 4B, and 5B, perturbing the image data in the first subset of the frame may include modifying the magnitude of the RGB value of the image data to m. In some cases, m may not be equal to zero. In addition, the absolute value of m can be between -5% and 5%, so that m can be less than or equal to 5% and greater than or equal to -5%.

In other aspects, as described in conjunction with the examples in Figure 2, Figure 3, Figure 4A, Figure 4B and Figure 5B, the image data in the first subset of the frame can be disturbed by a disturbance, where the disturbance can be based on Adjust the local variance of the RGB value of the image data. As described in conjunction with the examples in FIGS. 2, 3, 4A, 4B, and 5B, the local variance of the RGB values of the image data can be based on the first momentum estimate for the image data in the first subset of the frame. In addition, as described in conjunction with the examples in FIG. 2, FIG. 3, FIG. 4A, FIG. 4B, and FIG. 5B, the local variance of the RGB value of the image data can be estimated based on the previous variance of the RGB value. In other aspects, as described in conjunction with the examples in Figure 2, Figure 3, Figure 4A, Figure 4B and Figure 5B, the image data in the first subset of the frame can be disturbed by a disturbance, where the disturbance can be based on Adjust the local variance of the YUV value of the image data. In other aspects, as described in conjunction with the examples in Figure 2, Figure 3, Figure 4A, Figure 4B, and Figure 5B, GPU can convert image data of RGB image data into YUV image data, where the image data is transforming the RGB image Disturbance before the data is converted into YUV image data.

In one configuration, a device for momentum estimation is provided. The device can be a momentum estimation device in the GPU. In one aspect, the momentum estimation device may be the processing unit 120 in the device 104 or may be the device 104 or some other hardware in another device. The device may include means for generating at least one first motion vector in the first subset of the frame. The first motion vector may provide a first momentum estimate for the image data in the first subset of the frame. The device may also include means for disturbing the image data in the first subset of the frame. In addition, the device may include means for generating at least one second motion vector based on disturbing image data in the first subset of the frame. In some aspects, the second motion vector may provide a second momentum estimate for the image data in the first subset of the frame. The device may also include means for comparing the first motion vector with the second motion vector. In addition, the device may include means for determining at least one third motion vector for momentum estimation of the image data in the first subset of the frame based on the comparison between the first motion vector and the second motion vector.

In some aspects, the means for comparing the first motion vector and the second motion vector can be configured to determine the difference between the first motion vector and the second motion vector, and determine whether the difference is less than the threshold . The device may also include means for determining at least one fourth motion vector for momentum estimation of the second subset of the frame when the difference is greater than the threshold. The second subset of the frame may be different from the first subset of the frame, where the third motion vector is determined based on the determined first motion vector, second motion vector, and fourth motion vector. In addition, the device may include means for determining the second difference between the third motion vector and one or more neighboring vectors surrounding the third motion vector. When the difference is greater than the threshold and the second difference is less than the threshold, the third motion vector may be set based on one or more neighboring vectors. In addition, the member used to disturb the image data in the first subset of the frame can be configured to modify the magnitude of the RGB value of the image data to m, where m is not equal to zero. In addition, the device may include a component for converting image data from RGB image data into YUV image data, wherein the image data is disturbed before the RGB image data is converted into YUV image data.

The subject matter described herein can be implemented to realize one or more potential benefits or advantages. For example, the techniques described can be used by GPUs to reduce the amount of visualization workload. The system described in this article can use momentum estimation to extrapolate the frame from the previous presentation. In this way, this allows the GPU to render the frame at a reduced rate, and extrapolate the frame to be displayed at the position where the content is presented. Therefore, the present invention can reduce the rendering workload by rendering fewer frames on the GPU and using momentum estimation to fill the gaps in the image or motion vector. Therefore, the present invention can save GPU power and performance, and present higher quality frames on the GPU. In addition, the present invention can also reduce the cost of presenting content.

According to the present invention, the term "or" can be interpreted as "and/or", where the context does not indicate otherwise. In addition, although phrases such as "one or more" or "at least one" can be used for some of the features disclosed in this article (but not for others); features that are not used in this language can be interpreted as implying context and This meaning is not indicated otherwise.

In one or more examples, the functions described herein can be implemented in hardware, software, firmware, or any combination thereof. For example, although the term "processing unit" has been used throughout the present invention, the processing unit may be implemented in hardware, software, firmware, or any combination thereof. If any function, processing unit, technology described in this article, or other module is implemented in software, the function, processing unit, technology described in this article, or other module can be stored as one or more instructions or code On or through computer-readable media. Computer-readable media can include both computer data storage media and communication media. Communication media includes any media that facilitates the transfer of computer programs from one place to another. In this way, computer-readable media generally correspond to (1) non-transitory tangible computer-readable storage media, or (2) communication media such as signals or carrier waves. The data storage medium can be accessed by one or more computers or one or more processors to retrieve instructions, program codes, and/or data structures for implementing the techniques described in the present invention. By way of example and not limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices. Disks and optical discs as used in this article include compact discs (CD), laser discs, optical discs, digital versatile discs (DVD), floppy discs, and Blu-ray discs. Disks usually reproduce data magnetically, while optical discs are used The laser regenerates data optically. The combination of the above should also be included in the category of computer-readable media. The computer program product may include a computer readable medium.

Code writing can be performed by one or more processors, such as one or more digital signal processors (DSP), general-purpose microprocessors, application-specific integrated circuits (ASIC), arithmetic logic unit (ALU), field programmable logic Array (FPGA) or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein may refer to any of the foregoing structure or any other structure suitable for implementing the techniques described herein. Furthermore, these techniques can be fully implemented in one or more circuits or logic elements.

The technology of the present invention can be implemented in a variety of devices or devices, including wireless mobile phones, integrated circuits (ICs), or collections of ICs (eg, collections of chips). Various components, modules, or units are described in the present invention to emphasize the functional aspects of devices configured to perform the disclosed technology, but they do not necessarily require different hardware units to be implemented. More precisely, as described above, various units can be combined in any hardware unit, or combined by a collection of interoperable hardware units (including one or more processors as described above) Appropriate software and/or firmware provide these units.

Various examples have been described. These and other examples are within the scope of the following patent applications.

100: Content generation and coding system 102: source device 104: destination device 106: Processing Unit 107-1: Graphics processing pipeline 107-2: Graphics processing pipeline 108: content encoder 109: Internal memory 110: System memory 111: Internal memory 112: Communication interface 114: receiver 116: Transmitter 118: Transceiver 120: processing unit 121: Internal memory 122: content decoder 123: Internal memory 124: System memory 126: Communication Interface 127: display processor 128: receiver 130: transmitter 131: Display 132: Transceiver 134: Link 198: determination component 200: Momentum estimation 202: Present content block 204: RGB disturbance texture block 206: Color space conversion times 208: Color space conversion times 210: Momentum estimation 212: Momentum Estimation 214: delta or difference analysis 216: Motion vector calculation 300: Image 302: Repeat background 304: Insert texture image 310: Image 312: Repeat background 314: Insert texture image 400: Momentum estimation 402: frame 404: Repeat background 410: Import texture image 412: first motion vector 414: second motion vector 416: Difference 450: Momentum Estimate 452: frame 454: Repeat background 460: Import texture image 462: first motion vector 464: second motion vector 466: Difference 472: Fourth Motion Vector 500: Momentum estimate 502: Incorrect movement area 504: Incorrect movement area 550: Momentum Estimate 600: flow chart 602: step 604: step 606: step 608: step 610: Step

FIG. 1 is a block diagram illustrating an example content generation and coding system according to the technology of the present invention.

Figure 2 illustrates an example of momentum estimation according to the present invention.

3A and 3B illustrate examples of images on which momentum estimation is performed according to the present invention.

Figure 4A illustrates another example of momentum estimation according to the present invention.

Figure 4B illustrates another example of momentum estimation according to the present invention.

5A and 5B illustrate other examples of momentum estimation according to the present invention.

Figure 6 illustrates an example flowchart of an example method in accordance with one or more techniques of the present invention.

600: flow chart

602: step

604: step

606: step

608: step

610: Step

Claims (30)

A method of momentum estimation in a graphics processing unit (GPU), which includes: Generating at least one first motion vector in a first subset of a frame, the at least one first motion vector providing a first momentum estimate for image data in the first subset of the frame; Disturb the image data in the first subset of the frame; Generate at least one second motion vector based on the disturbed image data in the first subset of the frame, the at least one second motion vector providing a second for the image data in the first subset of the frame Momentum estimation Comparing the at least one first motion vector with the at least one second motion vector; and Based on the comparison between the at least one first motion vector and the at least one second motion vector, determine at least one third motion vector used for the momentum estimation of the image data in the first subset of the frame. Such as the method of claim 1, wherein comparing the at least one first motion vector with the at least one second motion vector includes: Determine a difference between the at least one first motion vector and the at least one second motion vector; and Determine whether the difference is less than a threshold value. Such as the method of claim 2, wherein when the difference is less than the threshold value, the at least one third motion vector is set as the at least one first motion vector. Such as the method of claim 3, which further includes determining at least one fourth motion vector used for momentum estimation of a second subset of the frame when the difference is greater than the threshold value, and the first The second subset is different from the first subset of the frame, in which the at least one third motion vector is determined based on the determined at least one first motion vector, at least one second motion vector, and at least one fourth motion vector. Such as the method of claim 2, wherein when the difference is greater than the threshold, the at least one third motion vector is set to have a zero motion value. Such as the method of claim 2, which further includes Determine a second difference between the at least one third motion vector and one or more neighboring vectors surrounding the at least one third motion vector; When the difference is greater than the threshold and the second difference is less than the threshold, the at least one third motion vector is set based on one or more neighboring vectors. Such as the method of claim 2, wherein the threshold value is based on a momentum estimation tolerance of the image data in the first subset of the frame. Such as the method of claim 1, wherein perturbing the image data in the first subset of the frame includes one of the red (R), green (G), and blue (B) (RGB) values of the image data The value is modified to m, where m is not equal to 0. Such as the method of claim 8, where m is less than or equal to 5% and greater than or equal to -5%. Such as the method of claim 1, wherein the image data in the first subset of the frame is disturbed by a disturbance, and the disturbance is based on the red (R), green (G), and blue (B) of the image data ) (RGB) value is adjusted based on the local variance. The method of claim 10, wherein the local variance of the RGB value of the image data is based on the first momentum estimate for the image data in the first subset of the frame. Such as the method of claim 10, wherein the local variance of the RGB value of the image data is estimated based on a previous variance of the RGB value. Such as the method of claim 1, wherein the image data in the first subset of the frame is disturbed by a disturbance, and the disturbance is based on the brightness (Y), the first chromaticity (U), and the first chromaticity (U) of the image data The dichromaticity (V) (YUV) value is adjusted by one of the local variances. Such as the method of claim 1, which further includes converting the image data from red (R), green (G), blue (B) (RGB) image data into brightness (Y), first chromaticity (U), The second chromaticity (V) (YUV) image data, wherein the image data is disturbed before the RGB image data is converted into the YUV image data. A device for momentum estimation in a graphics processing unit (GPU), which includes: A memory; and At least one processor coupled to the memory and configured to: Generating at least one first motion vector in a first subset of a frame, the at least one first motion vector providing a first momentum estimate for image data in the first subset of the frame; Disturb the image data in the first subset of the frame; Generate at least one second motion vector based on the disturbed image data in the first subset of the frame, the at least one second motion vector providing a second for the image data in the first subset of the frame Momentum estimation Comparing the at least one first motion vector with the at least one second motion vector; and Based on the comparison between the at least one first motion vector and the at least one second motion vector, determine at least one third motion vector for the momentum estimation of the image data in the first subset of the frame. The device of claim 15, wherein comparing the at least one first motion vector with the at least one second motion vector includes the at least one processor, which is further configured to: Determine a difference between the at least one first motion vector and the at least one second motion vector; and Determine whether the difference is less than a threshold value. Such as the device of claim 16, wherein when the difference is less than the threshold value, the at least one third motion vector is set as the at least one first motion vector. Such as the device of claim 17, wherein the at least one processor is further configured to: When the difference is greater than the threshold value, determine at least one fourth motion vector for momentum estimation of a second subset of the frame, the second subset of the frame being different from the one of the frame The first subset, wherein the at least one third motion vector is determined based on the determined at least one first motion vector, at least one second motion vector, and at least one fourth motion vector. Such as the device of claim 16, wherein when the difference is greater than the threshold, the at least one third motion vector is set to have a zero motion value. Such as the device of claim 16, wherein the at least one processor is further configured to: Determine a second difference between the at least one third motion vector and one or more neighboring vectors surrounding the at least one third motion vector; When the difference is greater than the threshold and the second difference is less than the threshold, the at least one third motion vector is set based on the one or more neighboring vectors. Such as the device of claim 16, wherein the threshold value is based on a momentum estimation tolerance of the image data in the first subset of the frame. Such as the device of claim 15, wherein perturbing the image data in the first subset of the frame includes the at least one processor, which is further configured to red (R), green (G) and the image data One of the blue (B) (RGB) values is modified to m, where m is not equal to zero. Such as the device of claim 22, where m is less than or equal to 5% and greater than or equal to -5%. Such as the device of claim 15, wherein the image data in the first subset of the frame is disturbed by a disturbance, and the disturbance is based on the red (R), green (G), and blue (B) of the image data ) (RGB) value is adjusted based on the local variance. Such as the device of claim 24, wherein the local variance of the RGB value of the image data is based on the first momentum estimate for the image data in the first subset of the frame. Such as the device of claim 24, wherein the local variance of the RGB value of the image data is estimated based on a previous variance of the RGB value. Such as the device of claim 15, wherein the image data in the first subset of the frame is disturbed by a disturbance, and the disturbance is based on the luminance (Y), the first chromaticity (U), and the first chromaticity (U) of the image data The dichromaticity (V) (YUV) value is adjusted by one of the local variances. Such as the device of claim 15, wherein the at least one processor is further configured to convert the image data from red (R), green (G), blue (B) (RGB) image data into brightness (Y), First chromaticity (U) and second chromaticity (V) (YUV) image data, wherein the image data is disturbed before converting the RGB image data into the YUV image data. A device for momentum estimation in a graphics processing unit (GPU), which includes: A component for generating at least one first motion vector in a first subset of a frame, the at least one first motion vector providing a first momentum estimate for image data in the first subset of the frame ； Component for disturbing the image data in the first subset of the frame; A component for generating at least one second motion vector based on the disturbed image data in the first subset of the frame, the at least one second motion vector providing the image data for the first subset of the frame One second momentum estimate; A component for comparing the at least one first motion vector with the at least one second motion vector; and Means for determining, based on the comparison between the at least one first motion vector and the at least one second motion vector, at least one of the first momentum estimates used for the image data in the first subset of the frame Three motion vectors. A computer readable medium storing computer executable code for momentum estimation in a graphics processing unit (GPU), which contains code to: Generating at least one first motion vector in a first subset of a frame, the at least one first motion vector providing a first momentum estimate for image data in the first subset of the frame; Disturb the image data in the first subset of the frame; Generate at least one second motion vector based on the disturbed image data in the first subset of the frame, the at least one second motion vector providing a second for the image data in the first subset of the frame Momentum estimation Comparing the at least one first motion vector with the at least one second motion vector; and Based on the comparison between the at least one first motion vector and the at least one second motion vector, determine at least one third motion vector for the momentum estimation of the image data in the first subset of the frame.

Images