KR101084806B1 - Video processing - Google Patents

Abstract

A delay tolerant system for performing video processing operations is described. A video processor with scalar and vector components, stream processing in the video processor, and multidimensional data path processing in the video processor are also described.

Description

Video Processing {VIDEO PROCESSING}

The present invention relates to a digital electronic computer system. Specifically, the present invention relates to a system for efficiently processing video information on a computer system. In one aspect, a delay tolerant system for performing video processing operations is described. In another aspect, stream processing in a video processor is described. Also described is multidimensional data path processing in a video processor. Also described is a video processor with scalar and vector components.

This application claims 35 U.S.C. Claims under Section 119 (e), the entire contents of this provisional application, are incorporated herein.

Display of images and full-motion video is an area of the electronics industry that has developed significantly in recent years. Display and rendering of high-quality video, especially high-definition digital video, is the main goal of modern video technology applications and devices. Video technology is used in a variety of products, including cellular telephones, personal video recorders, digital video projectors, and high definition televisions. The emergence and growth of devices capable of producing and displaying high definition video is an area of the electronics industry that is undergoing a high degree of innovation and progress.

Video devices, which are prevalent in many consumer electronic and professional level devices, rely on one or more video processors to format and / or enhance the video signal for display. This is especially true for digital video applications. For example, one or more video processors are contained within a typical set top box and used to convert HDTV broadcast signals into video signals available by the display. Such conversion includes, for example, scaling in which the video signal is converted from a non-16x9 video image for proper display on a true 16x9 (eg, wide screen) display. One or more video processors may be used to perform a scan conversion in which the video signal is converted from an interlace format in which odd and even scan lines are displayed separately, to a progressive format in which the entire frame is drawn within a single sweep.

Further examples of video processor applications include, for example, signal decompression where a video signal is received in a compressed format (eg, MPEG-2) and decompressed and formatted for display. Another example is re-interlacing scan conversion, which converts an incoming digital video signal from a DVI (Digital Visual Interface) format to a composite video format that is compatible with many older television displays on the market. It involves doing.

More sophisticated users can find, for example, in-loop / out-loop deblocking filters, improved motion adaptive deinterlacing, input noise filtering for encoding operations, multiphase scaling / resampling, sub-picture synthesis, and color space conversion, adjustment, pixel point It requires more sophisticated video processor functions such as manipulation (e.g., sharpening, histogram adjustment, etc.) and processor-amplifier operations such as various video surface format conversion support operations.

The problem with providing these sophisticated video processor features is that video processors with architectures that are powerful enough to implement these features can be overly expensive to include in many types of devices. The more sophisticated the video processing capabilities, the more expensive the integrated circuits needed to achieve these capabilities will be in terms of silicon die area, transistor count, memory speed requirements, and so on.

Thus, conventional system designers had to find a tradeoff in terms of video processor performance and cost. Conventional video processors, which are widely considered to have an acceptable cost / performance ratio, often have delay limitations (e.g., to prevent video shake or malfunction of video processing applications) and computational densities (e.g. dies). It was hardly enough in terms of the number of processor operations per square millimeter). Moreover, conventional video processors are generally suitable for linear scaling performance requirements, such as where a video device is expected to process multiple video streams (eg, simultaneous processing of multiple receive streams and outgoing display streams). Not.

Therefore, there is a need for a new video processor system that overcomes the limitations of the prior art. The new video processor system must be scalable and have high computational density to handle the sophisticated video processor functions expected by more sophisticated users.

Embodiments of the present invention provide a new video processor system that supports sophisticated video processing functions while efficiently utilizing integrated circuit silicon die area, transistor count, memory speed requirements, and the like. Embodiments of the present invention maintain high computational density to process multiple video streams and are easily scalable.

In one embodiment, a delay tolerant system for implementing video processing operations in a video processor is implemented. The system includes a host interface for implementing communication between the video processor and the host CPU, a scalar execution unit connected to the host interface, configured to execute a scalar video processing operation, and a vector connected to the hose and configured to execute vector video processing operation. An execution unit. A command FIFO is included to enable the vector execution unit to operate on a request drive basis by accessing the memory command FIFO. A memory interface is included for implementing communication between the video processor and the frame buffer memory. A DMA engine is built in the memory interface for implementing DMA transfers between a plurality of different memory locations and for loading data and instructions for the vector execution unit into the data storage memory and the instruction cache.

In one embodiment, the vector execution unit is configured to operate asynchronously with respect to the scalar execution unit by accessing the command FIFO to operate on demand drive basis. The request drive base may be configured to hide delays in data transfer from the vector execution unit to the command FIFO at different memory locations (eg, frame buffer memory, system memory, cache memory, etc.). The command FIFO may be a pipeline FIFO to prevent the functioning of the vector execution unit.

In one embodiment, the present invention is implemented as a video processor for performing video processing operations. The video processor includes a host interface for implementing communication between the video processor and the host CPU. The video processor includes a memory interface for implementing communication between the video processor and the frame buffer memory. A scalar execution unit is coupled to the host interface and the memory interface and configured to execute a scalar video processing operation. The vector execution unit is coupled to the host interface and the memory interface and is configured to execute the vector video processing operation. The video processor may be a standalone video processor integrated circuit or a component integrated within the GPU integrated circuit.

In one embodiment, the scalar execution unit functions as a controller of the video processor to control the operation of the vector execution unit. The scalar execution unit may be configured to execute the flow control algorithm of the application, and the vector execution unit may be configured to execute the pixel processing operation of the application. A vector interface unit for interfacing the scalar execution unit and the vector execution unit may be included in the video processor. In one embodiment, the scalar execution unit and the vector execution unit are configured to operate asynchronously. The scalar execution unit may execute at a first clock frequency, and the vector execution unit may execute at a different clock frequency (eg, faster, slower, etc.). The vector execution unit may operate on a request driving basis under the control of the scalar execution unit.

In one embodiment, the present invention is implemented as a multidimensional data path processing system for a video processor to perform video processing operations. The video processor includes a scalar execution unit configured to execute a scalar video processing operation and a vector execution unit configured to execute a vector video processing operation. A data storage memory for storing data for the vector execution unit is included. The data storage memory includes a plurality of tiles with symmetrical bank data structures arranged in an array. Bank data structures are configured to support access to different tiles in each bank.

Depending on the requirements of a particular configuration, each of the bank data structures may include a plurality of tiles (eg, 4x4, 8x8, 8x16, 16x24, etc.). In one embodiment, the banks are configured to support access to different tiles in each bank. This allows a single access to retrieve a row or column of tiles from two adjacent banks. In one embodiment, crossbars are used to select a configuration for accessing tiles (eg, rows, columns, blocks, etc.) of a plurality of bank data structures. A collector may be included for receiving the tiles of the banks accessed by the crossbar and for providing the tiles to the front end of the vector data path in clock units.

In one embodiment, the present invention is implemented as a stream based memory access system for a video processor. The video processor includes a scalar execution unit configured to execute a scalar video processing operation and a vector execution unit configured to execute a vector video processing operation. Frame buffer memory for storing data for the scalar execution unit and the vector execution unit is included. A memory interface for implementing communication between the scalar execution unit and the vector execution unit and the frame buffer memory is included. The frame buffer memory includes a plurality of tiles. The memory interface implements a first stream of first sequential tile access to a scalar execution unit and a second stream of second sequential tile access to a vector execution unit.

In one embodiment, the first stream and the second stream comprise a series of sequential prefetch tiles that are prefetched in a manner that hides access delays from source memory locations (eg, frame buffer memory, system memory, etc.). In one embodiment, the memory interface is configured to manage a plurality of different streams from a plurality of different origin locations and to a plurality of different end locations. In one embodiment, a DMA engine built in a memory interface is used to implement multiple memory reads and multiple memory writes to support multiple streams.

In general, the present invention discloses at least the following four methods.

A) A method broadly taught herein includes a method of a multidimensional data path processing system in a video processor for performing a video processing operation, the method comprising: performing a scalar video processing operation using a scalar execution unit; Executing a vector video processing operation using the vector execution unit; Storing data for the vector execution unit using the data storage memory, wherein the data storage memory includes a plurality of tiles including symmetrical bank data structures arranged in an array, wherein the bank data structures differ in each bank; Configured to support access to tiles. In addition, the method A includes bank data structures each including a plurality of tiles arranged in a 4x4 pattern. The method A also includes bank data structures each including a plurality of tiles arranged in an 8x8, 8x16 or 16x24 pattern. In addition, the method A includes bank data structures configured to support access to different tiles of each bank data structure, and at least one access includes two adjacent banks comprising a row of tiles of two bank data structures. For data structures. The method A also includes tiles configured to support access to different tiles of each bank data structure, wherein at least one access comprises a column of tiles of two adjacent bank data structures. It is about. The method A also includes selecting a configuration for accessing tiles of the plurality of bank data structures using a crossbar connected to the data storage memory. In this selection step, the crossbar accesses the tiles of the plurality of bank data structures to provide data to the vector data path on a clock basis. The method also includes receiving tiles of a plurality of bank data structures accessed by a crossbar using a collector, and providing tiles to the front end of the vector data path in clock units.

B) The methods broadly taught herein are also methods for performing video processing operations, implemented using a video processor of a computer system executing computer readable code, and using a host interface to provide a video processor and a host CPU. Establishing communication between them; Establishing communication between the video processor and the frame buffer memory using the memory interface; Executing a scalar video processing operation using a scalar execution unit coupled to the host interface and the memory interface; And executing a vector video processing operation using the vector execution unit connected to the host interface and the memory interface. The method B further includes a scalar execution unit that functions as a controller of the video processor to control the operation of the vector execution unit. The method B further includes a vector interface unit for interfacing the scalar execution unit and the vector execution unit. The method B further includes a scalar execution unit and a vector execution unit configured to operate asynchronously. Also, the scalar execution unit executes at the first clock frequency, and the vector execution unit executes at the second clock frequency. The method B includes a scalar execution unit configured to execute an application's flow control algorithm and a vector execution unit configured to execute an application's pixel processing operation. In addition, the vector execution unit is configured to operate on a request driving basis under the control of the scalar execution unit. In addition, the scalar execution unit is configured to send function calls to the vector execution unit using the memory command FIFO, the vector execution unit operating on a request drive basis by accessing the memory command FIFO. In addition, the asynchronous operation of the video processor is configured to support individual independent updates of the vector subroutine or scalar subroutine of the application. Finally, the method B includes a scalar execution unit configured to operate using very long instruction word (VLIW) code.

C) The method described herein also broadly teaches a method for stream-based memory access in a video processor for executing a video processing operation, which executes a scalar video processing operation using a scalar execution unit. Making; Executing a vector video processing operation using the vector execution unit; Storing data for the scalar execution unit and the vector execution unit using the frame buffer memory; And implementing communication between the scalar execution unit and the vector execution unit and the frame buffer memory using a memory interface, wherein the frame buffer memory comprises a plurality of tiles, and the memory interface is a vector execution unit or a scalar execution unit Implement a first stream comprising a first sequential tile access, and implement a second stream comprising a second sequential tile access. The method C also has a first stream and a second stream comprising at least one prefetch tile. The method C further includes a first stream originating from a first location in the frame buffer memory, and a second stream originating from a second location of the frame buffer memory. The method C further includes a memory interface configured to manage the plurality of streams from the plurality of different origin locations and to the plurality of different end locations. In this regard, at least one of the origin positions or at least one of the end positions is in system memory. The method C implements a plurality of memory reads to support the first and second streams using a DMA engine built in a memory interface, and implements a plurality of memory writes to support the first and second streams. It further comprises the step. Also, the method C includes a first stream that experiences a higher amount of delay than the second stream, and the first stream includes a larger number of tile storage buffers than the second stream. The method C also includes a memory interface configured to prefetch an adjustable number of tiles of the first or second stream to compensate for the delay of the first or second stream.

D) The method described herein also broadly includes a method for delay tolerance video processing operation, the method comprising implementing communication between a video processor and a host CPU using a host interface; Executing a scalar video processing operation using a scalar execution unit connected to the host interface; Executing a vector video processing operation using a vector execution unit coupled to the host interface; Enabling the vector execution unit to operate on a request drive basis by accessing the memory command FIFO; Implementing communication between the video processor and the frame buffer memory using a memory interface; And implementing DMA transfers between a plurality of different memory locations using a DMA engine built within the memory interface and configured to load data and instructions for the vector execution unit into the data storage memory and the instruction cache. The method D further includes a vector execution unit configured to operate asynchronously with respect to the scalar execution unit by accessing the command FIFO to operate on demand drive basis. The method D further includes a request drive base configured to hide a delay of data transfer from the vector execution unit to the command FIFO at different memory locations. In addition, the method D includes a scalar execution unit configured to implement algorithmic flow control processing, wherein the vector execution unit is configured to implement most of the video processing workload. In this regard, the scalar execution unit is configured to precalculate the working parameters for the vector execution unit to hide the data propagation delay. The method D includes a vector execution unit configured to schedule a memory read through the DMA engine to prefetch a command for subsequent execution of the vector subroutine. Here, the memory read is scheduled to prefetch a command for execution of the vector subroutine prior to the call of the vector subroutine by the scalar execution unit.

According to another feature of the invention, a first aspect comprises: a scalar execution unit configured to execute a scalar video processing operation; A vector execution unit configured to execute a vector video processing operation; And a data storage memory for storing data for the vector execution unit, wherein the data storage memory includes a plurality of tiles including symmetrical bank data structures arranged in an array, wherein the bank data structures comprise respective banks. Is a system configured to support access to different tiles of.

In a second aspect of the present invention, in a first aspect, the system is a multidimensional data path processing system for a video processor for performing video processing operations.

A third aspect of the invention provides a system for multidimensional data path processing for supporting video processing operations, comprising: a motherboard; A host CPU connected to the motherboard; And a video processor coupled to the motherboard and the CPU and comprising the system of claim 1.

In a fourth aspect of the invention, in any of the first to third aspects, each of the bank data structures comprises a plurality of tiles arranged in a 4x4 pattern.

In a fifth aspect of the invention, in any one of the first to third aspects, each of the bank data structures comprises a plurality of tiles arranged in an 8x8 or 8x16 or 16x24 pattern.

In a sixth aspect of the invention, in any one of the first to third aspects, the bank data structures are configured to support access to different tiles of each bank data structure, wherein at least one access is two. For two adjacent bank data structures comprising a row of tiles of the said bank data structures.

In a seventh aspect of the invention, in any one of the first to third aspects, the tiles are configured to support access to different tiles of each bank data structure, with at least one access being two contiguous. For two adjacent bank data structures containing a column of tiles of the bank data structures.

An eighth aspect of the invention, in any one of the first to third aspects, further comprises a crossbar connected to the data storage memory and for selecting a configuration for accessing tiles of the plurality of bank data structures. do.

In a ninth aspect of the invention, in an eighth aspect, the crossbar accesses the tiles of the plurality of bank data structures to provide data on a clock-by-clock basis to the vector data path.

A tenth aspect of the present invention provides a collector that, in a ninth aspect, receives tiles of the plurality of bank data structures accessed by the crossbar and provides the tiles on a clock-by-clock basis to a front end of a vector data path. It includes more.

An eleventh aspect of the present invention provides a video processor for performing a video processing operation, comprising: a host interface for implementing communication between the video processor and a host CPU; A memory interface for implementing communication between the video processor and frame buffer memory; A scalar execution unit coupled to the host interface and the memory interface and configured to execute a scalar video processing operation; And a vector execution unit coupled to the host interface and the memory interface, the vector execution unit configured to execute a vector video processing operation.

A twelfth aspect of the present invention is a system for performing a video processing operation, comprising: a motherboard; A host CPU connected to the motherboard; And the video processor of the eleventh aspect connected to the motherboard and the CPU.

In a thirteenth aspect of the present invention, in the eleventh aspect, the scalar execution unit functions as a controller of the video processor and controls the operation of the vector execution unit.

A fourteenth aspect of the present invention, in the eleventh aspect, further includes a vector interface unit for interfacing the scalar execution unit with the vector execution unit.

A fifteenth aspect of the present invention is the eleventh aspect, wherein the scalar execution unit and the vector execution unit are configured to operate asynchronously.

A sixteenth aspect of the present invention is the video processor of the fifteenth aspect or the system of the twelfth aspect, wherein the scalar execution unit executes at a first clock frequency and the vector execution unit executes at a second clock frequency.

A seventeenth aspect of the present invention is the video processor of the eleventh aspect or the system of the twelfth aspect, wherein the scalar execution unit is configured to execute a flow control algorithm of an application, and the vector execution unit performs a pixel processing operation of the application. Configured to execute.

In an eighteenth aspect of the invention, in a seventeenth aspect, the vector execution unit is configured to operate on a demand driven basis under control of the scalar execution unit.

In a nineteenth aspect of the present invention, in a seventeenth aspect, the scalar execution unit is configured to send function calls to the vector execution unit using a command FIFO, the vector execution unit being the command FIFO. It operates on a request driven basis by accessing it.

In a twentieth aspect of the invention, in the seventeenth aspect, the asynchronous operation of the video processor is configured to support individual independent update of the vector subroutine or the scalar subroutine of the application.

A twenty-first aspect of the present invention is the eleventh aspect, wherein the scalar execution unit is configured to operate using very long instruction word (VLIW) code.

A twenty second aspect of the present invention provides a stream based memory access system for a video processor for executing a video processing operation, comprising: a scalar execution unit configured to execute a scalar video processing operation; A vector execution unit configured to execute a vector video processing operation; A frame buffer memory for storing data for the scalar execution unit and the vector execution unit; And a memory interface for implementing communication between the scalar execution unit and the vector execution unit and the frame buffer memory, the frame buffer memory comprising a plurality of tiles, wherein the memory interface comprises the vector execution unit or the scalar A system for implementing a first stream comprising a first sequential tile access to an execution unit and implementing a second stream comprising a second sequential tile access.

A twenty third aspect of the present invention is a system for executing stream based memory accesses to support a video processing operation, comprising: a motherboard; A host CPU connected to the motherboard; A video processor coupled to the motherboard and the CPU, wherein the video processor is a host interface for establishing communication between the video processor and the host CPU, a scalar execution unit coupled to the host interface and configured to execute a scalar video processing operation A vector execution unit coupled to the host interface and configured to execute a vector video processing operation; And a memory interface coupled to the scalar execution unit and the vector execution unit, for establishing stream based communication between the scalar execution unit and the vector execution unit and a frame buffer memory, wherein the frame buffer memory includes a plurality of tiles. Wherein the memory interface implements a first stream comprising a first sequential tile access to the vector execution unit or the scalar execution unit and implements a second stream comprising a second sequential tile access.

In a twenty-fourth aspect of the invention, in the twenty-second aspect, the first stream and the second stream comprise at least one prefetched tile.

In a twenty-fifth aspect of the present invention, in a twenty-second aspect, the first stream is derived from a first position in the frame buffer memory, and the second stream is derived from a second position in the frame buffer memory.

In a twenty sixth aspect of the present invention, in a twenty-second or twenty-third aspect, the memory interface is configured to manage a plurality of streams from a plurality of different origin positions and to a plurality of different end positions.

In a twenty seventh aspect of the present invention, in the twenty sixth aspect, at least one of the origin positions or at least one of the end positions is in system memory.

A twenty eighth aspect of the present invention is directed to the twenty-second or twenty-third aspect of the present invention, wherein the twenty-second or twenty-third aspect is implemented in the memory interface to implement a plurality of memory reads to support the first stream and the second stream, And a DMA engine configured to implement a plurality of memory writes to support the second stream.

In a twenty-ninth aspect of the invention, in the twenty-second or twenty-third aspect, the first stream experiences a higher amount of delay than the second stream, and the first stream has a greater number of tiles than the second stream. Integrate storage buffers.

In a thirtieth aspect of the invention, in the twenty-second or twenty-third aspect, the memory interface further comprises an adjustable number of the first stream or the second stream to compensate for the delay of the first stream or the second stream. Configured to prefetch the tile.

A thirty first aspect of the invention is a host interface for implementing communication between a video processor and a host CPU; A scalar execution unit coupled to the host interface and configured to execute a scalar video processing operation; A vector execution unit coupled to the host interface and configured to execute a vector video processing operation; A command FIFO, comprising: a command FIFO that enables the vector execution unit to operate on a request drive basis by accessing the command FIFO; A memory interface for implementing communication between the video processor and frame buffer memory; And a DMA engine built in the memory interface, for implementing DMA transfers between a plurality of different memory locations, and for loading data and instructions for the vector execution unit into a data storage memory and an instruction cache.

In a thirty-second aspect of the present invention, in a thirty-first aspect, the system is a latency tolerant system for performing video processing operations.

A thirty-third aspect of the invention is the thirty-second aspect of the present invention comprising: a motherboard; A host CPU connected to the motherboard; It further comprises a video processor coupled to the motherboard and the CPU.

A thirty-fourth aspect of the present invention is the method of any one of thirty-first through thirty-third aspects, wherein the vector execution unit is configured to operate asynchronously with respect to the scalar execution unit by accessing the command FIFO to operate on a request driving basis. It is composed.

In a thirty-fifth aspect of the invention, in any one of thirty-first through thirty-third aspects, the request drive base is configured to hide a delay of data transfer from the different memory locations to the command FIFO of the vector execution unit.

A thirty sixth aspect of the present invention is the subject matter of any one of thirty-first to thirty-third aspects, wherein the scalar execution unit is configured to implement algorithmic flow control processing, and wherein the vector execution unit is configured to implement most of the video processing workload. It is composed.

In a thirty-seventh aspect of the present invention, in the thirty-sixth aspect, the scalar execution unit is configured to precalculate the work parameters for the vector execution unit to hide the data propagation delay.

In a thirty-eighth aspect of the present invention, in a thirty-first aspect, the vector execution unit is configured to schedule a memory read through the DMA engine to prefetch commands for subsequent execution of a vector subroutine.

In a thirty-third aspect of the present invention, in the thirty-eighth aspect, the memory read is scheduled to prefetch commands for execution of the vector subroutine prior to a call to the vector subroutine by the scalar execution unit.

In a forty-third aspect of the invention, in a thirty-third aspect, the vector execution unit is configured to schedule a memory read through the DMA engine to prefetch commands for subsequent execution of a vector subroutine, wherein the memory read is It is scheduled to prefetch commands for execution of the vector subroutine prior to the call to the vector subroutine by the scalar execution unit.

Embodiments of the present invention provide a new video processor architecture that supports sophisticated video processing functions while efficiently utilizing integrated circuit silicon die area, transistor count, memory speed requirements, and the like. Embodiments of the present invention maintain a high computational density to process multiple video streams and are easily scalable. Embodiments of the present invention provide for example MPEG-2 / WMV9 / H.264 encode support (e.g., in-loop decoder), MPEG-2 / WMV9 / H.264 decode (e.g. post entropy decoding). And a number of sophisticated video processing operations such as in-loop and out-of-loop non-blocking filters.

Additional video processing operations provided by embodiments of the present invention include, for example, improved motion adaptive deinterlacing, input noise filtering for encoding, polyphase scaling / resampling, and subpicture synthesis. The video processor architecture of the present invention is also used in certain video processor-amplifier (procamp) applications, such as pixel point manipulation such as color space conversion, color space adjustment, sharpening, histogram adjustment, and various video surface format conversions, for example. Can be.

The invention is illustrated by way of example and not by way of limitation, like reference numerals in the accompanying drawings indicate like elements.
1 is a schematic diagram illustrating basic components of a computer system according to an embodiment of the present invention.
2 illustrates internal components of a video processor according to an embodiment of the present invention.
3 illustrates an exemplary software program for a video processor in accordance with one embodiment of the present invention.
4 illustrates an example of subpicture mixing with video using a video processor in accordance with an embodiment of the present invention.
5 illustrates an internal component of a vector execution unit according to an embodiment of the present invention.
6 illustrates a layout of a data storage memory having a symmetric tile array in accordance with one embodiment of the present invention.

Reference is now made in detail to preferred embodiments of the invention, examples of which are illustrated in the accompanying drawings. While the invention has been described in connection with the preferred embodiments, it will be understood that the invention is not limited to these embodiments. Rather, the invention is intended to cover alternatives, modifications and equivalents that may be included within the spirit and scope of the invention as defined by the appended claims. Moreover, in the following detailed description of embodiments of the invention, various specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail in order not to unnecessarily obscure aspects of embodiments of the present invention.

Symbols and terms

Some portions of the detailed description that follows are provided using procedures, steps, logical blocks, processing, and other symbolic representations of operations on data bits in computer memory. These descriptions and representations are the means used by those in the data processing arts to most effectively convey the substance of their work to others. In this specification, generally, a procedure, computer execution step, logical block, processor, or the like is considered to be a non-contradictory sequence of steps or instructions that produce a desired result. These steps are those requiring physical manipulation of physical quantities. Typically, but not necessarily, these quantities take the form of electrical or magnetic signals that can be stored, transmitted, combined, compared, and manipulated in a computer system. Referencing these signals by bits, values, elements, symbols, characters, terms, numbers, etc. has sometimes proved to be convenient primarily for general use.

However, it should be noted that both these and similar terms are associated with appropriate physical quantities and are merely convenient labels applied to these quantities. As will be apparent from the description below, unless stated otherwise, and through this invention, descriptions using terms such as "processing" or "access" or "execution" or "storage" or "rendering", etc., shall be understood as registers of a computer system. And a computer system that manipulates and converts data represented by physical (electronic) quantities in memory into computer system memory or registers or other data similarly represented by physical quantities in other such information storage, transmission, or display devices (eg, It is understood to refer to the operation and process of computer system 100 of FIG. 1) or similar electronic computing device.

Computer system platform

1 illustrates a computer system 100 in accordance with one embodiment of the present invention. Computer system 100 represents components of a basic computer system in accordance with embodiments of the present invention that provide an execution platform for certain hardware-based and software-based functions. In general, computer system 100 includes at least one CPU 101, system memory 115, at least one graphics processor unit (GPU) 110, and one video processor unit (VPU) 111. The CPU 101 may be connected to the system memory 115 through the bridge component 105 or directly to the system memory 115 through a memory controller (not shown) inside the CPU 101. Bridge component 105 (eg, Northbridge) may support expansion buses that connect various I / O devices (eg, one or more hard disk drives, Ethernet adapters, CD ROM, DVD, etc.). GPU 110 and video processor unit 111 are connected to display 112. One or more additional GPUs may optionally be connected to the system 100 to further improve computing power. GPU 110 and video processor unit 111 are coupled to CPU 101 and system memory 115 through bridge component 105. System 100 may be implemented, for example, as a desktop computer system or server computer system with a powerful general purpose CPU 101 connected to a dedicated graphics rendering GPU 110. In such embodiments, components that add peripheral buses, special graphics memory and system memory, I / O devices, and the like may be included. Similarly, system 100 may be available from handheld devices (e.g., cell phones, etc.) or from Xbox (registered trademark) available from Microsoft Corporation, for example, Washington Redmond, or from Sony Computer Entertainment, Inc., Tokyo, Japan. Can be implemented as a set-top video game console device such as PlayStation3 (registered trademark).

GPU 110 is a discrete graphics card designed to be connected to a computer system via discrete components, connectors (e.g., AGP slots, PCI Express slots, etc.), discrete integrated circuit dies (e.g., mounted directly on the motherboard). Or, may be implemented as an integrated GPU (eg, integrated within bridge chip 105) contained within an integrated circuit die of a computer system chipset component. In addition, local graphics memory may be included for GPU 110 for high bandwidth graphics data storage. In addition, GPU 110 and video processor unit 111 may be integrated on the same integrated circuit die (eg, as component 120) or connected to or mounted on a motherboard of computer system 100. It should be understood that they may be circuit components.

Of the present invention Examples

2 is a diagram illustrating internal components of a video processor unit 111 according to an embodiment of the present invention. As shown in FIG. 2, the video processor unit 111 includes a scalar execution unit 201, a vector execution unit 202, a memory interface 203, and a host interface 204.

In the embodiment of FIG. 2, a video processor unit (hereinafter simply referred to as a video processor) 111 includes functional components for performing a video processing operation. The video processor 111 establishes communication between the video processor 111 and the host CPU 101 via the bridge 105 using the host interface 204. The video processor 111 uses the memory interface 203 to establish communication between the video processor 111 and the frame buffer memory 205 (eg, for a connected display 112 not shown). The scalar execution unit 201 is coupled to the host interface 204 and the memory interface 203, and is configured to execute a scalar video processing operation. The vector execution unit is coupled to the host interface 204 and the memory interface 203 and configured to execute a vector video processing operation.

2 illustrates a manner in which the video processor 111 divides its execution function into a scalar operation and a vector operation. Scalar operations are implemented by the scalar execution unit 201. Vector operation is implemented by the vector execution unit 202.

In one embodiment, the vector execution unit 202 is configured to function as a slave coprocessor for the scalar execution unit 201. In this embodiment, the scalar execution unit manages the workload of the vector execution unit 202 by supplying a control stream to the vector execution unit 202 and managing data input and output to the vector execution unit 202. Control streams generally include function parameters, subroutine independent variables, and the like. In an exemplary video processing application, the control flow of the processing algorithm of the application is executed on the scalar execution unit 210 while the actual pixel / data processing operation is implemented on the vector execution unit 202.

With continued reference to FIG. 2, scalar execution unit 201 may be implemented as a RISC style scalar execution unit that includes a RISC-based execution technique. The vector execution unit 202 can be implemented, for example, as a SIMD machine with one or more SIMD pipelines. For example, in the SIMD pipeline embodiment, each SIMD pipeline is implemented with a 16 pixel wide data path (or wider), thus outputting up to 32 pixels per clock to the vector execution unit 202. It can provide pure computing power to generate In one embodiment, scalar execution unit 201 includes hardware configured to operate using VLIW software code to optimize parallel execution of scalar operations on a clock basis.

In the embodiment of FIG. 2, the scalar execution unit 201 includes an instruction cache 211 and a data cache 212 coupled to the scalar processor 210. Caches 211 and 212 interface with memory interface 203 for access to external memory, such as, for example, frame buffer 205. The scalar execution unit 201 further includes a vector interface unit 213 for establishing communication with the vector execution unit 202. In one embodiment, the vector interface unit 213 may include one or more synchronous mailboxes 214 configured to enable asynchronous communication between the scalar execution unit 201 and the vector execution unit 202.

In the embodiment of FIG. 2, the vector execution unit 202 includes a vector control unit 220 configured to control the operation of the vector execution data path, ie the vector data path 221. The vector control unit 220 includes a command FIFO 225 for receiving instructions and data from the scalar execution unit 201. The instruction cache is connected to provide instructions to the vector control unit 220. Data storage memory 223 is coupled to provide input data to the vector data path 221 and to receive the resulting data from the vector data path 221. The data storage memory 223 functions as an instruction cache and data RAM for the vector data path 221. Instruction cache 222 and data storage memory 223 are coupled to memory interface 203 to access external memory, such as frame buffer 205. The embodiment of FIG. 2 also shows a second data vector path 231 and each second data storage memory 223 (eg, a dotted line). The second vector data path 231 and the second data storage memory 233 are shown to represent the case where the vector execution unit 202 has two vector execution pipelines (eg, dual SIMD pipeline configurations). You must understand that. Embodiments of the present invention are suitable for vector execution units having a large number of vector execution pipelines (eg, 4, 8, 16, etc.).

The scalar execution unit 201 provides data and command inputs to the vector execution unit 202. In one embodiment, scalar execution unit 201 sends function calls to vector execution unit 202 using memory mapped command FIFO 225. The vector execution unit 202 command is queued in the command FIFO 225.

The use of command FIFO 225 effectively separates scalar execution unit 201 and vector execution unit 202. The scalar execution unit 201 can function on its own individual clock, operating at its own individual clock frequency that is different from the clock frequency of the vector execution unit 202 and can be controlled individually.

The command FIFO 225 enables the vector execution unit 202 to operate as a request drive unit. For example, a job may be handed off from scalar execution unit 201 to command FIFO 225 and then accessed by vector execution unit 202 for processing in a detached asynchronous manner. The vector execution unit 202 handles its workload in this way by the scalar execution unit 201 as needed or as required. This function enables the vector execution unit 202 to conserve capacity when maximum performance is not required (eg, by reducing or stopping one or more internal clocks).

The division into a scalar portion (e.g., for execution by the scalar execution unit 201) and a vector portion (e.g., for execution by the vector execution unit 202) of the video processing function is performed by the video processor ( Video processing programs built for < RTI ID = 0.0 > 111 < / RTI > Scalar software code and vector software code can be compiled separately and then linked together to form a coherent application.

Partitioning allows vector software code functions to be written separately from scalar software code functions. For example, vector functions may be written individually (eg, at different times, by different teams of engineers, etc.) and may be used in conjunction with / by scalar functions (scalar threads, processes, etc.) It can be provided as a subroutine or library function. This enables individual independent update of the scalar software code and / or the vector software code. For example, vector subroutines can be updated independently of scalar subroutines (eg, by updating pre-released programs, new features added to improve the functionality of the distributed program, etc.), and vice versa. Hold. Partitioning is facilitated by separate caches (eg, caches 211, 212) of the scalar processor 210 and individual caches (eg, caches 222, 223) of the vector control unit 220 and the vector data path 221. Become. As described above, the scalar execution unit 201 and the vector execution unit 202 communicate via the command FIFO 225.

3 is a diagram illustrating an exemplary software program 300 for a video processor 111 in accordance with one embodiment of the present invention. As shown in FIG. 3, the software program 300 represents a property of the programming model for the video processor 111, whereby the scalar control thread 301 along with the vector data thread 302 may cause the video processor 111 to appear. Is executed by

An example of the software program 300 of the embodiment of FIG. 3 represents a programming model for the video processor 111, whereby a scalar control program (eg, scalar control thread 301) on the scalar execution unit 301 executes a vector. Executes a subroutine call (eg, vector data thread 302) on unit 202. An example of software program 300 is that a compiler or software programmer uses a scalar portion (e.g., a first thread) for a video processing application. ) And a vector portion (e.g., a second thread).

As shown in FIG. 3, the scalar control thread 301 executed on the scalar execution unit 201 quickly calculates job parameters and supplies these parameters to the vector execution unit 202 which performs most of the processing work. As mentioned above, the software code for the two threads 301, 302 can be written and compiled separately.

Scala threads are responsible for:

1. Interfacing with Host Unit 204 and Implementation of Class Interface

2. Initialization, setup and configuration of the vector execution unit 202

3. Run the algorithm in the working units, chunks or working sets in the loop, with each iteration

a. The parameters for the current working set are calculated,

b. Input data delivery to the vector execution unit is started,

c. Delivery of the output data from the vector execution unit is started.

A representative execution model of a scalar thread is "fire-and-forget." The term fire and fork means that commands and data are sent from the scalar execution unit 201 to the representative model for a video baseband processing application. Eg, via command FIFO 225) to the vector execution unit 202 and return data from the vector execution unit 202 until the algorithm is complete.

In the program 300 example of FIG. 3, the scalar execution unit 201 is in the vector execution unit 202 until there is no longer any space in the command FIFO 225 (eg, lend_of_alg &! Cmd_fifo_full). Maintain scheduling work for The job scheduled by the scalar execution unit 201 calculates the parameter, sends the parameter to the vector subroutine, and then calls the vector subroutine to perform the job. Execution of the subroutine (eg, vector_funcB) by the vector execution unit 202 is delayed in a timely manner mainly to hide delays from main memory (eg, system memory 115). Thus, the architecture of video processor 111 provides a delay compensation mechanism on the vector execution unit 202 side for both instruction and data traffic. This delay compensation mechanism will be described later.

It should be noted that the example of the software program 300 is more complicated in the case where there are two or more pipelines (eg, the vector data path 221 and the second vector data path 231 of FIG. 2). . Similarly, the example of software program 300 is more for situations where program 300 is written for a computer system with two vector execution pipelines but still retains the ability to run on a system with a single vector execution pipeline. It will be complicated.

Thus, as described above in the description of FIGS. 2 and 3, the scalar execution unit 201 is responsible for the start of computation on the vector execution unit 202. In one embodiment, the command passed from the scalar execution unit 201 to the vector execution unit 202 has the following main types.

1. Read command (eg, memRd) initiated by the scalar execution unit 201 to transfer the current job setting data from the memory to the data RAM of the vector execution unit 202.

2. Parameters passed from scalar execution unit 201 to vector execution unit 202

3. An execution command in the form of a PC (eg program counter) of the vector subroutine to be executed.

4. Write command initiated by scalar execution unit 201 to copy the result of the vector operation into memory (eg, memWr)

In one embodiment, the vector execution unit 202 schedules the memRd command for the memory interface 203 upon receipt of these commands (eg, to read the requested data from the frame buffer 205). . The vector execution unit 202 also examines the execute command and prefetches the vector subroutine to be executed (if not present in the cache 222).

The purpose of the vector execution unit 202 in this situation is for the vector execution unit 202 to preschedule the instruction and data stream of the next few execution commands while working on the current execution command. The prescheduling feature effectively hides the delays involved in fetching instructions / data from memory locations. To quickly make this read request, the vector execution unit 202, data storage memory (e.g., 223), and instruction cache (e.g., 222) are implemented using high speed optimal hardware.

As described above, the data storage memory (e.g., 223) functions as a working RAM of the vector execution unit 202. The scalar execution unit 201 recognizes and interacts with the data storage memory as a collection of FIFOs. The FIFO includes the "stream" in which the video processor 111 operates. In one embodiment, the stream is generally an input / output FIFO where the scalar execution unit 201 initiates a transmission (eg, to the vector execution unit 202). As described above, the operations of the scalar execution unit 201 and the vector execution unit 202 are separated.

When the input / output stream is full, the DMA engine in the vector control unit 220 stops processing of the command FIFO 225. This soon fills up the command FIFO 225. The scalar execution unit 201 stops issuing further work for the vector execution unit 202 when the command FIFO 225 is full.

In one embodiment, the vector execution unit 202 may require an intermediate stream in addition to the input and output streams. Thus, the entire data storage memory 223 can be viewed as a collection of streams in relation to interaction with the scalar execution unit 201.

4 illustrates an example of subpicture mixing with video using a video processor according to an embodiment of the present invention. 4 illustrates an example case where a video surface is mixed with a subpicture and then converted to an ARGB surface. The data comprising the surface is located in the frame buffer memory 205 as the Luma parameter 412 and the Chroma parameter 413. As shown, the subpicture pixel element 414 is also located within the frame buffer memory 205. Vector subroutine instructions and parameters 411 are instantiated in memory 205 as shown.

In one embodiment, each stream includes a FIFO of working 2D chunks of data called "tiles." In this embodiment, the vector execution unit 202 maintains a read tile pointer and a write tile pointer for each stream. For example, for an input stream, when a vector subroutine is executed, the vector subroutine may consume or read from the current (read) tile. In the background, data is sent to the current (write) tile by the memRd command. The vector execution unit may also generate an output tile for the output stream. These tiles are then moved to memory by the memWr0 command following the execute command. This effectively prefetches the tiles and prepares the tiles to be manipulated to hide the delay effectively.

In the subpicture blending example of FIG. 4, the vector data path 221 is constructed by an instantiated instance of the vector subroutine instruction and parameter 411 (eg, & v_subp_blend). This is shown by line 421. The scalar execution unit 201 reads chunks (eg, tiles) of the surfaces and uses them to store the data storage memory 223 using the DMA engine 401 (eg, in the memory interface 203). Load in The loading operation is shown by lines 422, 423 and 424.

With continued reference to FIG. 4, there are multiple input surfaces, so multiple input streams need to be maintained. Each stream has a corresponding FIFO. Each stream may have a different number of tiles. The example of FIG. 4 shows that the subpicture surface is in system memory 115 (eg, subpicture pixel element 414) and thus additional buffering (eg, n, n + 1, n + 2, n + 3, etc.). ), While the video stream (eg, luminance 412, chromaticity 413, etc.) may have fewer tiles. The number of buffers / FIFOs used can be adjusted according to the degree of delay experienced by the stream.

As described above, the data storage memory 223 uses a predictive prefetch method to hide the delay. Because of this, when data is prefetched for appropriate vector data path execution hardware (e.g., shown as FIFO n, n + 1, n + 2, etc.), the stream may have data in more than one tile.

Once the data storage memory is loaded, the FIFO is accessed by the vector data path hardware 221 and manipulated by the vector subroutine (eg, 430). The result of the vector data path operation includes an output stream 403. This output stream is copied by the scalar execution unit 201 to the frame buffer memory 205 (eg, ARGB_OUT 415) via the DMA engine 401. This is shown by line 425.

As such, embodiments of the present invention utilize an important aspect of stream processing in which data storage memory is abstracted as a plurality of memory tiles. Thus, a stream can be viewed as a set of tiles that are sequentially accessed. Streams are used to prefetch data. This data is in the form of tiles. The tiles are prefetched to hide delays from the particular memory source from which the data originates (eg, system memory, frame buffer memory, etc.). Similarly, the streams may point to different locations (eg, cache for vector execution units, cache for scalar execution units, frame buffer memory, system memory, etc.). Another feature of the streams is that they generally access the tiles in predictive prefetch mode. As mentioned above, the greater the delay, the more difficult the prefetch is, and the more buffering is used for the stream (eg, as shown in FIG. 4).

5 is a diagram illustrating internal components of a vector execution unit according to an embodiment of the present invention. 5 shows an arrangement of various functional units and register / SRAM resources of the vector execution unit 202 from the point of view of the program.

In the embodiment of FIG. 5, the vector execution unit 202 includes a VLIW digital signal processor optimized for the performance of video baseband processing and execution of various codecs (compression-decompression algorithms). Thus, the vector execution unit 202 has a number of attributes aimed at improving the efficiency of video processing / codec execution.

In the embodiment of FIG. 5, the attributes include the following.

1. Scalable performance by providing options for inclusion of multiple vector execution pipelines

2. Assignment of Two Data Address Generators (DAGs) per Pipe

3. Memory / Register Operands

4. 2D (x, y) pointers / repeaters

5. Deep pipeline (eg 11-12) stages

6. Scalar (integer) / branch unit

7. Variable command width (long / short command)

8. Data sorter for extracting operands

9. Representative Operands and Resulting 2D Data Path (4x4) Geometry

10. Slave vector execution unit for scalar execution unit executing remote procedure call

In general, the programmer's view of the vector execution unit 202 is as a SIMD data path with two DAGs 503. The instructions are issued in a VLIW manner (eg, instructions are issued simultaneously to the vector data path 504 and the address generator 503) and decoded by the instruction decoder 501 and sent to the appropriate execution unit. The instructions have a variable length, and the most commonly used instructions are encoded in short form. The full instruction set is available in the long form as a VLIW type instruction.

Legend 502 represents three clock cycles with three VLIW instructions. According to legend 510, the top of VLIW instructions 502 includes two address instructions (eg, for two DAGs 503) and one instruction for the vector data path 504. The intermediate VLIW instruction includes one integer instruction (eg, for integer unit 505), one address instruction and one vector instruction. The lowest VLIW instruction includes a branch instruction (eg, for branch unit 506), one address instruction and one vector instruction.

The vector execution unit can be configured to have a single data pipe or multiple data pipes. Each data pipe consists of local RAM (eg, data storage memory 511), crossbar 516, two DAGs 503, and a SIMD execution unit (eg, vector data path 504). 5 shows a basic configuration for description where only one data pipe is instantiated. When two data pipes are instantiated, they can run as individual threads or as cooperative threads.

Six different ports (eg, four reads and two writes) may be accessed by address register file unit 515. These registers receive parameters from scalar execution units or from the results of integer unit 505 or address unit 503. The DAG 503 also functions as a collection controller and manages the distribution of registers to address the contents of the data storage memory 511 (e.g. RA0, RA1, RA2, RA3, WA0 and WA1). Crossbar 516 is coupled to assign output data ports R0, R1, R2, R3 to the vector data path 504 in any order / combination to implement a given instruction. The output of the vector data path 504 can be fed back to the data storage memory 511 as indicated (eg, W0). Constant RAM 517 is used to provide operands that are frequently used in integer unit 505 to vector data path 504 and data storage memory 511.

6 shows a layout of a data storage memory 610 having a plurality of banks 601-604 and a symmetric tile array of a memory 600 according to one embodiment of the invention. As shown in FIG. 6, for the sake of explanation, only a portion of the data storage memory 610 is shown. Data storage memory 610 logically includes an array (or arrays) of tiles. Each tile is an array of 4x4 sub tiles. Physically, as shown by memory 600, data storage memory 610 is stored in an array of “N” physical memory banks (eg, banks 601-604).

Data storage memory 610 also visually illustrates logical tiles in the stream. In the embodiment of Figure 6, this tile has a height of 16 bytes and a width of 16 bytes. This tile is an array of subtiles (4x4 in this example). Each subtile is stored in a physical bank. If there are eight physical memory banks (e.g., banks 0-7), this is indicated by the number in each 4x4 subtile in FIG. The configuration of the subtiles in the banks is such that there is no common bank in the 2x2 array of subtiles. This allows any unaligned access (eg in both x and y directions) without any bank collision.

Banks 601-604 are configured to support access to different tiles in each bank. For example, in one case, crossbar 516 may access a 2x4 tile set (eg, the first two rows of bank 601) from bank 601. In another example, crossbar 516 may access a 1x8 tile set from two adjacent banks. Likewise, in other cases, crossbar 516 may access an 8x1 tile set from two adjacent banks. In each case, the DAG / collector 503 may receive the tiles as the banks are accessed by the crossbar 516 and provide these tiles to the front end of the vector data path 504 in clock units.

In this manner, embodiments of the present invention provide a new video processor architecture that supports sophisticated video processing functions while efficiently utilizing integrated circuit silicon die area, transistor count, memory speed requirements, and the like. Embodiments of the present invention maintain a high computational density to process multiple video streams and are easily scalable. Embodiments of the present invention provide for example MPEG-2 / WMV9 / H.264 encode support (e.g., in-loop decoder), MPEG-2 / WMV9 / H.264 decode (e.g. post entropy decoding). And a number of sophisticated video processing operations such as in-loop and out-of-loop non-blocking filters.

Additional video processing operations provided by embodiments of the present invention include, for example, improved motion adaptive deinterlacing, input noise filtering for encoding, polyphase scaling / resampling, and subpicture synthesis. The video processor architecture of the present invention is also used in certain video processor-amplifier (procamp) applications, such as pixel point manipulation such as color space conversion, color space adjustment, sharpening, histogram adjustment, and various video surface format conversions, for example. Can be.

Broadly and without limitation, the present invention discloses the following. A delay tolerant system for performing video processing operations has been described. The system includes a host interface for implementing communication between the video processor and the host CPU, a scalar execution unit connected to the host interface and configured to execute a scalar video processing operation, and a vector execution connected to the host interface and configured to execute vector video processing operation. It includes a unit. A command FIFO is included for causing the vector execution unit to operate on a request drive basis by accessing the memory command FIFO. A memory interface is included for implementing communication between the video processor and the frame buffer memory. A DMA engine is built in the memory interface for implementing DMA transfers between a plurality of different memory locations and for loading data and instructions for the vector execution units into the command FIFO. A video processor for performing video processing operations has been described. The video processor includes a host interface for implementing communication between the video processor and the host CPU. A memory interface is included for implementing communication between the video processor and the frame buffer memory. A scalar execution unit is coupled to the host interface and the memory interface and configured to execute a scalar video processing operation. The vector execution unit is coupled to the host interface and the memory interface and is configured to execute the vector video processing operation. A multidimensional data path processing system for a video processor for performing video processing operations has been described. The video processor includes a scalar execution unit configured to execute a scalar video processing operation and a vector execution unit configured to execute a vector video processing operation. A data storage memory for storing data for the vector execution unit is included. The data storage memory includes a plurality of tiles with symmetrical bank data structures arranged in an array. Bank data structures are configured to support access to different tiles in each bank. A stream based memory access system for a video processor for performing video processing operations has been described. The video processor includes a scalar execution unit configured to execute a scalar video processing operation and a vector execution unit configured to execute a vector video processing operation. Frame buffer memory for storing data for the scalar execution unit and the vector execution unit is included. A memory interface for establishing communication between the scalar execution unit and the vector execution unit and the frame buffer memory is included. The frame buffer memory includes a plurality of tiles. The memory interface implements a first stream comprising a first sequential tile access and a second stream comprising a second sequential tile access for a vector execution unit or a scalar execution unit.

The foregoing descriptions of specific embodiments of the present invention have been presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the invention to the precise forms disclosed, and many modifications and variations are possible in light of the above teaching. The embodiments have been selected and described in order to best explain the principles of the invention and its practical use, so that those skilled in the art can best utilize the various embodiments in which the invention and various modifications have been made when suited for the particular use contemplated. It is intended that the scope of the invention be defined by the claims appended hereto and their equivalents.

Appendix A

1 Overview

VP2 is a VLIW SIMD video DSP connected to a scalar control processor. His main focus is on video codecs and video baseband processing.

1.1 VP2 Spirit of .0

Efficiency: VP2.0 is a computationally efficient machine for video applications in terms of perf / mm2 and perf / mW.

Programmability: Very programmable, easily compatible, and safer to program the machine.

Scalability: The VP2.0 design / architecture must be scalable to match the performance requirements for multiple application areas.

1.2 Design Goals

Computational density

Provides significant perf / mm2 advantage over VP1.0.

Efficient implementation of new application areas such as H.264

-Tolerate delays in HW to ease the burden on software developers

Hide data fetch delays by rearranging memory accesses and operations

Automatic prefetch of command streams

-Hide data path delay

-Selective delivery of intermediate results

Streaming computation model

Scalability

Structurally, the VP2 vector unit can scale its data path up by 2x and down by 1 / 2x.

Frequency improvement can be achieved by selective refining.

1.3 Application target

The VP2.0 design and instruction set are optimized to perform the following applications very efficiently.

mpeg2 / wmv9 / H.264 encode support (in-loop decoder)

mpeg2 / wmv9 / H.264 decode (post-entropy decoding)

In-loop / out-loop deblocking filter

Enhanced motion adaptive deinterlacing

Input Noise Filtering for Encoding

Multiphase Scaling / Resampling

Subpicture Synthesis

Pixel point operations such as procamping, color space conversion, adjustment, sharpening, histogram adjustment, etc.

-Support various video surface format conversion

Architecturally, VP2.0 can be efficient in the following areas.

-2D primitive functions, bullets, rotates, etc.

Precision-based Software Motion Estimation Algorithm

16 / 32-bit MAC applications

2 top-level architecture

The VP2.0 machine is divided into scalar and vector processors. The vector processor operates as a slave coprocessor to the scalar processor. The scalar processor is responsible for supplying control streams (parameters, subroutine independent variables) to the vector processor and managing data I / O to the vector processor. All control flow of the algorithm is executed on the scalar machine, while the actual pixel / data processing operations are done on the vector processor.

The scalar processor is a representative RISC style scalar, and the vector coprocessor is a SIMD machine with one or two SIMD pipes (each SIMD pipe has a 16 pixel data path). Thus, the vector coprocessor can produce up to 32 pixels of the result with pure computing power.

The scalar processor sends function calls to the vector coprocessor using a memory mapped command FIFO. Coprocessor commands are queued to this FIFO. The scalar processor uses this FIFO to completely separate it from the vector processor. The scalar processor can run on its own clock. The vector processor operates as a request drive unit.

The top level diagram of VP2.0 is given below.

VP2.O can be compiled into separate scalar and vector code and then linked together later. Separately, vector functions can be written separately and provided to a scalar thread as a subroutine or library function. Scalar processors have their own instruction and data caches. The vector unit also has an instruction cache and a data RAM (referred to as data storage memory). These two engines are separated and communicate via FIFO.

3 Simple Programming Model

The simplest programming model for VP2.0 is a scalar control program that executes subroutine calls on the vector slave coprocessor. There is a unique assumption here that the programmer breaks down the problem into two threads. A thread running on a scalar processor quickly calculates job parameters and supplies them to the main machine, the vector processor. Programs for these two threads are expected to be written and compiled separately.

Scala threads are responsible for:

1. Implement interface and class interface with host unit

2. Initialization, setup and configuration of the vector unit

3. Run the algorithm in the working units, chunks or working sets in the loop so that each iteration

a. Calculate the parameters for the current working set.

b. Initiates delivery of input data to the vector processor.

c. Start delivery of output data from the vector processor.

The typical execution model of Scala threads is Fire and Forget. This is expected to be a representative model for video baseband processing where there is no return data from the vector coprocessor. The scalar processor will keep scheduling for the vector processor until there is space in the command FIFO. Execution of subroutines by the vector processor is delayed in time, mainly due to delays from main memory. Therefore, it is important to provide a delay compensation mechanism on the vector side. In VP2.0, the vector processor provides delay compensation for both instruction and data traffic. The mechanism for him is summarized in the section.

A typical VP program might look like this:

A more complex programming model is when you have two data pipes. Or, you may want to write code for two pipes and run them on one data pipe machine. The programming model for that is explored in section 6.

4 streaming models

As summarized above, the scalar engine is responsible for initiating operations on the vector processor. Commands sent from the scalar engine to the vector engine have the following main types:

1. Read command (memRd) initiated by scalar to transfer current working set data from memory to vector RAM in vector engine

2. Parameters passed from scalar to vector

3. Execution command in PC form of vector subroutine to be executed

4. Write command (memWr) initiated by scalar to copy the result of the vector operation into memory

The vector processor schedules the memRd command for the frame buffer (FB) interface upon receipt of these commands. It also checks the execute command and prefetches the vector subroutine to be executed (if not present in the cache). One purpose is to preschedule the instruction and data streams of the next few execution commands while the vector engine is currently working on the execution commands. To quickly make this read request, the vector engine manages data storage memory and instruction cache in hardware.

The data storage memory is the working RAM of the vector processor. Scalar processors see this data storage memory as a set of FIFOs or streams. The streams are essentially input / output FIFOs where Scala initiates delivery. When the input / output stream is full, the vector DMA engine stops processing the command FIFO from the scalar and is soon full. Thus, Scala stops publishing more jobs for the vector engine. In addition to the input and output streams, vectors may require intermediate streams. Thus, the entire data storage memory can be seen as a collection of streams from the scalar side. Each stream is a FIFO of working 2D chunks called tiles. The vector processor maintains a read tile pointer and a write tile pointer for each stream. For an input stream, when a vector subroutine is executed, it can consume or read from the current (read) tile. In that background, data is sent to the current (write) tile by the memRd command. The vector processor may also generate an output tile for the output stream. These tiles are then moved to memory by the memWr () command following the execute command.

This model is described as an example of subpicture mixing with video. For example, consider a simple example where a video surface (eg, NV12 format) is mixed with a subpicture and then converted to an ARGB surface. These surfaces are located in the memory. The scalar processor reads chunks (tiles) of these surfaces and loads them into data storage memory. Since there are multiple input surfaces, it is necessary to maintain multiple input streams. Each stream can have a different number of tiles (eg, in this example, it can be assumed that the subpicture surface is in system memory, and therefore must buffer it further), but the video stream has fewer May have tiles.

5 Vector Coprocessor

The VP2's vector coprocessor is a VLIW DSP designed for video baseband processing and codecs. Some important design attributes of this processor include:

1. Scalable performance, 1 or 2 data pipes.

2. Each pipe has two data address generators (DAGs).

3. Memory / Register Operands

4. 2D (x, y) pointers / repeaters

5. Deep pipeline (11-12) stage

6. Scalar (integer) / branch unit

7. Variable command width (long / short command)

8. Data sorter for extracting operands

9. Representative Operands and Resulting 2D Data Path (4x4) Geometry

10. Execute the slave processor, remote procedure call to the scalar processor.

The programmer's view of the vector coprocessor is, in simplest terms, a SIMD data path with two DAGs. The instructions are issued in a VLIW manner (ie, the instructions are issued simultaneously for the vector data path and address generator). The instructions have a variable length, and the most commonly used instructions are encoded in short form. The full instruction set is available in long form. For example, from the programmer's perspective, the arrangement of the various functional units and register / SRAM resources is as shown below.

The vector unit instantiates a single data pipe or dual data pipes. Each data pipe consists of local RAM (data storage memory), two DAGs, and a SIMD execution unit. There is only one data pipe in the basic configuration. When there are two data pipes, they can run as independent threads or as cooperative threads. The complete pipeline diagram of the vector processor is described below. Here, the overall configuration has two data pipes.

6 Improved Programming Model

In section 3, the RPC model was introduced to describe the basic architecture. In this section, more advanced concepts are introduced.

6.1 Dual  Data pipe configuration

In a dual pipe configuration, the following processor resources are shared.

-Scalar controller

-Vector control unit in the vector coprocessor

DMA engine for command / data fetch

Instruction cache (can be dual ported)

The following resources are duplicated.

Data pipes (addresses / branches / vector execution units)

Data storage memory

-Register file

Note the following:

1. A program can be written for two pipes on an instance with only one pipe. The vector control unit will map the execute command for each pipe on the same physical pipe. However, since the streams for both pipes exist in only one data storage memory, the size of the data storage must be adjusted. A simple way is to reduce the tile size or halve the number of tiles in the stream. This is done by a scalar thread at construction time. Problems exist such as replication of global registers and stream mapping that must be addressed at the micro-architecture stage.

2. A program written on 1 pipe can run on an instance with 2 pipes. However, this code runs on only one pipe and never uses the other. The machine is half idle.

3. A program can be written for two pipes, each running two completely different threads. This may be undesirable as it only has a single scalar that is not multithreaded. Since only one scalar execution thread is supported, this may not be desirable, but this model may be supported.

4. A program can be written for two pipes each running the same thread. This is a representative model expected for parallelable algorithms such as most video baseband processing. This allows using the same instruction stream to operate on two streams of video, two halves, and so forth. Each data pipe has its own execution unit and data store. Scalar controllers must provide two data pipes. However, the parameters, read and write commands are related to each other (offset), so the scalar performance requirements do not exactly double. An example of this model is shown below.

5. A program with two cooperative threads can be written. This is the expected model for codecs that have a single scalar control thread but may require that multiple functional vector function blocks be connected together. This is similar to the direct-show pin model of functional blocks. An example of such an application is shown below. This model is limited to only two cooperative threads, because it has only two data pipes. Another limitation is that threads must balance between two threads. Otherwise, there is a performance loss. Within this limitation, this model operates on two data pipes and can also be scaled to a single pipe.

6. Two data pipes can be synchronized with each other. The basic approach to synchronization is data driven. Vector functions are executed when data is available to the process. The streams are filled by reads from memory or writes from other data pipes. If data is available, the vector control unit activates and executes the execute command. Streams can also be used as counting semaphores. Both the scalar controller and the vector data pipe can increment or decrement the tile pointers and use the stream descriptor as counting semaphore even when no data transfer exists.

Supplementary summary:

In general, embodiments of the present invention perform the following.

1. Decomposition into scalar and vector parts of media algorithms

Off the self scalar design, which also provides the ability to run scalar and vector portions at different clock rates based on power and performance requirements.

2. Stream Processing

3. 2D data path processing

4. Lazy hide (for both data and command fetch)

Application area

hiding:

Opcode hidden

The encryption program can only hide the chip. The scalar / controller block only requires certain operations to be performed, and the crypto engine will fetch instructions and the like. Scala is very safe because it doesn't even know if the algorithm is running. This provides a mechanism for hiding the encryption algorithm from the user.

2D

The VP2 instruction set architecture supports instructions for 2D processing. These include the ROP3 and ROP4 support used in many GUI / windows systems. This allows the media processor to execute 2D operations on the media processor. The unique advantage here is power savings.

ISA

Condition code as an instruction slot:

Have individual issue slots (in a multiple issue instruction bundle) for condition code operations. The prior art is a human use SIMD instruction that may affect the condition code / predicate register. However, in the approach taken in VP2, data processing and predicate register processing can be scheduled independently, resulting in higher performance.

Memory I / O

Microcoded DMA Engine:

The DMA engine can be programmed (or have its own small microcode) to perform various operations, such as data prefetch, stream stream formatting, edge padding, and so on. Generally, it is a programmable DMA engine and has no wired functionality. Thus, the combination of memory I / O processors and media processing cores improves overall system level performance. The media processor core is not burdened with data I / O processing.

Memory hierarchy:

In the VP2 architecture, the memory hierarchy is optimized to provide delay compensation as well as to minimize memory BW. Many different schemes are provided, including:

The first level of streaming data storage that can be seen by the vector core as scratch RAM. Managed by the HW to examine the request stream generated by the scalar processor in advance. This data store is optionally augmented by the L2 cache for data reuse. The L2 cache may be divided into individual sectors on a stream basis.

L1 cache augmented by streaming data store. The data store prefetched the following related data sets.

-Cache using the stream pointer as the data tag

Prefetch / caching of L1 data store and L2 cache using scalar generated stream addresses

Optimized scalar-vector communication link:

MemRd / Wr Format:

Compact command from scalar to read and write system memory to local memory. Store on the control flow bandwidth needed to manage the DMA engine. At the same time, it does not limit the types of supported transactions.

Considerations on scalar-vectors for vector L2:

Parameter compression to support parameter modifiers and repeaters to reduce communication bandwidth.

Pipeline Cache:

Pipelined instruction cache. Various schemes are supported as follows.

Manage the life of each cache line by tracking in-flight execution between vectors and scalar processors. This allows the instructions to be prepared before the vector processor starts executing. If the instructions are not yet in the cache, the instructions are prefetched.

For low latency configurations, the instruction cache is minimized by changing to a small FIFO. Existing runs in the FIFO can be reused, otherwise fetched again.

Full architecture:

The data store can be shared between the various processing elements. They can communicate via a stream and feed each other. This architecture envisions a set of heterogeneous functional units such as SIMD vector cores, DMA engines, fixed function units connected via a stream.

Arithmetic / DP

Arbitrary / flexible shape / half pipe:

The data path operates on variable shapes. The shape of the data path can be configured to match the problem set. In general, people follow a 1D data path. VP2 may process shapes that may have varying sizes, such as 4x4, 8x4 or 16x1, to match the algorithm.

Scalability:

The VP2 data path architecture uses instruction guidance technology to execute wider SIMD instructions on narrower data paths over multiple cycles to reduce area. You can have an 8-way SIMD pipe (grouping 2 pipes together) and a wider SIMD data path where each operand is 2 bytes, likewise a 4-way SIMD pipe (grouping 4 pipes together) and each Operand may have a wider SIMD data path of 4 bytes).

For example, VP2 may scale the data path from 16 way SIMD to 8 way SIMD.

Connection of byte lanes:

Connection of SIMD ways to increase operand width. For example, current 16-way SIMDs have 8-bit operands. This can be increased with 16-bit operands of 8-way SIMDs and 32-bit operands of 4-way SIMDs.

SIMD address generator

Individual stream address generator for each way of the SIMD pipe.

VP2 can use the SIMD address generator where requests are coalesced with minimal access to the data store.

Extend data with crossbars and collectors

Ability to create more data operands using crossbars. Reduce read port pressure and reduce power.

X2 commands:

Not all instructions can use every HW element (adder / multiplier) in the data path. Thus, it is possible for simple instructions such as addition / subtraction which can handle wider data shapes than complex instructions. Thus, instead of limiting performance to the minimum common size, VP2 uses a flexible instruction set that properly attempts to operate on a wider shape as long as the read ports can maintain operating bandwidth.

Multithreaded / multicore media processing

The VP2 architecture supports a variety of multithreading options, including:

A multithreaded scalar processor schedules procedure calls on multiple vector units connected over a stream.

Multiple threads are executed on a single vector engine by thread switching in units of instructions or units of execution.

Power Management with Different Vectors / Scalars

If the scalar and vector portions are separated, these two blocks can run at different speeds based on power and performance requirements.

Context switch:

The media processor has a registerless structure and therefore has the ability to support very fast context switches. HW support exists to track the scalar-vector command queue, store it, and play it back to achieve context switching. In addition, the context switch can be initiated upon page fault.

This allows the media processor to support non real-time tasks such as 2D acceleration or optimal video enhancement to supply the display pipeline while maintaining real-time processing tasks such as input and output display processing.

This context switch capability, along with its instruction set, enables VP2 to integrate pixel / codec processing.

Data configuration:

VP2 uses a data storage configuration with the following characteristics:

Up to 16 pixels in each direction can be accessed without bank collision. This is done while keeping the stride requirements to a minimum.

The data store configuration enables efficient transition of data shapes.

2D addressing is supported within the data store, eliminating SW operations of linear addresses in most media processing applications such as video.

101: CPU
105: the bridge
112: display
115: memory
210: scalar processor
211: processor instruction cache
212: Processor Data Cache
511: data storage memory
515: address register file

Claims (22)

As a latency tolerant system for performing video processing operations,
A host interface for implementing communication between the video processor and the host CPU;
A scalar execution unit coupled to the host interface and configured to execute scalar video processing operations;
A vector execution unit coupled to the host interface and configured to execute vector video processing operations;
Command FIFO for enabling the vector execution unit to operate on a demand driven basis by accessing a command FIFO, wherein the vector execution unit is configured to execute a performance level of the vector execution unit based on the request driving basis. operable to adjust the level);
A memory interface for implementing communication between the video processor and frame buffer memory; And
A DMA engine built within the memory interface, implementing DMA transfers between a plurality of different memory locations, and loading data and instructions for the vector execution unit into a data storage memory and an instruction cache
Delay tolerant system comprising a. The method of claim 1,
And the vector execution unit is configured to operate asynchronously with respect to the scalar execution unit by accessing the command FIFO to operate on the request drive basis. The method of claim 1,
And the request drive base is configured to hide a delay of data transfer from the vector execution unit to the command FIFO at the different memory locations. The method of claim 1,
And the scalar execution unit is configured to implement algorithm flow control processing. The method of claim 4, wherein
And the scalar execution unit is configured to precalculate the working parameters for the vector execution unit to hide the data propagation delay. The method of claim 1,
And the vector execution unit is configured to schedule a memory read through the DMA engine to prefetch commands for subsequent execution of a vector subroutine. The method of claim 6,
And the memory read is scheduled to prefetch commands for execution of the vector subroutine prior to calls of the vector subroutine by the scalar execution unit. A method for delay tolerant video processing operations,
Implementing communication between the video processor and the host CPU using the host interface;
Executing scalar video processing operations using a scalar execution unit coupled to the host interface;
Executing vector video processing operations using a vector execution unit coupled to the host interface;
Enabling the vector execution unit to operate on a demand drive basis by accessing a memory command FIFO, the vector execution unit being operable to adjust the performance level of the vector execution unit based on the request drive basis;
Implementing communication between the video processor and frame buffer memory using a memory interface; And
Implementing DMA transfers between a plurality of different memory locations using a DMA engine built in the memory interface and configured to load data and instructions for the vector execution unit into a data storage memory and an instruction cache.
The method for delay tolerant video processing operations comprising a. The method of claim 8,
And the vector execution unit is configured to operate asynchronously with respect to the scalar execution unit by accessing the memory command FIFO to operate on the request drive basis. The method of claim 8,
And the request drive base is configured to hide a delay of data transfer from the vector execution unit to the memory command FIFO at the different memory locations. The method of claim 8,
And the scalar execution unit is configured to implement algorithmic flow control processing. The method of claim 11,
And the scalar execution unit is configured to precalculate the working parameters for the vector execution unit to hide the data propagation delay. The method of claim 8,
And the vector execution unit is configured to schedule a memory read through the DMA engine to prefetch commands for subsequent execution of a vector subroutine. The method of claim 13,
And the memory read is scheduled to prefetch commands for execution of the vector subroutine prior to calls of the vector subroutine by the scalar execution unit. A delay tolerant system for performing video processing operations,
Motherboard;
A host CPU connected to the motherboard;
A video processor coupled to the motherboard and the host CPU;
A host interface for implementing communication between a video processor and the host CPU;
A scalar execution unit coupled to the host interface and configured to execute scalar video processing operations;
A vector execution unit coupled to the host interface and configured to execute vector video processing operations;
Command FIFO to enable the vector execution unit to operate on a demand drive basis by accessing a command FIFO, wherein the vector execution unit may be operable to adjust the performance level of the vector execution unit based on the request drive basis. -;
A memory interface for implementing communication between the video processor and frame buffer memory; And
A DMA engine built within the memory interface, implementing DMA transfers between a plurality of different memory locations, and loading data and instructions for the vector execution unit into a data storage memory and an instruction cache
Delay tolerant system comprising a. 16. The method of claim 15,
And the vector execution unit is configured to operate asynchronously with respect to the scalar execution unit by accessing the command FIFO to operate on the request drive basis. 16. The method of claim 15,
And the request drive base is configured to hide a delay of data transfer from the vector execution unit to the command FIFO at the different memory locations. 16. The method of claim 15,
And the scalar execution unit is configured to implement algorithmic flow control processing. The method of claim 18,
And the scalar execution unit is configured to precalculate the working parameters for the vector execution unit to hide the data propagation delay. 16. The method of claim 15,
The vector execution unit is configured to schedule a memory read through the DMA engine to prefetch commands for subsequent execution of a vector subroutine, wherein the memory read is prior to calls of the vector subroutine by the scalar execution unit. Scheduled to prefetch commands for execution of the vector subroutine. 16. The method of claim 15,
The vector execution unit may operate at a first power level and a second power level based on the command FIFO, wherein the second power level uses less power than the first power level. 16. The method of claim 15,
And the vector delay unit can operate based on the request drive based on the contents of the command FIFO.

Images