JP4906734B2 - Video processing - Google Patents

Abstract

A latency tolerant system for executing video processing operations is described. So too is a stream processing in a video processor, a video processor having scalar and vector components and multidimensional datapath processing in a video processor.

Description

Cross-reference of related applications

  This application is entitled “A METHOD AND SYSTEM FOR VIDEO PROCESSING” and is filed on Nov. 15, 2004 in US Provisional Patent Application No. 60 / 628,414 of Gadre et al. The provisional application of which is hereby incorporated by reference in its entirety.

Field of Invention

  [001] The field of this description relates to digital electronic computer systems. More particularly, this description relates to a system that efficiently processes video information in a computer system. In one aspect, a latency-tolerant system that performs video processing operations is described. Regarding another aspect, stream processing in a video processor will be described. Further, multidimensional data path processing in the video processor will be described. A video processor having a scalar component and a vector component is also described.

background

  [002] Display of images and full motion video is an area of the electronics industry that is being improved with significant progress in recent years. The display and rendering of high quality video, especially high definition digital video, is a major goal of modern video technology applications and devices. Video technology is used in a wide variety of products ranging from mobile phones, personal video recorders, digital video projectors, high-definition televisions and the like. The advent and growth of devices that support the generation and display of high-definition video is one area of the electronics industry that has experienced significant technological innovation and progress.

  [003] Video technologies deployed in many consumer electronics types and business-level devices utilize one or more video processors to format and / or enhance the video signal for display. This is especially true for digital video applications. For example, one or more video processors are incorporated into a standard set-top box and are used to convert HDTV broadcast signals into video signals that can be used by a display. Such conversion includes, for example, scaling, where a video signal is converted from a video image other than 16: 9 for proper display on a 16: 9 display (eg, widescreen). Is done. It is possible to perform scan conversion using one or more video processors, where the entire frame is drawn in a single sweep, from an interlaced format where the odd and even scan lines are displayed separately. The video signal is converted into a progressive format.

  [004] A further example of a video processor application is signal decompression, in which, for example, a video signal is received in a compressed format (eg MPEG-2) and decompressed for display. To be formatted. Another example is a re-interlaced scan conversion that converts an incoming digital video signal from the DVI (Digital Visual Interface) format to the vast number of older television displays introduced on the market. With conversion to some composite video format.

  [005] More advanced users include, for example, in-loop / out-of-loop deblocking filters, advanced motion adaptive progressive scan conversion, input noise filtering for encoding operations, polyphase scaling / resampling, Higher performance such as sub-picture synthesis and processor amplifier operations such as color space conversion, adjustment, pixel point operations (eg sharpening, histogram adjustment, etc.) and various video surface format conversion support operations Requires video processor function.

  [006] The problem with providing such high performance video processor functionality is that a video processor with a sufficiently powerful architecture to implement such functionality is too expensive to be incorporated into many types of devices. That is. As video processing functions become more sophisticated, the integrated circuit devices required to implement such functions are also more expensive in terms of silicon die area, number of transistors, memory speed requirements, and the like.

  [007] Accordingly, prior art system designers have been forced to make trade-offs regarding video processor performance and cost. Conventional video processors, widely regarded as having an acceptable cost / performance ratio, often have latency constraints (eg, to prevent video stuttering or stagnation of video processing applications), And in terms of computational density (eg, number of processor operations per square millimeter of die). In addition, prior art video processors typically meet linear scaling performance requirements, such as when a video device is expected to process multiple video streams (eg, simultaneous processing of multiple incoming and outgoing display streams). Not compatible.

  [008] Accordingly, there is a need for new video processor systems that overcome the limitations of the prior art. New video processor systems need to be scalable and have a high computational density to handle the high performance video processor functions expected by increasingly sophisticated users.

Overview

  [009] Embodiments described herein provide a novel video processor system that supports high performance video processing functions, integrated circuit silicon die area, transistor count, memory. Use speed requirements efficiently. Embodiments of this description retain high computational density and can be easily extended to handle multiple video streams.

  [010] In one embodiment, a latency-tolerant system is implemented that performs video processing operations on a video processor. The system is coupled to a host interface that implements communication between the video processor and the host CPU, a scalar execution unit coupled to the host interface and configured to perform scalar video processing operations, and to the host interface; A vector execution unit configured to perform vector video processing operations. A command FIFO is provided to allow the vector execution unit to operate on a request driven basis by accessing the memory command FIFO. A memory interface is provided for performing communication between the video processor and the frame buffer memory. The DMA engine is incorporated into the memory interface to perform DMA transfers between different storage locations and load vector execution unit data and instructions into the data store memory and instruction cache.

  [011] In an embodiment, the vector execution unit is configured to operate asynchronously with respect to the scalar execution unit by accessing a command FIFO and to operate on a request driven basis. The request driven base can be configured to conceal the latency of data transfer from different storage areas (eg, frame buffer memory, system memory, cache memory, etc.) to the command execution unit command FIFO. The command FIFO may be a pipelined FIFO to prevent stagnation of vector execution units.

  [012] In one embodiment, the present invention is implemented as a video processor for performing video processing operations. The video processor includes a host interface that implements communication between the video processor and the host CPU. The video processor includes a memory interface that implements communication between the video processor and the frame buffer memory. A scalar execution unit is coupled to the host interface and the memory interface and is configured to perform scalar video processing operations. A vector execution unit is coupled to the host interface and the memory interface and is configured to perform vector video processing operations. The video processor may be a stand-alone video processor integrated circuit or may be a component embedded in a GPU integrated circuit.

  [013] In an embodiment, the scalar execution unit functions as a controller of the video processor and controls the operation of the vector execution unit. The scalar execution unit can be configured to execute the flow control algorithm of the application, and the vector execution unit can be configured to execute the pixel processing operation of the application. A vector interface unit may be provided in the video processor to interface the scalar execution unit to the vector execution unit. In some embodiments, the scalar execution unit and the vector execution unit are configured to operate asynchronously. The scalar execution unit can execute at a first clock frequency and the vector execution unit can execute at a different clock frequency (eg, faster, slower, etc.). The vector execution unit can operate on a demand driven basis under the control of the scalar execution unit.

  [014] In one embodiment, the present invention is implemented as a multi-dimensional data path processing system for a video processor for performing video processing operations. The video processor comprises a scalar execution unit configured to perform scalar video processing operations and a vector execution unit configured to perform vector video processing operations. A data store memory is provided for storing vector execution unit data. The data store memory has a plurality of tiles having a symmetric bank data structure arranged in an array. The bank data structure is configured to support access to different tiles in each bank.

  [015] Depending on individual configuration requirements, each bank data structure may comprise multiple tiles (eg, 4x4, 8x8, 8x16, 16x24, etc.). In one embodiment, the banks are configured to support access to different tiles in each bank. As a result, a single row or column of tiles can be extracted from two adjacent banks with a single access. In one embodiment, a crossbar is used to select a configuration (eg, row, column, block, etc.) that accesses multiple bank data structure tiles. A collector can also be provided to receive tiles for banks accessed by the crossbar and to provide tiles at the front end of the vector data path every clock.

  [016] In one embodiment, the present invention is implemented as a stream-based memory access system for a video processor. The video processor comprises a scalar execution unit configured to perform scalar video processing operations and a vector execution unit configured to perform vector video processing operations. A frame buffer memory is provided for storing scalar execution unit and vector execution unit data. A memory interface is provided for performing communication between the scalar execution unit, the vector execution unit, and the frame buffer memory. The frame buffer memory includes a plurality of tiles. The memory interface implements a first stream of first sequential accesses of tiles of scalar execution units and a second stream of second sequential access of tiles of vector execution units.

  [017] In an embodiment, the first stream and the second stream are prefetched in a manner that conceals access latency from original storage (eg, frame buffer memory, system memory, etc.), With a series of sequential prefetch tiles. In one embodiment, the memory interface is configured to manage a plurality of different streams including streams from a plurality of different original storage areas and a stream to a plurality of different terminal storage areas. In one embodiment, a DMA engine embedded in the memory interface is used to perform multiple memory reads and multiple memory writes to support multiple streams.

[018] In general, this document discloses at least the following four methods.
A) The method broadly taught in this description is a method for a multi-dimensional data path processing system in a video processor that performs video processing operations, and performing a scalar video processing operation by using a scalar execution unit And executing the vector video processing operation by using the vector execution unit, and storing the data of the vector execution unit by using the data store memory, wherein the data store memory is arranged in an array. A plurality of tiles having a symmetric bank data structure configured to support access to different tiles in each bank. Further, in the method A described above, each bank data structure includes a plurality of tiles arranged in a 4 × 4 pattern. In the method A described above, each bank data structure includes a plurality of tiles configured in an 8 × 8, 8 × 16, or 16 × 24 pattern. In addition, the method of A above is configured such that the bank data structures support access to different tiles of each bank data structure, and at least one access is associated with the two adjacent bank data structures. Access including one row tile of one bank data structure. In the above method A, the tile is also configured to support access to different tiles of each bank data structure, and at least one access is the two adjacent banks to the two adjacent bank data structures. Access that includes one column of tiles in the data structure. Further, the method A described above includes selecting a configuration for accessing a plurality of bank data structure tiles by using a crossbar coupled to the data store. In this selection step, the crossbar accesses a plurality of bank data structure tiles and supplies data to the vector data path every clock. Also, receiving a plurality of bank data structure tiles accessed by the crossbar by using a collector and providing the tiles to the front end of the vector data path every clock.

B) The method broadly taught in this description is also a method for performing video processing operations, the method being implemented using a video processor of a computer system executing computer readable code, and The method establishes communication between the video processor and the host CPU by using the host interface, and establishes communication between the video processor and the frame buffer memory by using the memory interface. Performing a scalar video processing operation by using a scalar execution unit coupled to the host interface and the memory interface; and vector video processing operation to the host interface and And executing by the use of combined vector execution unit to the memory interface, a. In the above method B, the scalar execution unit further functions as a controller of the video processor and controls the operation of the vector execution unit. Method B described above also comprises a vector interface unit for interfacing the scalar execution unit with the vector execution unit. In method B above, the scalar execution unit and the vector execution unit are also configured to operate asynchronously. The scalar execution unit executes at the first clock frequency, and the vector execution unit executes at the second clock frequency. In the method B above, the scalar execution unit is configured to execute the flow control algorithm of the application, and the vector execution unit is configured to execute the pixel processing operation of the application. Further, the vector execution unit is configured to operate on a demand driven basis under the control of the scalar execution unit. In addition, the scalar execution unit is configured to send function calls to the vector execution unit using a memory command FIFO, and the vector execution unit operates on a request driven basis by accessing the memory command FIFO. Also, the asynchronous operation of the video processor is configured to support separate and independent updates of the application's vector subroutine or scalar subroutine. Finally, in method B above, the scalar execution unit is configured to operate using VLIW (very long instruction word) code.

C) The method described herein also broadly teaches a method for stream-based memory access in a video processor that performs video processing operations, which includes scalar video processing operations. Executing by using an execution unit; performing a vector video processing operation by using a vector execution unit; and using scalar execution unit and vector execution unit data by using a frame buffer memory. Storing, and performing communication between the scalar execution unit, the vector execution unit, and the frame buffer memory using a memory interface, wherein the frame buffer memory includes a plurality of frame buffer memories. And a memory interface for a vector execution unit or a scalar execution unit that implements a first stream that includes a first sequential access of tiles and a second stream that includes a second sequential access of tiles. carry out. The method C described above also has a first stream and a second stream that include at least one prefetched tile. The method C described above further comprises a first stream originating from a first location in the frame buffer memory and a second stream originating from a second location in the frame buffer memory. In method C above, the memory interface is also configured to manage multiple streams, including streams from multiple different origin locations and streams to multiple different end locations. In this regard, at least one origin location or at least one end location is in system memory. The method C described above also performs multiple memory reads to support the first stream and the second stream, and uses the DMA engine built into the memory interface to perform multiple memory reads. Implementing to support a first stream and a second stream. Further, in method C, the first stream experiences more latency than the second stream, and the first stream incorporates more buffers than the second stream to store tiles. In method C, the memory interface is also configured to prefetch an adjustable number of tiles of the first stream or second stream to compensate for the latency of the first stream or second stream. The

D) The methods described herein also broadly include methods of latency-tolerant video processing operations that use a host interface for communication between the video processor and the host CPU. Performing a scalar video processing operation by using a scalar execution unit coupled to the host interface, and using a vector execution unit coupled to the host interface. The communication between the video processor and the frame buffer memory, and the step of enabling the vector execution unit to operate on a demand driven basis by accessing the memory command FIFO. The steps implemented by using the interface and DMA transfers between different storage areas are incorporated into the memory interface and load the data execution memory and instruction cache into the data store memory and instruction cache. Implementing by using a DMA engine configured to: In the method D described above, the vector execution unit is further configured to operate asynchronously with respect to the scalar execution unit by accessing the command FIFO and operating on a request driven basis. In method D above, the request driven base is also configured to conceal the latency of data transfer from different storage to the command FIFO of the vector execution unit. Further, in the method D described above, the scalar execution unit is configured to perform the flow control processing of the algorithm, and the vector execution unit is configured to perform the majority of the video processing workload. Here, the scalar execution unit is configured to pre-calculate the working parameters of the vector execution unit to conceal the data transfer latency. In the method D above, the vector execution unit is configured to schedule memory reads via the DMA engine and prefetch commands for subsequent execution of the vector subroutine. Here, the memory read is scheduled to prefetch a command for execution of the vector subroutine prior to calling the vector subroutine by the scalar execution unit.

  [019] The present invention will now be described for purposes of illustration and not limitation in the figures of the accompanying drawings. In the drawings, like elements are referred to by like reference numerals.

Detailed Description of the Invention

  [026] Reference will now be made in detail to the preferred embodiments of the invention, examples of which are illustrated in the accompanying drawings. While the invention will be described in conjunction with the preferred embodiments, it will be understood that these embodiments are not intended to limit the invention thereto. On the contrary, the invention is intended to cover alternatives, modifications, and equivalents, which are included within the spirit and scope of the invention as defined by the appended claims. Furthermore, in the following detailed description of the embodiments of the present invention, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be appreciated by one skilled in the art that the present invention may be practiced without these specific details. In some instances, well known methods, procedures, components, and circuits have not been described in detail as not to unnecessarily obscure aspects of the embodiments of the present invention.

<Notation and terminology>
[027] Some portions of the detailed descriptions that follow are presented in terms of procedures, steps, logic blocks, processes, and other symbolic representations of operations on data bits within a computer memory. These descriptions and notations are the means used by those skilled in the data processing arts to most effectively convey the substance of their art to others skilled in the art. Procedures, computer-executed steps, logic blocks, processes, etc., are considered herein and generally a consistent series of steps or instructions that lead to a desired result. Steps are those requiring physical manipulation of physical quantities. Usually, though not necessarily, these quantities take the form of electronic or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

  [028] However, it should be borne in mind that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to those quantities. Unless otherwise specifically stated, as will be clear from the following description, the terms “process”, “access”, “execute”, “store”, “render”, etc. are used throughout the present invention. Similarly, data expressed as physical (electronic) quantities in computer system registers and memories are also represented as physical quantities in computer system memory or registers or other such information storage, transmission or display devices. It should be understood that it refers to the actions and processes of a computer system (eg, computer system 100 of FIG. 1) or similar electronic computing device that manipulates and converts to other data.

<Computer system platform>
[029] FIG. 1 illustrates a computer system 100 according to an embodiment of the invention. Computer system 100 illustrates the components of a basic computer system according to an embodiment of the present invention and provides an execution platform for specific hardware-based and software-based functions. In general, the computer system 100 includes at least one CPU 101, a system memory 115, at least one graphics processing unit (GPU) 110, and one video processing unit (VPU) 111. The CPU 101 can be coupled to the system memory 115 via the bridge component 105 or can be directly coupled to the system memory 115 via a memory controller (not shown) internal to the CPU 101. The bridge component 105 (eg, Northbridge) can support an expansion bus that connects various input / output devices (eg, one or more hard disk drives, Ethernet adapters, CD-ROMs, DVDs, etc.). GPU 110 and video processing device 111 are coupled to display 112. One or more additional GPUs can optionally be coupled to the system 100 to further enhance its computing power. The GPU 110 and the video processing unit 111 are coupled to the CPU 101 and the system memory 115 via the bridge component 105. The system 100 may be implemented, for example, as a desktop computer system or server computer system having a powerful general purpose CPU 101 coupled to a dedicated graphics rendering GPU 110. In such an embodiment, a component for adding a peripheral bus, a dedicated graphics memory and system memory, an input / output device and the like may be included. Similarly, the system 100 may be a handheld device (eg, a mobile phone, etc.) or provided by, for example, Xbox® provided by Microsoft Corporation of Redmond, Washington or Sony Computer Entertainment Corporation of Tokyo, Japan. It may be implemented as a set-top video game console device such as PlayStation3®.

  [030] The GPU 110 is a separate graphics card, separate integrated circuit die (eg, motherboard) designed to couple to the computer system 100 via separate components, connectors (eg, AGP slots, PCI-Express slots, etc.). It should be understood that it can be implemented as an integrated GPU included directly in the bridge chip 105 (e.g., embedded in the bridge chip 105) or directly on the integrated circuit die of a chipset component of a computer system. In addition, local graphics memory may be included for high bandwidth graphics data storage of the GPU 110. Further, the GPU 110 and the video processing unit 111 may be integrated into the same integrated circuit die (eg, as the component 120) or may be separate discrete integrated connected to or mounted on the motherboard of the computer system 100. It should be understood that it may be a circuit component.

Embodiment of the present invention

  [031] FIG. 2 is a diagram illustrating the internal components of the video processing unit 111 according to an embodiment of the present invention. As shown in FIG. 2, the video processing unit 111 includes a scalar execution unit 201, a vector execution unit 202, a memory interface 203, and a host interface 204.

  [032] In the embodiment of FIG. 2, the video processing unit (hereinafter simply video processor) 111 has functional components that perform video processing operations. The video processor 111 uses the host interface 204 to establish communication between the video processor 111 and the host CPU 101 via the bridge 105. Video processor 111 uses memory interface 203 to establish communication between video processor 111 and frame buffer memory 205 (eg, for a combined display 112 (not shown)). The scalar execution unit 201 is coupled to the host interface 204 and the memory interface 203 and is configured to perform scalar video processing operations. The vector execution unit is coupled to the host interface 204 and the memory interface 203 and is configured to perform vector video processing operations.

  [033] The embodiment of FIG. 2 illustrates a manner in which the video processor 111 divides its execution function into scalar operations and vector operations. Scalar operations are performed by the scalar execution unit 201. Vector operations are performed by the vector execution unit 202.

  [034] In one embodiment, vector execution unit 202 is configured to function as a slave coprocessor of scalar execution unit 201. In such an embodiment, the scalar execution unit manages the workload of the vector execution unit 202 by supplying a control stream to the vector execution unit 202 and managing the data input / output of the vector execution unit 202. To do. A control stream typically includes function parameters, subroutine arguments, and the like. In a typical video processing application, the control flow of the application processing algorithm is executed by the scalar execution unit 201, while the actual pixel / data processing operations are executed by the vector execution unit 202.

  [035] Still referring to FIG. The scalar execution unit 201 can be implemented as a RISC type scalar execution unit incorporating RISC-based execution technology. Vector execution unit 202 may be implemented, for example, as a SIMD machine having one or more SIMD pipelines. In two SIMD pipeline embodiments, for example, each SIMD pipeline can be implemented with a 16 pixel wide data path (or more), thus, computing power to produce a data output of up to 32 pixels per clock. To the vector execution unit 202. In one embodiment, scalar execution unit 201 comprises hardware that operates using VLIW (very long instruction word) software code and is configured to optimize parallel execution of scalar operations on a per clock basis. .

  In the embodiment of FIG. 2, scalar execution unit 201 includes an instruction cache 211 and a data cache 212 coupled to scalar processor 210. The caches 211 to 212 interface with a memory interface 203 for accessing an external memory such as the frame buffer 205, for example. The scalar execution unit 201 further comprises a vector interface unit 213 for establishing communication with the vector execution unit 202. In certain embodiments, the vector interface unit 213 can comprise one or more synchronous mailboxes 214 configured to allow asynchronous communication between the scalar execution unit 201 and the vector execution unit 202.

  [037] In the embodiment of FIG. 2, the vector execution unit 202 comprises a vector control unit 220, which controls the operation of the vector data path 221 that is a vector execution data path. It is configured. The vector control unit 220 includes a command FIFO 225 for receiving instructions and data from the scalar execution unit 201. Instruction cache 222 is coupled to provide instructions to vector control unit 220. The data store memory 223 is coupled to provide input data to the vector data path 221 and receive the resulting data from the vector data path 221. The data store 223 functions as an instruction cache and data RAM for the vector data path 221. Instruction cache 222 and data store 223 are coupled to a memory interface 203 for accessing external memory, such as frame buffer 205. The embodiment of FIG. 2 also shows a second vector data path 231 and a corresponding second data store 233 (eg, a dotted outline). The second vector data path 231 and the second data store 233 are shown to illustrate the case where the vector execution unit 202 has two vector execution pipelines (eg, a duplexed SIMD pipeline configuration). I want you to understand. Embodiments of the present invention are suitable for vector execution units having multiple vector execution pipelines (eg, 4, 8, 16, etc.).

  [038] The scalar execution unit 201 provides data and command inputs to the vector execution unit 202. In one embodiment, scalar execution unit 201 sends a function call to vector execution unit 202 using memory map command FIFO 225. The command of the vector execution unit 202 is put in the queue of this command FIFO 225.

  The use of command FIFO 225 effectively separates scalar execution unit 201 from vector execution unit 202. The scalar execution unit 201 functions with its own individual clock, which is different from the clock frequency of the vector execution unit 202 and is controlled independently of the clock frequency of the vector execution unit 202. Can operate at frequency.

  [040] Command FIFO 225 enables vector execution unit 202 to operate as a request drive unit. For example, work is passed from the scalar execution unit 201 to the command FIFO 225 and then accessed by the vector execution unit 202 for processing in a separate asynchronous manner. Accordingly, the vector execution unit 202 will process the workload as needed by the scalar execution unit 201 or upon request. Such a function allows the vector execution unit 202 to save power when maximum performance is not required (eg, by mitigating / or stopping one or more internal clocks).

  [041] A video processing program built for video processor 111 by dividing the video processing function into a scalar part (eg, for execution by scalar execution unit 201) and a vector part (eg, for execution by vector execution unit 202) Can be compiled into separate scalar software code and vector software code. Scalar software code and vector software code can be compiled separately and then linked to form a consistent application.

  [042] This partitioning allows vector software code functions to be written separately to distinguish them from scalar software code functions. For example, vector functions can be written individually (eg, at different times by different teams of engineers, etc.) and provided as one or more subroutines or library functions for use with / with scalar functions (eg, scalar threads, processes, etc.) It is possible. This allows for independent and independent updates of scalar software code and / or vector software code. For example, vector subroutines can be updated independently of scalar subroutines (eg, through updates to previously distributed programs, new features added to enhance the functionality of distributed programs, etc.). The reverse is also possible. Partitioning is facilitated by the scalar processor 210 (eg, caches 221-212) and separate caches of vector control unit 220 and vector data path 221 (eg, caches 222-223). As described above, the scalar execution unit 201 and the vector execution unit 202 communicate via the command FIFO 225.

  [043] FIG. 3 illustrates an exemplary software program 300 of the video processor 111 according to an embodiment of the present invention. As shown in FIG. 3, the software program 300 represents the characteristics of the programming model of the video processor 111, whereby the scalar control thread 301 is executed by the video processor 111 along with the vector data thread 302.

  [044] The example software program 300 of the embodiment of FIG. 3 represents the programming model of the video processor 111 so that the scalar control program (eg, scalar control thread 301) on the scalar execution unit 201 is: A subroutine call (eg, vector data thread 302) in the vector execution unit 202 is executed. The example software program 300 illustrates the case where a compiler or software programmer breaks a video processing application into a scalar part (eg, a first thread) and a vector part (eg, a second thread).

  [045] As shown in FIG. 3, the scalar control thread 301 executing on the scalar execution unit 201 calculates work parameters in advance and uses these parameters for the vector execution unit 202 that performs most of the processing work. To supply. As described above, the software code for the two threads 301 and 302 can be written and compiled separately.

[046] A scalar thread is responsible for:
1. 1. Interface with host unit 204 and implement class interface 2. Initialization, setup and configuration of vector execution unit 202 Run the algorithm in a unit of work, chunk, or working set in a loop so that each iteration does the following: a. Calculate the parameters of the current working set b. Initiate transfer of input data to the vector execution unit c. Start transfer of output data from the vector execution unit

  [047] The standard execution model for scalar threads is "fire-and-forget". The term response-free transmission means that in the standard model of video baseband processing, commands and data are transmitted from the scalar execution unit 201 to the vector execution unit 202 (eg, via the command FIFO 225), completing the algorithm. Up to this point, there is no return data from the vector execution unit 202.

  [048] In the example program 300 of FIG. 3, the scalar execution unit 201 continues to schedule work for the vector execution unit 202 until there is no more space in the command FIFO 225 (eg,! End_of_alg &! Cmd_fifo_full). The work scheduled by the scalar execution unit 201 calculates parameters, sends them to the vector subroutine, and then calls the vector subroutine to perform the work. Execution of a subroutine (eg, vector_funcB) by the vector execution unit 202 is delayed by an appropriate amount of time, mainly to hide latency from the main storage (eg, system memory 115). Thus, the architecture of the video processor 111 provides a latency compensation mechanism on the vector execution unit 202 side for both instruction and data traffic. These latency compensation mechanisms are described in more detail below.

  [049] The example software program 300 would be more complicated if there are two or more vector execution pipelines (eg, the vector data path 221 and the second vector data path 231 of FIG. 2). Please keep in mind. Similarly, the example software program 300 is written for a computer system that has two vector execution pipelines, but still retains the ability to execute on a system that has a single vector execution pipeline. If so, it will be more complicated.

Accordingly, as described above in the description of FIGS. 2 and 3, the scalar execution unit 201 is responsible for initiating the calculation of the vector execution unit 202. In one embodiment, there are the following main types of commands passed from the scalar execution unit 201 to the vector execution unit 202:
1. A read command (eg, memRd) initiated by the scalar execution unit 201 to transfer the current working set data from memory to the data RAM of the vector execution unit 202
2. 2. Parameter passing from the scalar execution unit 201 to the vector execution unit 202 3. Execution command in the form of a PC (eg, program counter) of the vector subroutine to be executed A write command (eg, memWr) initiated by the scalar execution unit 201 to copy the result of the vector calculation to memory

  [051] In some embodiments, upon receiving these commands, the vector execution unit 202 immediately schedules a memRd command to the memory interface 203 (eg, reading requested data from the frame buffer 205, etc.). Vector execution unit 202 also examines the execute command and prefetches the vector subroutine to be executed (if not present in cache 222).

  [052] The purpose of the vector execution unit 202 in this situation is to schedule the next several execution instructions and data streams in advance while the vector execution unit 202 is engaged in the current execution. This pre-scheduling function effectively hides the latency required to fetch instructions / data from its storage. To perform these read requests in advance, the vector execution unit 202, the data store (eg, data store 223), and the instruction cache (eg, cache 222) are implemented by using fast optimization hardware.

  [053] As described above, the data store (eg, data store 223) functions as a working RAM for the vector execution unit 202. The scalar execution unit 201 recognizes the data store and interacts with the data store as if it were a collection of FIFOs. The FIFO includes a “stream” through which the video processor 111 operates. In one embodiment, the stream is generally an input / output FIFO that the scalar execution unit 201 initiates transfer (eg, to the vector execution unit 202). As described above, the operations of scalar execution unit 201 and vector execution unit 202 are separated.

  [054] When the input / output stream is full, the DMA engine in the vector control unit 220 stops processing the command FIFO 225. As a result, the command FIFO 225 will soon be full. The scalar execution unit 201 stops issuing further work to the vector execution unit 202 when the command FIFO 225 becomes full.

  [055] In one embodiment, vector execution unit 202 requires an intermediate stream in addition to the input and output streams. Thus, the entire data store 223 can be viewed as a collection of streams for interaction with the scalar execution unit 201.

  [056] FIG. 4 illustrates an example of blending (blending) sub-pictures with video using a video processor according to an embodiment of the present invention. FIG. 4 shows an exemplary case where a video surface is mixed with a sub-picture and then converted to an ARG surface. Data including the surface exists in the frame buffer memory 205 as a luminance parameter 412 and a chromaticity parameter 413. A sub-picture pixel element 414 is also present in the frame buffer memory 205 as shown. Vector subroutine instructions and parameters 411 are instantiated in memory 205 as shown.

  [057] In one embodiment, each stream comprises a working 2D chunk FIFO of data called "tiles". In such an embodiment, the vector execution unit 202 maintains a read tile pointer and a write tile pointer for each stream. For example, for an input stream, when the vector subroutine is executed, the vector subroutine can consume or read the current (read) tile. In the background, data is transferred to the current (write) tile by the memRd command. The vector execution unit can also generate output tiles for the output stream. These tiles are then moved to memory by the memWr () command following the execute command. This effectively prefetches tiles and prepares them to be manipulated, effectively hiding latency.

  [058] In the example of sub-picture mixing of FIG. 4, the vector data path 221 is composed of instantiated instances of vector subroutine instructions and parameters 411 (eg, & v_subp_blend). This is indicated by line 421. Scalar execution unit 201 reads chunks (eg, tiles) of the surface and loads them into data store 223 using DMA engine 401 (eg, in memory interface 203). The load operation is indicated by line 422, line 423, and line 424.

  [059] Still referring to FIG. Since there are multiple input surfaces, multiple input streams need to be maintained. Each stream has a corresponding FIFO. Each stream can have a different number of tiles. In the example of FIG. 4, the sub-picture surface is in system memory 115 (eg, sub-picture pixel element 414) and thus has additional buffering (eg, n, n + 1, n + 2, n + 3, etc.), but the video stream (E.g., luminance 412, chromaticity 413, etc.) indicates a case where a smaller number of tiles can be provided. The number of buffers / FIFOs used can be adjusted according to the degree of latency experienced by the stream.

  [060] As described above, the data store 223 uses a look-ahead prefetch method to conceal latency. This allows the stream to have the data in more than one tile when the data is prefetched to the appropriate vector data path execution hardware (eg, shown as FIFO n, n + 1, n + 2, etc.) )

  [061] When the data store is loaded, the FIFO is accessed by the vector data path hardware 221 and manipulated by a vector subroutine (eg, subroutine 430). The result of the vector data path operation constitutes the output stream 403. This output stream is copied by the scalar execution unit 201 via the DMA engine 401 and returned to the frame buffer memory 205 (for example, ARGB_OUT 415). This is indicated by line 425.

  [062] Thus, embodiments of the present invention use an important feature of stream processing, which is that data storage and memory are abstracted as multiple memory titles. Thus, a stream can be viewed as a collection of tiles that are accessed sequentially. The stream is used to prefetch data. This data takes the form of tiles. Tiles are prefetched to conceal latency from specific memory sources from which data originates (eg, system memory, frame buffer memory, etc.). Similarly, streams can be directed to different locations (eg, vector execution unit cache, scalar execution unit cache, frame buffer memory, system memory, etc.). Another feature of the stream is to access the tiles generally in a look-ahead prefetch mode. As described above, the higher the latency, the deeper the prefetch and the more buffering used per stream (eg, as shown in FIG. 4).

  [063] FIG. 5 is a diagram illustrating the internal components of a vector execution unit according to an embodiment of the invention. FIG. 5 illustrates the configuration of the various functional units and register / SRAM resources of the vector execution unit 202 from a programming perspective.

  [064] In the embodiment of FIG. 5, vector execution unit 202 comprises a VLIW digital signal processor optimized for video baseband processing performance and various codec (compression-decompression algorithm) execution. Thus, the vector execution unit 202 has a number of characteristics that are aimed at increasing the efficiency of video processing / codec execution.

[065] In the embodiment of FIG. 5, the characteristics include the following.
1. 1. Extended performance by providing options for incorporating multiple vector execution pipelines 2. Assignment of two data address generators (DAG) per pipe Memory / Register Operand 4.2D (x, y) Pointer / Iterator
5. 5. Deep pipeline (eg, 11-12) stage 6. Scalar (integer) / branching unit Variable instruction width (Long / Short instruction)
8). 8. Data aligner for operand extraction Standard operand and resulting 2D data path (4x4) shape 10. Slave vector execution unit to scalar execution unit to execute remote procedure calls

  [066] In general, from the programmer's perspective, the vector execution unit 202 is viewed as a SIMD data path comprising two DAGs 503. Instructions are issued in a VLIW manner (eg, instructions are issued simultaneously to the vector data path 504 and the address generator 503), decoded by the instruction decoder 501, and dispatched to the appropriate execution unit. The instructions are variable length, and the most commonly used instructions are encoded in short form. The complete instruction set is available in long form as a VLIW type instruction.

  [067] Legend 502 shows three clock cycles with three such VLIW instructions. According to legend 510, the top of VLIW instruction 502 comprises two address instructions (eg, for two DSGs 503) and one instruction for vector data path 504. The intermediate VLIW instruction includes one integer instruction (for example, for the integer unit 505), one address instruction, and one vector instruction. The lowest VLIW instruction comprises one branch instruction (eg, for branch unit 506), one address instruction, and one vector instruction.

  [068] The vector execution unit may be configured to have a single data pipe or multiple data pipes. Each data pipe consists of local RAM (eg, data store 511), crossbar 516, two DAGs 503, and SIMD execution units (eg, vector data path 504). FIG. 5 shows a basic configuration for purposes of illustration, where only one data pipe is instantiated. When two data pipes are instantiated, they can run as independent threads or as joint threads.

  [069] Six different ports (eg, four reads and two writes) can be accessed via the address register file unit 515. These registers receive parameters from a scalar execution unit or from the result of integer unit 505 or address unit 503. The DAG 503 also functions as a collection controller, managing register placement and addressing the contents of the data store 511 (eg, RA0, RA1, RA2, RA3, WA0, and WA1). Crossbar 516 is coupled to execute predetermined instructions by assigning output data ports R0, R1, R2, R3 to vector data path 504 in any order / combination. The output of the vector data path 504 can be fed back to the data store 511 as shown (eg, W0). The constant RAM 517 is used to supply frequently used operands from the integer unit 505 to the vector data path 504 and the data store 511.

  [070] FIG. 6 is a diagram illustrating a layout of a data store having multiple banks 601-604 of memory 600 and symmetrically arranged tiles 610, according to an embodiment of the present invention. As shown in FIG. 6, only a portion of the data store 610 is shown for illustrative purposes. The data store 610 logically comprises an array (or multiple arrays) of tiles. Each tile is an array of 4 × 4 sub-tiles. Physically, as indicated by memory 600, data store 610 is stored in an array of “N” physical banks (eg, banks 601-604) of memory.

  [071] In addition, the data store 610 visually shows the logical tiles in the stream. In the embodiment of FIG. 6, this tile is 16 bytes high and 16 bytes wide. This tile is an array of subtiles (4 × 4 in this example). Each subtile is stored in a physical bank. This is indicated in FIG. 6 by the number in each 4 × 4 subtile when there are 8 banks of physical memory (eg, banks 0-7). The organization of the subtiles in the bank is such that there is no common bank in the 2 × 2 subtitles. This allows misaligned accesses (eg, in the x and y directions) without causing bank collisions.

  [072] Banks 601-604 are configured to support access to different tiles in each bank. For example, in one instance, the crossbar 516 can access a set of 2 × 4 tiles from the bank 601 (eg, the first two rows of the bank 601). In another case, the crossbar 516 can access a set of 1 × 8 tiles from two adjacent banks. Similarly, in another case, the crossbar 516 can access an 8 × 1 set of tiles from two adjacent banks. In either case, the DAG / collector 503 can receive tiles as the bank is accessed by the crossbar 516 and supply the tiles to the front end of the vector data path 504 every clock.

  [073] Thus, embodiments of the present invention utilize a new video processor architecture that supports high performance video processing functions and that efficiently utilizes integrated circuit silicon die area, transistor count, memory speed requirements, and the like. provide. Embodiments of the present invention retain high computational density and can be easily extended to handle multiple video streams. The embodiment of the present invention is, for example, MPEG-2 / WMV9 / H. H.264 encoding assist (for example, in-loop decoder), MPEG-2 / WMV9 / H.264. A number of high performance video processing operations can be provided, such as H.264 decoding (eg, post-entropy decoding) and in-loop / out-of-loop deblocking filters.

  [074] Further video processing operations provided by embodiments of the present invention include, for example, high performance motion adaptive progressive scan conversion, input noise filtering for encoding, polyphase scaling / resampling, and sub-picture synthesis. There is. The video processor architecture of the present invention is also suitable for specific video processor amplifier applications such as color space correction, color space adjustment, pixel point operations such as sharpening and histogram adjustment, and various video surface format conversions. Can also be used.

  [075] Broadly and without limitation, this document discloses the following: That is, a latency-tolerant system that performs video processing operations is described. The system is coupled to a host interface that implements communication between the video processor and the host CPU, a scalar execution unit coupled to the host interface and configured to perform scalar video processing operations, the host interface, and A vector execution unit configured to perform vector video processing operations. A command FIFO is provided to allow the vector execution unit to operate on a request driven basis by accessing the memory command FIFO. A memory interface is provided for performing communication between the video processor and the frame buffer memory. A DMA engine is incorporated into the memory interface to perform DMA transfers between different storage locations and load vector execution unit data and instructions into the command FIFO. A video processor that performs video processing operations is also described. The video processor includes a host interface that implements communication between the video processor and the host CPU. A memory interface is provided for performing communication between the video processor and the frame buffer memory. A scalar execution unit is coupled to the host interface and the memory interface and is configured to perform scalar video processing operations. A vector execution unit is coupled to the host interface and the memory interface and is configured to perform vector video processing operations. A multi-dimensional data path processing system for a video processor that performs video processing operations is also described. The video processor comprises a scalar execution unit configured to perform scalar video processing operations and a vector execution unit configured to perform vector video processing operations. A data store memory is provided for storing vector execution unit data. The data store memory comprises a plurality of tiles having a symmetric bank data structure arranged in an array. The bank data structure is configured to support access to different tiles in each bank. A stream-based memory access system for a video processor that performs video operations is also described. The video processor comprises a scalar execution unit configured to perform scalar video processing operations and a vector execution unit configured to perform vector video processing operations. A frame buffer memory is provided for storing scalar execution unit and vector execution unit data. A memory interface is provided for establishing communication between the scalar execution unit, the vector execution unit, and the frame buffer memory. The frame buffer memory includes a plurality of tiles. The memory interface implements a first sequential access of tiles and a second stream including a second sequential access of tiles of vector execution units or scalar execution units.

  [076] The foregoing descriptions of specific embodiments of the present invention have been presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the invention to the precise forms disclosed, and many modifications and variations are possible under the above teachings. This embodiment best describes the principles of the invention and its practical application, so that those skilled in the art can make various modifications to the invention and various embodiments to suit the particular use contemplated. It has been chosen and described in order to be best used. It is intended that the scope of the invention be defined by the claims appended hereto and their equivalents.

1 Overview VP2 is a VLIW SIMD video DSP coupled to a scalar control processor. Its main focus is video codec and video baseband processing.

1.1 VP2.0 Core / Efficiency: VP2.0 is a computationally efficient machine for video applications with respect to perf / mm2 and perf / mW. Program Capability: VP2.0 has high programming capabilities Can be easily compiled and made more secure to the program machine. Extensibility: VP2.0 design / architecture should be extensible to meet the performance requirements of multiple application areas.

1.2 Design Goals / Calculation Density Provides perf / mm2 advantage over VP1.0. Efficient implementation of new application fields such as H.264 HW latency tolerance to reduce the burden on SW developers Hiding data fetch latency by reordering memory access and computation Automatic prefetching of instruction streams Hide data path latency Selective transfer of intermediate results Streaming computation model / extensibility:
Architecturally, the VP2 vector unit can expand its data path by a factor of 2 and reduce it by a factor of 2. Frequency improvement can be achieved by selective re-pipelining

1.3 Application Target The design and instruction set of VP2.0 is optimized to perform the following applications very efficiently.
Mpeg2 / wmv9 / H. H.264 encoding assist (in-loop decoder)
Mpeg2 / wmv9 / H. H.264 decoding (post-entropy decoding)
・ In-loop / out-of-loop deblocking filter ・ High-performance motion adaptive progressive scan conversion ・ Encoding input noise filtering ・ Polyphase scaling / resampling ・ Sub-picture synthesis ・ procamp, color space conversion, color space adjustment, sharpening Hexe point operation such as histogram adjustment, various video surface format conversion support

Architecturally, VP2.0 can be efficient in the following areas:
-2D primitive, blitz, rotation, etc.-Fine-tuned software motion estimation algorithm-16 / 32-bit MAC application

ＶＰ２．０プログラムは、別個のスカラーコード及びベクトルコードにコンパイルされ、後に結合することができる。別々に、ベクトル関数は、個々に書き込まれ、サブルーチン又はライブラリ関数としてスカラースレッドに供給することができる。スカラープロセッサは、独自の命令及びデータキャッシュを有する。ベクトルユニットもまた、命令キャッシュ及びデータＲＡＭ（データストアと呼ばれる）を有する。これらの二つのエンジンは分離され、ＦＩＦＯを通じて通信する。 2 Top Level Architecture The VP2.0 machine is divided into a scalar processor and a vector processor. The vector processor operates as a slave coprocessor for the scalar processor. The scalar processor is responsible for supplying control streams (parameters, subroutine arguments) to the vector processor and managing data input and output to the vector processor. All control flow of the algorithm is performed on a scalar machine, but the actual pixel / data processing operations are performed on a vector processor.
The scalar processor is a standard RISC style scalar and the vector coprocessor is a SIMD machine with one or two SIMD pipes (each SIMD pipe has a 16 pixel data path). Thus, the vector coprocessor can create a processing result of up to 32 pixels as raw computing power.
The scalar processor uses the memory mapped command FIFO to send function calls to the vector coprocessor. Coprocessor commands are queued in this FIFO. The scalar processor is completely separated from the vector processor using this FIFO. A scalar processor can operate with its own clock. The vector processor operates as a request drive unit.
A top level view of VP2.0 is shown below.

The VP2.0 program can be compiled into separate scalar and vector code and later combined. Separately, vector functions can be written individually and supplied to scalar threads as subroutines or library functions. A scalar processor has its own instruction and data cache. The vector unit also has an instruction cache and a data RAM (called a data store). These two engines are separated and communicate through a FIFO.

より複雑なプログラミングモデルは、二つのデータパイプを有する場合である。或いは、二つのデータパイプのコードを書いて、そのコードを一つのデータパイプマシン上で実行する場合である。そのプログラミングモデルを、第６節において考察する。 3 Simple Programming Model The simplest programming model of VP2.0 is a scalar control program that executes subroutine calls on a vector slave coprocessor. Here is an essential assumption that the programmer breaks down the problem into these two threads. A thread executing on a scalar processor calculates work parameters in advance and supplies them to a main vector processor. These two threaded programs are expected to be written and compiled separately.
The scalar thread is responsible for:
1. Take the interface with the host unit and implement the class interface. 2. Vector unit initialization, setup and configuration. Run the algorithm in a unit of work, chunk, or working set in a loop so that each iteration does the following: a. Calculate current working set parameters b. Start transfer of input data to vector processor c. Start transfer of output data from vector processor The standard execution model of scalar threads is response-free transmission. This is expected to be a standard model for video baseband processing in the absence of return data from the vector coprocessor. The scalar processor continues to schedule work for the vector processor as long as there is space in the command FIFO. The execution of the subroutine by the vector processor is delayed mainly by the time due to the waiting time from the main memory. Therefore, it is important to provide a latency compensation mechanism on the vector side. In VP2.0, the vector processor provides latency compensation for both instruction and data traffic. The mechanism is outlined in the section.
A standard VP program is as follows.

A more complex programming model is the case with two data pipes. Alternatively, code for two data pipes is written and the code is executed on one data pipe machine. The programming model is discussed in Section 6.

4 Streaming model As outlined above, the scalar engine is responsible for initiating calculations on a vector processor. There are the following main types of commands passed from the scalar engine to the vector engine.
1. Read command (memRd) initiated by scalar to transfer current working set data from memory to vector engine data RAM
2. 2. Parameters passed from scalar to vector 3. Execution command in PC format of vector subroutine to be executed A write command (eg memWr) initiated by a scalar to copy the result of the vector calculation to memory
Upon receipt of these commands, the vector processor immediately schedules a memRd command to the frame buffer (FB) interface. The vector processor also examines the execute command and prefetches the vector subroutine to be executed (if it is not in the cache). One goal is to schedule the next few execution instructions and data steam in advance while the vector engine is engaged in the current execution. In order to make these read requests in advance, the vector engine manages a data store and instruction cache in hardware.
The data store is the working RAM of the vector processor. The scalar processor sees this data store as a FIFO or a collection of streams. The stream is essentially an input / output FIFO where the scalar begins to transfer. When the I / O stream is full, the vector DMA engine stops processing the command FIFO from the scalar and it will soon be full. Thus, the scalar stops issuing further work to the vector engine. In addition to input and output streams, vectors may require intermediate streams. Thus, the entire data store can be viewed as a collection of streams from the scalar side. Each stream is a working 2D chunk FIFO called a tile. The vector processor maintains a read tile pointer and a write tile pointer for each stream. For an input stream, when the vector subroutine is executed, the vector subroutine can consume or read the current (read) tile. In the background, data is transferred to the current (write) tile by the memRd command. The vector processor can also generate output tiles for the output stream. These tiles are then moved to memory by the memWr () command following the execute command.
This model is illustrated by an example of mixing sub-pictures with video. For example, consider a simplified example where a video surface (eg, NV12 format) is mixed with a subpicture and then converted to an ARGB surface. These surfaces exist in memory. The scalar processor reads these surface chunks (tiles) and loads them into the data store. Since there are multiple input surfaces, it is necessary to hold multiple input streams. Each stream can have a different number of tiles (eg, in this example, assume that the sub-picture surface is in system memory and therefore should be buffered further), but the video stream is more May have a small number of tiles.

ベクトルユニットは、単一のデータパイプ又は二重のデータパイプをインスタンス化する。各データパイプは、ローカルＲＡＭ（データストア）、二つのＤＡＧ、及びＳＩＭＤ実行ユニットから成る。基本構成では、一つのデータパイプのみが存在する。二つのデータパイプが存在する場合、それらは独立したスレッドとして、又は共同のスレッドとして実行することができる。ベクトルプロセッサの完全なパイプラインの図を以下に示す。これは、二つのデータパイプを備える完全構成である。

5 Vector Coprocessor The vector coprocessor of VP2 is a VLIW DSP designed for video baseband processing and codecs. Some important design characteristics of the processor include:
1. Highly scalable performance, one or two data pipes 2. Each pipe has two data address generators (DAG). 4. Memory / register operands 4.2D (x, y) pointer / iterator 5. Deep pipeline (11-12) stage 6. Scalar (integer) / branching unit Variable instruction width (Long / Short instruction)
8). 8. Data aligner for operand extraction Standard operand and resulting 2D data path (4 × 4) shape Slave processor to scalar processor that performs remote procedure calls The vector coprocessor from the programmer's point of view is simply a SIMD data path with two DAGs. Instructions are issued in a VLIW manner (ie, instructions are issued simultaneously to the vector data path and address generator). The instructions are variable length and the most commonly used instructions are encoded in short form. The complete instruction set is available in long form. For example, from the programmer's point of view, the configuration of the various functional units and registers / SRAM resources is as follows.

A vector unit instantiates a single data pipe or a double data pipe. Each data pipe consists of a local RAM (data store), two DAGs, and a SIMD execution unit. In the basic configuration, there is only one data pipe. If there are two data pipes, they can run as independent threads or as joint threads. A complete pipeline diagram of the vector processor is shown below. This is a complete configuration with two data pipes.

５．プログラムは、二つの共同するスレッドを用いて作成することができる。これは、単一のスカラー制御スレッドを有するが、複数の機能ベクトルの機能ブロックが相互に接続される必要である場合のコーデックに期待されるモデルである。これは、機能ブロックのＤｉｒｅｃｔＳｈｏｗのｐｉｎモデルと類似している。そのようなアプリケーションの例を、以下に示す。このモデルは、二つのデータパイプしか備えていないので、二つの共同のスレッドのみに制限される。更に注意することは、スレッドが二つのスレッド間でバランスを取る必要があることである。そのようにしないと、性能が損なわれる。これらの制約の範囲内で、このモデルは二つのデータパイプで機能し、さらに単一パイプに縮小することができる。

６．二つのデータパイプは、相互に同期することができる。同期化の基本的な手法は、データ駆動である。ベクトル関数は、データが処理に使用可能である場合に実行される。ストリームは、メモリからの読み取り、又は他のデータパイプからの書き込みによって満たされる。データが使用可能となると、ベクトル制御ユニットは実行をアクティブ化してそれを動作させる。ストリームはまた、計数セマフォーとして使用することもできる。スカラーコントローラ及びベクトルデータパイプは何れも、タイルポインタを増分及び減分して、データ転送が行われない場合にも計数セマフォーとしてストリーム記述子を使用することができる。 6 Advanced programming model In Section 3, the RPC model was presented to illustrate the basic architecture. This section presents more advanced concepts.

6.1 Dual data pipe configuration In the dual pipe configuration, the following resources of the processor are shared.
-Scalar controller-Vector coprocessor vector control unit-DMA engine for instruction / data fetch-Instruction cache (may be dual port)
The following resources are replicated:
Data pipe (address / branch / vector execution unit)
• Data store • Register file Note the following items.
1. A program can be created for two pipes in an instance that uses one pipe. The vector control unit maps the execution of each pipe onto the same physical pipe. However, since the streams for both pipes exist only in one data store, it is necessary to adjust the size of the data store. A simple way is to reduce the size or number of tiles in the stream by half. This is done by a scalar thread at setup time. There are problems such as global register duplication and stream mapping that need to be resolved in the macro architecture stage.
2. A program created for one pipe can be executed on an instance with two pipes. However, this code works with only one pipe and does not use the other pipe. The machine becomes semi-idle.
3. A program can be created for two pipes, each running two completely different threads. This may be undesirable because it has only a single scalar that is not multi-threaded. Although this may not be desirable because it only supports one scalar execution thread, it can support this model.
4). Programs can be created for two pipes, each executing the same thread. This is the standard model expected for parallelizable algorithms, such as many video baseband processing. This allows the same instruction stream to be used to manipulate two strips of video or bisection and the like. Each data pipe has its own execution unit and data store. The scalar controller needs to supply two data pipes. However, because the parameters, read and write commands are interrelated (offset), the scalar performance requirement does not double exactly. An example of this model is shown below

5. A program can be created using two collaborative threads. This is the model expected for a codec when it has a single scalar control thread, but the function blocks of multiple function vectors need to be interconnected. This is similar to the DirectShow pin model of the functional block. An example of such an application is shown below. Since this model has only two data pipes, it is limited to only two joint threads. Note further that the thread needs to be balanced between the two threads. Otherwise, performance will be compromised. Within these constraints, the model works with two data pipes and can be further reduced to a single pipe.

6). The two data pipes can be synchronized with each other. The basic method of synchronization is data driving. A vector function is executed when data is available for processing. The stream is filled by reading from memory or writing from other data pipes. As data becomes available, the vector control unit activates execution and runs it. The stream can also be used as a counting semaphore. Both the scalar controller and the vector data pipe can increment and decrement the tile pointer and use the stream descriptor as a counting semaphore even when no data transfer takes place.

Supplemental overview:
In general, embodiments of the present invention do the following:
1. Decompose the media algorithm into scalar and vector parts.
An off-the-shelf scalar design, which also provides the ability to run scalar and vector parts at different clock speeds based on power and performance requirements.
2. 3. Stream processing 3.2D data path processing Latency hiding (for both data and command fetches)

Application areas
Cipher:
Hiding the instruction code The encryption program can simply be placed on the chip. The scalar / controller block simply requests that a specific operation be performed, and the encryption engine fetches instructions and the like. A scalar is very safe because it cannot even see which algorithm is running. This provides a mechanism to hide the encryption algorithm from the user.
2D
The VP2 instruction set architecture supports 2D processing instructions. These include support for ROP3 and ROP4 used in many GUI / window systems. This allows the media processor to perform 2D operations on the media processor. A unique advantage here is labor saving.

ISA
Condition code as instruction slot:
Provide separate issue slots (in multiple issue instruction bundles) for condition code operations. The prior art uses SIMD instructions that can also affect the condition code / predicate register. However, with VP2 data processing and the approach performed with predicate registers, processing can be scheduled independently, resulting in higher performance.

Memory I / O
Microcoded DMA engine:
The DMA engine can be programmed to perform a variety of operations (or have its own small microcode), such as prefetching data in a stream, data format formatting, edge padding, etc. Generally, it is a programmable DMA engine, not a hardwired function. Thus, the combination of a memory input / output processor and a media processing core enhances overall system level performance. The media processor core reduces the load because it needs to perform data input / output processing.

Storage hierarchy architecture:
In the VP2 architecture, the storage hierarchy is optimized to minimize the memory BW and provide latency compensation. A variety of methods are provided as follows.
A first level of streaming data store that is visible to the vector core as a scratch ram. Managed by the HW to examine in advance the request stream generated by the scalar processor. This data store is optionally supported by an L2 cache for data reuse. The L2 cache can be divided into individual sectors on a stream basis.
-L1 cache supported by a streaming data store. The data store is prefetching the next related data set.
-A cache that uses a Stream pointer as a data tag.
-Use of scalar generated stream address to prefetch / cache L1 data store and L2 cache.

Optimized scalar for vector communication links:
MemRd / Wr format Short command from scalar to read and write between system memory and local memory. Saves the control flow bandwidth required to manage the DMA engine. At the same time, it does not limit the types of transactions that are supported.

Scalar 2 vector guess for vector L2

Parameter compression with support for parameter modifiers and iterators to reduce communication bandwidth.

Pipeline cache:
Pipeline instruction cache. Various schemes are supported:
-Manage the life cycle of each cache line by tracking operational execution between the vector and the scalar processor. This allows the instructions to be ready before the vector processor begins execution. If the instruction is not already in the cache, it is prefetched.
-For small latency configurations, instruction cache is minimized by changing to a smaller FIFO. Execution already in the FIFO can be reused, otherwise it is fetched again.

General Architecture Data stores can be shared between various processing elements. These can be communicated through a stream and supplied to each other. The architecture assumes a heterogeneous set of functional units such as SIMD vector cores, DMA engines, fixed functional units connected through streams.

Calculation / DP
Arbitrary / Flexible format / Halfpipe The data path operates in various formats. The format of the data path can be configured to match the problem set. Usually, a 1D data pass is performed. VP2 processes a format that can be a variable size, such as 4 × 4, 8 × 4, or 16 × 1, to fit the algorithm.

Scalability The VP2 datapath architecture uses instruction transport techniques (Note: with 16-way SIMD pipes, each operand is 1 byte wide. With 8-way SIMD pipes (bundling two pipes) Each operand can have a wider SIMD data path of 2 bytes, as well as a 4-way SIMD pipe (collecting 4 pipes), each operand can have a wider SIMD data path of 4 bytes. ) Wider SIMD instructions can be executed with narrower data paths over multiple cycles to save space.
For example, VP2 can scale the data path from 16-way SIMD to 8-way SIMD.

Combine byte lanes Combine SIMD ways to increase operand width. Example: 16-way SIMD currently with 8-bit operands. It can be expanded to 16-bit operands with 8-way SIMD and 32-bit operands with 4-way SIMD.

SIMD address generator A separate stream address generator for each way of the SIMD pipe.
VP2 can use a SIMD address generator where requests are combined into a minimum access to the data store.

Data expansion using crossbars and collectors The ability to create more data operands using crossbars. Reduces read port pressure to save power.

X2 instruction:
Not all instructions can use all HW elements (adders / multipliers) in the data path. Therefore, for simple instructions such as addition / subtraction, a wider data format can be processed than in the case of complex instructions. Therefore, instead of limiting performance to the smallest common size, VP2 uses a flexible instruction set that conveniently attempts to operate in a wider format as long as the read port can maintain the operation bandwidth.

Multi-thread / multi-core media processing The VP2 architecture supports various multi-threading options, including:
A multi-thread scalar processor schedules a procedure call with multiple vector units connected through a stream.
Multiple threads executing on a single vector engine with / with instructions or by execution thread switching.

Power management using different vectors / scalars Separated scalar and vector parts allow these two blocks to run at different speeds based on power and performance requirements.

Context switch:
This media processor has the ability to support very fast context switches due to its low register architecture. HW support exists to track, save and replay a scalar-vector command queue to achieve context switching. A context switch can also be initiated with a page fault.
This allows the media processor to hold real-time processing tasks such as input / output display processing, and at the same time non-real-time video extensions such as just-in-time video extensions to provide 2D acceleration or display pipelines. It can also support tasks.
This context switch function, along with its instruction set, allows VP2 to be an integrated pixel / codec process.

Data store organization:
VP2 uses a data store organization with the following characteristics:
Up to 16 pixels in each direction can be accessed without causing bank contention. This is done while minimizing stride requirements.
Data store organization enables efficient replacement of data formats.
2D addressing is supported in the data store and eliminates the linear address SW calculation in many media processing applications such as video.

  application

  architecture

High computational density ・ perf / mm2 twice that of VP1 architecture
・ Efficient with new applications (compared to fixed function HW)
Program Capability • Compile “reasonably”: C for scalar, unique for vector • HW-managed latency hiding • Scalability protection against hangs and illegal addresses • Double and _ times expansion options • Clock frequency Supports high-precision operations that can be extended through 10b + 20b integer data types

Block overview-Scalar processor-Program flow control-Tile rasterization and address calculation-Supply loosely coupled vector units-Vector processor-3 issue VLIW (v + I, A, B)
-4x4 SIMD data path-Vector operands directly from the data store (no register file transfer)
Scalar / vector interface Loosely coupled processor-scalar supplies command FIFO Instruction and data streaming managed by HW (latency hiding)
-Vector command issued when command and data are in operation

Inheritance from VP1 ・ Inner product-oriented data path ・ VP2 calculation from 2 taps to 4 taps ・ VLIW with A / I / B / V instructions (changed issuance rules)
・ Data path architecture: Accumulator, data transfer, HW interlock ・ Free replacement, change to 16 × 16-> 4 × 4 ・ Dedicated address unit ・ Complex instruction set (optimized for video)

Feature 1 of VP2
-Separate scalar processor-Optimal target for compilable external loop code in C-Instruction streaming-HW-managed instruction prefetch (uses pipeline I $)
・ Application can have a larger footprint than instruction cache ・ Data prefetch ・ HW management stream in data store ・ No need for external loop control ・ Improved accuracy ・ 10b single precision ・ New 20b double precision ・ More Excellent ISA
-Better error management-Memory protection, instruction trap

Feature 2 of VP2
-4x4 SIMD data path-VP1 was a 16x1 SIMD machine-Very efficient organization for high-performance code (H.264, WMV9)
-Reduced bandwidth required for 2D processing-Improved database organization-Tile-type 4x4 structure that fits the data path and provides a much cheaper transposition function compared to VP1-SPA that emulates multi-port RAM Collector structure-Vector pipe operates directly from the data store (no vector register file)
-Dedicated crossbar stage-Extract unaligned operands-Create multiple operands from just a few read ports-Constant RAM
・ Reduces data store read bandwidth pressure ・ Supports very flexible condition codes / predicates

Applicable application codec mpeg2 / wmv9 / H. H.264 encoding assist (in-loop decoder)
Mpeg2 / wmv9 / H. H.264 decoding (post-VLD decoding)
-In-loop / out-of-loop deblocking filter-Image processing / enhancement-High performance motion adaptive progressive scan conversion-Encoding input noise filtering-Multiphase scaling / resampling-Sub-picture synthesis-Pixel-by-pixel conversion: procamp, color Spatial correction, gamma, LCD overdrive, histogram adjustment, etc.Video surface format conversion

Potential target applications • 2D primitive, blitz, rotation, etc. • Fine-tuned software motion estimation algorithm • 16 / 32-bit MAC application (audio?)

  Programming model

1 is a schematic diagram illustrating basic components of a computer system according to an embodiment of the present invention. FIG. 3 is a diagram showing internal components of a video processor unit according to an embodiment of the present invention. FIG. 3 illustrates an exemplary software program for a video processor according to an embodiment of the present invention. FIG. 7 is a diagram illustrating an example of mixing a sub-picture with video using a video processor according to an embodiment of the present invention. FIG. 6 is a diagram illustrating internal components of vector execution according to an embodiment of the present invention. FIG. 3 is a diagram showing a layout of a data store memory having a symmetrical arrangement of tiles according to an embodiment of the present invention.

Claims (16)

A scalar execution unit configured to perform scalar video processing operations;
A vector execution unit configured to perform vector video processing operations;
A data store memory for storing data of the vector execution unit;
A memory interface for performing communication between the scalar execution unit, the vector execution unit, and the data store memory;
With
The data store memory comprises a plurality of tiles having a symmetric bank data structure arranged in an array,
The bank data structure is configured to support access to different tiles in each bank;
A first stream including a first sequential access to the tile and a second stream including a second sequential access to the tile to the vector execution unit or the scalar execution unit; Carried out,
Wherein the memory interface, in order to compensate for the latency of the first stream and the second stream, and based on the said waiting time of the first and second streams, adjust the number of tiles to prefetch Te, it starts to prefetch data in the tile for the first sequential access and the second sequential access of,
system.   The system of claim 1, wherein the system is a multi-dimensional data path processing system for a video processor for performing video processing operations. A system for multidimensional data path processing that supports video processing operations,
With the motherboard,
A host CPU coupled to the motherboard;
The video processor having the system of claim 1 coupled to the motherboard and coupled to the CPU;
A system comprising: Each of the bank data structures comprises a plurality of tiles arranged in a 4x4, 8x8, 8x16, or 16x24 pattern;
The bank data structure is configured to support access to different tiles of each bank data structure and includes at least a row of tiles of the two adjacent bank data structures to two adjacent bank data structures Configured to support a single access,
The tiles are configured to support access to different tiles of each bank data structure, and at least one access of the tiles of the two adjacent bank data structures to two adjacent bank data structures. Contains columns,
The system further comprises a crossbar coupled to the data store memory and selecting a configuration for accessing the plurality of bank data structure tiles;
The crossbar accesses the tiles of the plurality of bank data structures to supply data to the vector data path every clock;
The system further includes a collector for receiving data in the tiles of the plurality of bank data structures accessed by the crossbar and supplying the data in the tiles to the front end of the vector data path every clock. The system as described in any one of Claims 1-3 provided. A video processor for performing video processing operations,
A host interface for performing communication between the video processor and a host CPU;
A memory interface for performing communication between the video processor and a frame buffer memory;
A scalar execution unit coupled to the host interface and the memory interface and configured to perform scalar video processing operations;
A vector execution unit coupled to the host interface and the memory interface and configured to perform vector video processing operations;
With
The frame buffer memory comprises a plurality of tiles;
A first stream including a first sequential access to the tile and a second stream including a second sequential access to the tile to the vector execution unit or the scalar execution unit; Carried out,
Wherein the memory interface, in order to compensate for the latency of the first stream and the second stream, and based on the said waiting time of the first and second streams, adjust the number of tiles to prefetch Te, it starts to prefetch data in the tile for the first sequential access and the second sequential access of,
Video processor. A system for performing video processing operations,
With the motherboard,
A host CPU coupled to the motherboard;
The video processor of claim 5 coupled to the motherboard and coupled to the CPU;
A system comprising: The scalar execution unit functions as a controller of the video processor and controls the operation of the vector execution unit;
The video processor further comprises a vector interface unit that interfaces the scalar execution unit and the vector execution unit;
The video processor of claim 5, wherein the scalar execution unit and the vector execution unit are configured to operate asynchronously. The scalar execution unit executes at a first clock frequency, and the vector execution unit executes at a second clock frequency;
The scalar execution unit is configured to execute an application flow control algorithm, and the vector execution unit is configured to execute a pixel processing operation of the application;
The vector execution unit is configured to operate on a demand driven basis under the control of the scalar execution unit;
The scalar execution unit is configured to send a function call to the vector execution unit using a command FIFO such that the vector execution unit operates on a request driven basis by accessing the command FIFO. And
The video processor of claim 7 or the system of claim 6, wherein the asynchronous operation of the video processor is configured to support separate independent updates of the application's vector subroutine or scalar subroutine.   The video processor of claim 5, wherein the scalar execution unit is configured to operate using VLIW (very long instruction word) code. A stream-based memory access system for a video processor that performs video processing operations, comprising:
A scalar execution unit configured to perform scalar video processing operations;
A vector execution unit configured to perform vector video processing operations;
A frame buffer memory for storing data for the scalar execution unit and the vector execution unit;
A memory interface for performing communication between the scalar execution unit, the vector execution unit, and the frame buffer memory;
With
The frame buffer memory comprises a plurality of tiles;
A first stream including a first sequential access to the tile and a second stream including a second sequential access to the tile to the vector execution unit or the scalar execution unit; Carried out,
Wherein the memory interface, in order to compensate for the latency of the first stream and the second stream, and based on the said waiting time of the first and second streams, adjust the number of tiles to prefetch Te, it starts to prefetch data in the tile for the first sequential access and the second sequential access of,
system. A system that performs stream-based memory access to support video processing operations,
With the motherboard,
A host CPU coupled to the motherboard;
A video processor coupled to the motherboard and coupled to the CPU,
A host interface establishing communication between the video processor and the host CPU;
A scalar execution unit coupled to the host interface and configured to perform scalar video processing operations;
A vector execution unit coupled to the host interface and configured to perform vector video processing operations;
A memory interface coupled to the scalar execution unit and the vector execution unit to establish stream-based communication between the scalar execution unit, the vector execution unit, and a frame buffer memory;
The video processor comprising:
With
The frame buffer memory comprises a plurality of tiles;
A first stream including a first sequential access to the tile and a second stream including a second sequential access to the tile to the vector execution unit or the scalar execution unit; Carried out,
Wherein the memory interface, in order to compensate for the latency of the first stream and the second stream, and based on the said waiting time of the first and second streams, adjust the number of tiles to prefetch Te, it starts to prefetch data in the tile for the first sequential access and the second sequential access of,
system. The first stream and the second stream include data in at least one prefetched tile;
The first stream originates from a first location of the frame buffer memory and the second stream originates from a second location of the frame buffer memory;
The memory interface is configured to manage a plurality of streams including streams from a plurality of different origin locations and streams to a plurality of different end locations;
At least one of the source locations or at least one of the end locations is in system memory;
The system is embedded in the memory interface, performs a plurality of memory reads to support the first stream and the second stream, and supports the first stream and the second stream A DMA engine configured to perform a plurality of memory writes to
The memory interface prefetches data in an adjustable number of tiles of the first stream or the second stream to compensate for latency of the first stream or the second stream; The system of claim 10, wherein the system is configured. A host interface for performing communication between the video processor and the host CPU;
A scalar execution unit coupled to the host interface and configured to perform scalar video processing operations;
A vector execution unit coupled to the host interface and configured to perform vector video processing operations;
A command FIFO that allows the vector execution unit to operate on a request driven basis by accessing the command FIFO;
A memory interface for performing communication between the video processor and a frame buffer memory;
A DMA engine embedded in the memory interface for performing DMA transfers between a plurality of different storage areas and loading the vector execution unit data and instructions into a data store memory and instruction cache;
With
The frame buffer memory comprises a plurality of tiles;
A first stream including a first sequential access to the tile and a second stream including a second sequential access to the tile to the vector execution unit or the scalar execution unit; Carried out,
Wherein the memory interface, in order to compensate for the latency of the first stream and the second stream, and based on the said waiting time of the first and second streams, adjust the number of tiles to prefetch Te, it starts to prefetch data in the tile for the first sequential access and the second sequential access of,
system.   The system of claim 13, wherein the system is a tolerant system that performs video processing operations. With the motherboard,
A host CPU coupled to the motherboard;
A video processor coupled to the motherboard and coupled to the CPU;
15. The system of claim 14, further comprising: The vector execution unit is configured to operate asynchronously with respect to the scalar execution unit by accessing the command FIFO to operate on the request driven basis;
The request driven base is configured to conceal the latency of data transfer from the different storage to the command FIFO of the vector execution unit;
The scalar execution unit is configured to perform algorithmic flow control processing, and the vector execution unit is configured to perform most of the video processing workload;
The scalar execution unit is configured to precalculate the working parameters of the vector execution unit to conceal data transfer latency;
The vector execution unit is configured to schedule memory reads via the DMA engine to prefetch commands for subsequent execution of vector subroutines;
The memory read is scheduled to prefetch commands for the execution of the vector subroutine prior to invoking the vector subroutine by the scalar execution unit;
The vector execution unit is configured to schedule a memory read via the DMA engine and prefetch commands for subsequent execution of a vector subroutine, wherein the memory read is performed by the scalar execution unit. Scheduled to prefetch commands for the execution of the vector subroutine prior to calling the vector subroutine;
The system according to any one of claims 13 to 15.

Images