JP3589565B2 - Video and audio processing device - Google Patents

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention belongs to the technical field of digital signal processing, and relates to an image processing apparatus that performs expansion of compressed video and audio data, compression of video and audio data, and graphics processing.
[0002]
[Prior art]
In recent years, combined with the establishment of digital video data compression / decompression technology and the improvement of LSI technology, a decoder for expanding compressed video and audio data, an encoder for compressing video and audio data, and a graphic Various types of video / audio processing devices such as graphics processing for performing image processing are regarded as important.
[0003]
As a first prior art, there is a video / audio decoder (JP-A-8-116429) for expanding compressed video and audio data of the Moving Picture Experts Group (MPEG) standard. This video / audio decoder performs both video decoding and audio decoding using one signal processing unit.
FIG. 1 is an explanatory diagram of a decoding process by the video / audio decoder. In the figure, the vertical axis represents time, and the horizontal axis represents the amount of calculation.
[0004]
When viewed along the vertical axis, video decoding and audio decoding are alternately performed. This is for decoding both video and audio with common hardware. As shown in the figure, video decoding is divided into sequential processing and block processing. The sequential processing is processing that requires a wide variety of condition determinations, such as decoding other than blocks, that is, analysis of the header of an MPEG stream, and the amount of calculation is small. The block decoding is a process of decoding a variable-length code of an MPEG stream and performing inverse quantization and inverse DCT (discrete cosine transform) on a block-by-block basis, and the amount of calculation is large. As shown in the figure, audio decoding is also divided into sequential processing similar to the above, which requires a wide variety of condition judgments, and decoding processing of the audio data itself. The decoding process of the audio data itself requires higher accuracy than the image data and must be processed within a limited time. Therefore, it is necessary to perform the processing with high accuracy and high speed, and the amount of calculation is large.
[0005]
As described above, the first conventional technology enables one-chip integration and realizes efficient audio-video decoding with hardware as small as one chip.
As a second conventional technique, there is a decoder having a two-chip configuration. One chip is used as a video decoder, and the other chip is used as an audio decoder. FIG. 2 is an explanatory diagram of a decoding process by a two-chip decoder. Both the video decoder and the audio decoder perform sequential processing including a large number of condition determinations such as header analysis, and block decoding processing mainly for decoding the data body. Since both the video decoder and the audio decoder process independently, the performance of each chip may be lower than in the first prior art.
[0006]
[Problems to be solved by the invention]
However, according to the above prior art, there are the following problems.
According to the first prior art, the signal processing unit must decode both video and audio, so that high processing performance is required. That is, it is necessary to operate using a high-speed clock of 100 MHz or more, and there is a problem that the cost is high as a semiconductor for consumer use. In order to increase the processing capacity without using a high-speed clock, it is conceivable to use a VLIW (Very Long Instruction Word) processor or the like. However, the cost of the VLIW processor itself is high and separate sequential processing is performed. Unless a processor is used, there is a problem that the entire processing becomes inefficient.
[0007]
According to the second conventional technique, there is a problem that the cost is high because two processors are used. In other words, neither a video processor nor an audio processor can use a general-purpose inexpensive processor with low processing capability. This is because a video processor is required to be capable of processing a large amount of image data in real time. Also, although the processor for audio does not require as much computational complexity as the processor for video, the audio data is required to have higher accuracy than the image data. Therefore, an inexpensive or low-performance processor does not satisfy the required processing capability for both video and audio.
[0008]
Further, when the video / audio processing device is used in an AV decoder used in a digital (satellite) broadcast tuner (called STB (Set Top Box)) or a DVD (Digital Versatile / Video Disc) reproducing device, An MPEG stream received from a broadcast wave or read from a disc is input, the MPEG stream is decoded, and a series of video signals and audio signals are finally output to a display, a speaker, and the like. Would be enormous. Recently, there has been an increasing demand for a video and audio processing device that efficiently executes such a series of enormous processes.
[0009]
The present invention performs a series of processing of inputting, decoding, and outputting stream data representing compressed image and compressed audio data, has high processing capability without operating at high frequency, and can reduce manufacturing cost. It is an object to provide a video and audio processing device.
It is another object of the present invention to provide a video / audio processing apparatus which realizes decoding of compressed video data, encoding of video data, and graphics processing at low cost.
[0010]
[Means for Solving the Problems]
In order to solve the above problems, the video and audio processing apparatus of the present invention is an apparatus that externally inputs and decodes a data stream including compressed audio data and compressed video data, and outputs the decoded data to an output device. Input / output processing means for performing input / output processing asynchronously generated by an external factor; and decoding processing means for performing decoding processing mainly for decoding a data stream stored in a memory in parallel with the input / output processing. The video data decoded by the decoding processing means and the decoded audio data are stored in a memory, and the input / output processing receives the data stream asynchronously input from the outside, and further stores the data stream in a memory. Supplying the data stream stored in the memory to the decoding processing means, and providing an external display device and audio output. Device read from the memory in accordance with the respective output rates, and is configured to perform and outputting them as input and output processing.
[0011]
According to this configuration, in addition to the input / output processing means and the decoding processing means operating in parallel in a pipeline, the asynchronous processing and the decoding processing are shared between the input / output processing means and the decoding processing means. The processing means can be released from the processing that occurs asynchronously and can exclusively use the decoding processing. As a result, the video and audio processing apparatus efficiently executes a series of processing of stream data input, decode, and output, thereby enabling full decoding of stream data (without dropping frames) without using a high-speed operation clock. ing.
[0012]
BEST MODE FOR CARRYING OUT THE INVENTION
Embodiments of the video and audio processing apparatus of the present invention will be described in the following sections.
1. First Embodiment
1.1 Schematic configuration of video and audio processing device
1.1.1 Input / output processing unit
1.1.2 Decoding processing unit
1.1.2.1 sequential processing unit
1.1.2.2 Standard processing unit
1.2 Configuration of video and audio processing device
1.2.1 Configuration of input / output processing unit
1.2.2 Decoding processing unit
1.2.2.1 sequential processing unit
1.2.2.2 Standard processing unit
1.3 Detailed configuration of each part
1.3.1 Processor 7 (sequential processing unit)
1.3.2 Standard processing unit
1.3.2.1 Code conversion unit
1.3.2.2 Pixel operation unit
1.3.2.3 Pixel read / write unit
1.3.3 Input / output processing unit
1.3.3.1 IO processor
1.3.3.1.1 Instruction reading circuit
1.3.2.1.2 Task management unit
1.4 Description of operation
2 Second embodiment
2.1 Configuration of video and audio processing device
2.1.1 Pixel operation unit
<1. First Embodiment>
The video / audio processing device according to the present embodiment is provided in a satellite broadcast receiving device (STB: Set Top Box), a DVD (Digital Versatile Disc) reproducing device, a DVD-RAM recording / reproducing device, and the like, and serves as compressed video / audio data. An MPEG stream from a satellite broadcast or a DVD is input, decompressed (hereinafter, simply referred to as decoding), and a video signal and an audio signal are output to an external output device.
<1.1 Schematic Configuration of Video / Audio Processing Device>
FIG. 3 is a block diagram illustrating a schematic configuration of the video and audio processing device according to the first embodiment of the present invention.
[0013]
The video / audio processing apparatus 1000 includes an input / output processing unit 1001, a decoding processing unit 1002, and a memory controller 6, and is configured to perform input / output processing and decoding processing separately and in parallel. The external memory 3 is used as a working memory for temporarily storing an MPEG stream and audio data after decoding, and a frame memory for storing video data after decoding.
<1.1.1 Input / output processing unit>
The input / output processing unit 1001 performs input / output processing that occurs asynchronously with the internal operation of the video / audio processing apparatus 1000. This input / output processing includes (a) inputting an MPEG stream that is asynchronously input from the outside and temporarily storing it in the external memory 3, and (b) decoding the MPEG stream stored in the external memory 3 into a decoding processing unit 1002. And (c) reading the decoded video data and audio data from the external memory 3 and outputting them in accordance with the output rates of the external display device and audio output device (not shown).
<1.1.2 Decoding processing unit>
The decoding processing unit 1002 decodes the MPEG stream supplied by the input / output processing unit 1001 and stores the decoded video data and audio data in the external memory 3 in parallel with the operation of the input / output processing unit 1001. I do. The decoding processing of the MPEG stream requires a large amount of calculation and a wide variety of processing contents. It is configured to execute a large amount of operations mainly and a fixed process suitable for parallel operations separately and in parallel. Here, the sequential processing is, for example, header analysis of an MPEG stream, and includes a number of conditions such as header detection and header content determination. In addition, since the routine processing needs to perform various operations on a block unit including a predetermined number of pixels, it is suitable for parallel processing in a pipeline manner, and performs exactly the same operation on different data (pixels). Suitable for parallel processing such as vector operation.
<1.1.2.1 Sequential processing unit>
The sequential processing unit 1003 performs the header analysis of the compressed audio data and the compressed video data supplied from the input / output processing unit 1001, the control of activating the standard processing unit 1004 for each macroblock, and the decoding processing of the compressed audio data. This is performed as a sequential process. The header analysis includes analysis of a macroblock header in an MPEG stream and decoding of a motion vector. Here, the block represents an image composed of 8 * 8 pixels. The macro block includes four luminance blocks and two color difference blocks. The motion vector is a vector indicating a rectangular area of 8 * 8 pixels in the reference frame, and indicates a difference between the block and the rectangular area in the reference frame.
<1.1.2.2 Standard processing unit>
The routine processing unit 1004 receives the decoding start instruction for each macroblock from the sequential processing unit 1003, and performs the macroblock decoding processing as the above-described routine processing in parallel with the audio decoding processing of the sequential processing unit 1003. This decoding process includes decoding of a variable length code (VLD: Variable Length Code Decoding), inverse quantization (IQ: Inverse Quantization), inverse discrete cosine transform (IDCT: Inverse Discrete Cosine Transform), and motion compensation (MC: Motion Conversion). Are performed in the same order. In the motion compensation, the routine processing unit 1004 stores the decoded block in the external memory 3 as a frame memory via the memory controller 6.
<1.2 Configuration of Video / Audio Processing Device>
FIG. 4 is a block diagram showing a more detailed configuration of the video and audio processing device 1000.
<1.2.1 Configuration of input / output processing unit>
In the figure, an input / output processing unit 1001 includes a stream input unit 1, a buffer memory 2, an input / output processor 5 (hereinafter abbreviated as an IO processor 5), a DMAC (Direct Memory Access Controller) 5a, a video output unit 12, and an audio output unit 13. , A host I / F unit 14.
[0014]
The stream input unit 1 converts an externally serially input MPEG data stream into parallel data (hereinafter, referred to as MPEG data). At this time, the stream input unit 1 detects a start code of a GOP (Group of Picture: an MPEG data stream including one I picture and corresponding to a moving image for about 0.5 seconds) from the MPEG data stream, and to that effect. Is notified to the IO processor 5. According to this notification, the converted MPEG data is transferred to the buffer memory 2 under the control of the IO processor 5.
[0015]
The buffer memory 2 is a buffer memory that temporarily holds the MPEG data transferred from the stream input unit 1. The MPEG data held in the buffer memory 2 is further transferred to the external memory 3 via the memory controller 6 under the control of the input / output processor 5.
The external memory 3 includes an SDRAM (Synchronous Dynamic Random Access Memory) chip, and temporarily holds MPEG data transferred from the buffer memory 2 via the memory controller 6. Further, the external memory 3 also holds decoded video data (hereinafter also referred to as frame data) and decoded audio data.
[0016]
The input / output processor 5 controls data input / output between the stream input unit 1, the buffer memory 2, the external memory 3 (with the memory controller 6 interposed), and the FIFO memory 4. That is, it controls the data transfer (DMA transfer) of the paths shown in the following (1) to (4).
(1) Stream input unit 1 → buffer memory 2 → memory controller 6 → external memory 3
(2) External memory 3 → memory controller 6 → FIFO memory 4
(3) External memory 3 → memory controller 6 → buffer memory 2 → video output unit 12
(4) External memory 3 → memory controller 6 → buffer memory 2 → audio output unit 13
In these paths, the input / output processor 5 controls the transfer of video data and audio data in the MPEG data independently. (1) and (2) are transfer paths of MPEG data before decoding. In the transfer paths (1) and (2), the input / output processor 5 transfers the compressed video data and the compressed audio data separately. (3) and (4) are transfer paths of the decoded video and audio data, respectively. The decoded video and audio data are transferred according to the respective output rates of an external display device (not shown) and an audio output device (not shown).
[0017]
The DMAC 5a includes a DMA transfer between the stream input unit 1, the video output unit 12, the audio output unit 13 and the buffer memory 2, a DMA transfer between the buffer memory 2 and the external memory 3, and a DMA transfer between the external memory 3 and the FIFO memory 4. DMA transfer is executed under the control of the IO processor 5.
The video output unit 12 issues a data request to the input / output processor 5 in accordance with the output rate (for example, the cycle of the horizontal synchronizing signal Hsync) of an external display device (CRT or the like), and the input / output processor 5 transfers the above (3). The video data input through the path is output to the display device.
[0018]
The audio output unit 13 issues a data request to the input / output processor 5 in accordance with the output rate of the external audio output device, and converts the audio data input by the input / output processor 5 through the transfer path (4) into the audio output device ( D / A converter, audio amplifier, combination of speakers, etc.).
The host I / F unit 14 is an interface for performing communication with an external host processor, for example, a processor that performs overall control of a DVD playback device. In this communication, instructions such as decoding start, stop, fast forward playback, and reverse playback of the MPEG stream are sent from the host processor.
<1.2.2 Decoding processing unit>
4 includes a FIFO memory 4, a sequential processing unit 1003, and a standard processing unit 1004, and decodes MPEG data supplied from the input / output processing unit 1001 via the FIFO memory 4. Further, the sequential processing unit 1003 includes a processor 7 and an internal memory 8. The routine processing unit 1004 includes a code conversion unit 9, a pixel operation unit 10, a pixel read / write unit 11, a buffer 200, and a buffer 201.
[0019]
The FIFO memory 4 includes two FIFOs (hereinafter, referred to as a video FIFO and an audio FIFO), and stores compressed video data and compressed audio data transferred from the external memory 3 under the control of the input / output processor 5 in a first-in first-out manner. I do.
<1.2.2.1 Sequential processing unit>
The processor 7 controls reading of the compressed video data and the compressed audio data from the FIFO memory 4 and performs a partial decoding process on the compressed video data and a full decoding process on the compressed audio data. The decoding of a part of the compressed video data includes analysis of header information in the MPEG data, calculation of a motion vector, and control of the compressed video decoding process. This is because the entire decoding process of the compressed video data is shared between the processor 7 and the standard processing unit 1004. That is, the processor 7 shares sequential processing that requires a wide variety of condition judgments, and the routine processing unit 1004 shares a large amount of routine arithmetic processing. On the other hand, the processor 7 is in charge of the audio decoding since the amount of calculation is smaller than that of the video decoding.
[0020]
The function of the processor 7 will be specifically described with reference to FIG. FIG. 5 shows the operation timing of each section of the video and audio processing apparatus together with the MPEG stream in a hierarchical manner. In the figure, the horizontal axis is the time axis. The first layer shows the flow of the MPEG stream. An MPEG stream for one second like the second layer includes a plurality of frames (I, P, and B pictures). As in the third layer, one frame includes a picture header and a plurality of slices. As in the fourth layer, one slice includes a slice header and a plurality of macroblocks. As in the fifth layer, one macroblock includes a macroblock header and six blocks.
[0021]
The data structure of the first to fifth layers shown in the figure is described in detail in a known document, for example, ASCII "Point Illustrated Latest MPEG Textbook".
The processor 7 analyzes the header up to the macroblock layer in the MPEG stream and decodes the compressed audio data as shown in the fifth and lower layers in FIG. At this time, the processor 7 instructs the code conversion unit 9, the pixel operation unit 10, and the pixel read / write unit 11 to start decoding the macro block according to the header analysis result in the macro block unit, and the code conversion unit 9, the pixel operation While the macroblock is being decoded by the unit 10 and the pixel read / write unit 11, the compressed audio data is read from the FIFO memory 4 and decoded. When the decoding of the macroblock is completed by the code conversion unit 9, the pixel operation unit 10, and the pixel read / write unit 11, the processor 7 receives the notification to that effect by an interrupt signal, suspends the decoding of the compressed audio data, and Start parsing the macroblock header.
[0022]
The internal memory 8 is a work memory of the processor 7, and temporarily stores the decoded audio data. The held audio data is transferred to the external memory 3 by the input / output processor 5 through the path (4).
<1.2.2.2 Routine processing unit>
The code converter 9 performs variable length decoding (VLD) on the compressed video data read from the FIFO memory 4. As shown in FIG. 5, the code conversion unit 9 transfers the header information and the information on the motion vector (broken line section in the figure) among the decoded data to the processor 7, and the macro block (the luminance blocks Y0 to Y3 and Data (six blocks composed of the color difference blocks Cb and Cr) (solid line sections in the drawing) are transferred to the pixel operation unit 10 via the buffer 200. The macroblock data decoded by the code conversion unit 9 is data representing a spatial frequency component.
[0023]
The buffer 200 holds data representing a spatial frequency component for one block (8 × 8 pixels) written by the code conversion unit 9.
The pixel operation unit 10 performs an inverse quantization process (IQ) and an inverse discrete cosine transform (IDCT) on the block data transferred from the code conversion unit 9 via the buffer 200 in block units. The processing result by the pixel operation unit 10 is data representing a luminance value of a pixel or its difference in the case of a luminance block, and data representing a color difference or its difference in the case of a chrominance block. The data is transferred to the read / write unit 11.
[0024]
The buffer 201 holds one block (8 × 8 pixels) of pixel data.
The pixel read / write unit 11 performs motion compensation on the processing result of the pixel operation unit 10 in block units. That is, for the P picture and the B picture, a rectangular area indicated by the motion vector is cut out from the decoded reference frame in the external memory 3 via the memory controller 6 and synthesized with the block of the processing result of the pixel operation unit 10. To decode the original block image. The decoding result by the pixel read / write unit 11 is stored in the external memory 3 via the memory controller 6.
[0025]
Since the contents of the motion compensation, IQ, and IDCT are well-known technologies, detailed descriptions thereof are omitted (see the above-mentioned document).
<1.3 Detailed configuration of each part>
Next, a detailed configuration of main components of the video and audio processing device 1000 will be described.
<1.3.1 Processor 7 (sequential processing unit)>
FIG. 6 is a diagram showing the analysis of the macroblock header by the processor 7 and the contents of control to other units. First, the respective data in the macroblock header indicated by the abbreviations in the figure are described in the above-mentioned documents and the like, and thus the description is omitted here.
[0026]
As shown in the figure, the processor 7 issues a command to the code conversion unit 9 to sequentially obtain data of the header portion subjected to variable length decoding, and according to the contents thereof, the code conversion unit 9, the pixel operation unit 10, the pixel read / write unit 11 , The data necessary for decoding the macroblock is set.
Specifically, first, the processor 7 issues a command for obtaining a macro block address increment (MBAI) to the code conversion unit 9 (S101), and obtains the MBAI from the code conversion unit 9. If the macroblock data is a skipped macroblock based on the MBAI (if the macroblock to be decoded is the same as the previous macroblock), the macroblock data has been omitted, so the process proceeds to S117 and must be a skipped macroblock. If so, the header analysis is continued (S102, 103).
[0027]
Next, the processor 7 issues a command for obtaining an MBT (Macro Block Type), and obtains the MBT from the code conversion unit 9. From the MBT, it is determined whether the scan type of the block is a zigzag scan or an alternate scan, and the reading order of the buffer 200 is instructed to the pixel operation unit 10 (S104).
Further, the processor 7 determines whether or not an STWC (Spatial Temporal Weight Code) exists from the already acquired header data (S105), and issues and acquires a command if it exists (S106).
[0028]
In the same manner, the processor 7 obtains FrMT (Frame Motion Type), FiMT (Field Motion Type), DT (DCT type), QSC (Quantizer Scale Code), MV (Motion Vector), and CB (CBP) to obtain BP (Motion Vector). S107-116). At that time, the processor 7 notifies the pixel read / write unit 11 of the analysis results of FrMT, FiMT, and DT, notifies the pixel operation unit 10 of the analysis result of QSC, and notifies the code conversion unit 9 of the analysis result of CBP. As a result, information necessary for IQ, IDCT, and motion compensation is set in the code conversion unit 9, the pixel operation unit 10, and the pixel read / write unit 11.
[0029]
Further, the two-processor configuration has a redundant configuration because each processor individually performs the above-described sequential processing that requires various condition judgments.
Next, the processor 7 issues a macroblock decoding start instruction to the code converter 9 (S117). Accordingly, the code conversion unit 9 starts VLD for each block in the macroblock, and outputs the result of VLD to the pixel operation unit 10 via the buffer 200. Further, the processor 7 calculates a motion vector based on the MV data (S118), and notifies the pixel read / write unit 11 of the calculation result (S119).
[0030]
In the above processing, regarding the motion vector, a series of processing of obtaining the motion vector data (MV) (S113), calculating the motion vector (S118), and setting the motion vector in the pixel read / write unit 11 (S119) is necessary. It is. In this regard, the processor 7 does not calculate and set the motion vector (S118, 119) immediately after obtaining the motion vector data (MV) (S113, 119), but issues a decode start instruction to the The vector is calculated and set. Thereby, the motion vector calculation and setting process of the processor 7 and the decoding process to the routine processing unit 1004 are processed in parallel. That is, the decoding start timing of the routine processing unit 1004 is advanced.
[0031]
Since the header analysis of the compressed video data for one macroblock is completed as described above, the processor 7 acquires the compressed audio data from the FIFO memory 4 and starts the audio decoding process (S120). The audio decoding process is continued until an interrupt signal indicating that macroblock decoding has been completed is input from the code conversion unit 9. In response to this interrupt signal, the processor 7 starts the header analysis for the next macro block.
<1.3.2 Standard processing unit>
Next, the routine processing unit 1004 performs a decoding process on the six blocks in the macro block by operating the code conversion unit 9, the pixel operation unit 10, and the pixel read / write unit 11 in parallel (in a pipeline). I have. Here, the configurations of the pixel operation unit 10, the pixel read / write unit 11, and the code conversion unit 9 will be described in detail in this order.
<1.3.2.1 Code conversion unit 9>
FIG. 19 is a block diagram showing a configuration of the code conversion unit 9.
[0032]
The code conversion unit 9 shown in the figure includes a VLD unit 901, a counter 902, an incrementer 903, a selector 904, a scan table 905, a scan table 906, a flip-flop (hereinafter abbreviated as FF) 907, and a selector 908, and The (VLD) result is written in the buffer 200 so as to be arranged in a block unit in the order of zigzag scan or alternate scan.
[0033]
The VLD unit 901 performs variable-length decoding (VLD) on the compressed video data read from the FIFO memory 4 and, among the decoded data, header information and motion vector information (broken line sections in FIG. 5). And outputs macroblock data (six blocks consisting of luminance blocks Y0 to Y3 and color difference blocks Cb and Cr) (solid line sections in FIG. 5) to the buffer 200 in block (64 spatial frequency data) units. I do.
[0034]
The circuit portion including the counter 902, the incrementer 903, and the selector 904 repeatedly counts from 0 to 63 in synchronization with the output of the spatial frequency data from the VLD section 901.
The scan table 905 is a table that stores the addresses of the block storage areas of the buffer 200 in the order of zigzag scan. The output values (0 to 63) of the counter 902 are sequentially input, and the addresses are sequentially output. FIG. 20 shows a block storage area for storing 8 × 8 pieces of spatial frequency data in the buffer 200 and a zigzag scan route. The scan table 905 sequentially outputs the pixel addresses in the route shown in FIG.
[0035]
The scan table 906 is a table that stores the addresses of the block storage areas of the buffer 200 in the order of the alternate scan. The output values (0 to 63) of the counter 902 are sequentially input, and the addresses are sequentially output. FIG. 21 shows a block storage area in the buffer 200 for storing 8 × 8 pieces of spatial frequency data and a route of the alternate scan. The scan table 905 sequentially outputs the pixel addresses in the route shown in FIG.
[0036]
The FF 907 holds a flag indicating a scan type (zigzag scan or alternate scan). This flag is set by the processor 7.
The selector 908 selects an address output from the scan table 905 and the scan table 906 according to the flag of the FF 907, and outputs the address to the buffer 200 as a write address.
<1.3.2.2 Pixel operation unit>
FIG. 7 is a block diagram illustrating a configuration of the pixel operation unit 10.
[0037]
As shown in the drawing, the pixel operation unit 10 includes an execution unit 501 including a multiplier 502 and an adder / subtractor 503, a first program counter (hereinafter abbreviated as a first PC) 504, and a second program counter (hereinafter a second PC). 505, a first instruction memory 506, a second instruction memory 507, and a selector 508, and are configured so that IQ and a part of the IDCT can be overlapped and executed in parallel. .
[0038]
The execution unit 501 accesses and operates the buffers 200 and 201 according to micro instructions sequentially output from the first instruction memory 506 and the second instruction memory 507.
The first instruction memory 506 and the second instruction memory 507 are control memories for storing microprograms for realizing IQ and IDCT for blocks (frequency components) held in the buffer 200. FIG. 8 shows an example of the microprogram stored in the first instruction memory 506 and the second instruction memory 507.
[0039]
In the figure, a first instruction memory 506 stores an IDCT1A microprogram and an IQ microprogram, and the first PC 504 specifies a read address. The IQ microprogram is an arithmetic processing mainly including reading of the buffer 200 and multiplication, and does not use the adder / subtractor 503.
The second instruction memory 507 stores the IDCT1B microprogram and the IDCT2 microprogram, and the read address is specified by the first PC 504 or the second PC 505 via the selector 508. Here, IDCT1 means the first half of processing of the IDCT mainly including multiplication and addition / subtraction, and is executed using the entire execution unit 501 by reading the IDCT1A microprogram and the IDCT1B microprogram at the same time. Also, IDCT2 means a process of the latter half of the IDCT mainly including addition and subtraction and a process of writing to the buffer 201, and is executed using the adder / subtractor 503 by reading the IDCT2 microprogram in the second instruction memory 507. .
[0040]
Since IQ is processed by the multiplier 502 and IDCT2 is processed by the adder / subtractor 503, they can be operated in parallel. FIG. 9 shows an operation timing chart of IQ, IDCT1, and IDCT2 by the pixel operation unit 10.
9, when the code conversion unit 9 writes the data of the luminance block Y0 into the buffer 200 (timing t0), the code conversion unit 9 notifies the pixel calculation unit 10 to that effect by the control signal 102. The pixel operation unit 10 reads the IQ microprogram in the first instruction memory 506 in accordance with the address designation of the first PC 504 by using the QS (Quantizer Scale) value set at the time of header analysis of the processor 7, and IQ. At this time, the selector 508 selects the first PC 504 (timing t1).
[0041]
Further, the pixel operation unit 10 performs IDCT1 on the data in the buffer 200 by reading the IDCT1A and IDCT1B microprograms according to the address designation of the first PC 504. At this time, since the selector 508 selects the first PC 504, the address from the first PC 504 is specified in both the first instruction memory 506 and the second instruction memory 507 (timing t2).
[0042]
Next, the pixel operation unit 10 reads the IQ microprogram in the first instruction memory 506 in accordance with the address designation of the first PC 504 by using the QS (Quantizer Scale) value, thereby IQ-processing the data in the block Y1 of the buffer 200. At the same time, the second half of the IDCT processing is performed on the block Y0 by reading the IDCT2 microprogram in the second instruction memory 507 according to the address designation of the second PC 505. At this time, the selector 508 selects the second PC 505. The first PC 504 and the second PC 505 specify addresses independently (timing t3).
[0043]
Thereafter, similarly, the pixel operation unit 10 continues the processing in block units (after timing t4).
<1.3.2.3 Pixel read / write unit>
FIG. 10 is a block diagram illustrating a detailed configuration of the pixel read / write unit 11.
As shown in the figure, the pixel read / write unit 11 includes buffers 71 to 74 (hereinafter, buffers A to D), a half-pel interpolator 75, a synthesizer 76, selectors 77 and 78, and a read / write controller 79. Consists of
[0044]
The read / write control unit 79 performs motion compensation on the block data input via the buffer 201 using the buffers A to D, and transfers the final decoded image to the external memory 3 in units of two blocks. More specifically, the memory controller 6 is controlled so that a rectangular area corresponding to two blocks is read from the reference frame in the external memory 3 according to the motion vector set at the time of the header analysis of the processor 7. As a result, data of a rectangular area for two blocks indicated by the motion vector is stored in the buffer A or the buffer B. After that, the combining unit 76 performs half-pel interpolation of a rectangular area of two blocks according to the type of picture (I, P, or B picture). Further, by combining (adding) the block data input via the buffer 201 and the rectangular area after the half-pel interpolation, the pixel value of the block is calculated and stored in the buffer B. The final decoded block stored in the buffer B is transferred to the external memory 3 via the memory controller 6.
<1.3.3 Input / output processing unit>
The input / output processing unit 1001 switches a plurality of tasks sharing various data transfers without overhead in order to execute a large number of data inputs / outputs (data transfers) as described above, and furthermore, responds to data input / output requests. It is configured not to cause a delay. The overhead referred to here is the saving and restoring of the context that occurs at the time of task switching. That is, the input / output processor 5 is configured to eliminate the overhead caused by saving and restoring the instruction address and the register data of the program counter in the memory (stack area). Here, the detailed configuration will be described.
<1.3.3.1 IO processor>
FIG. 11 is a block diagram illustrating a configuration of the IO processor 5. In the figure, the IO processor 5 includes a state monitoring register 51, an instruction memory 52, an instruction reading circuit 53, an instruction register 54, a decoder 55, an operation execution unit 56, a general-purpose register set group 57, and a task management unit 58, and asynchronously. In order to cope with a plurality of events that occur, the system is configured to execute the task while switching the task at an extremely short cycle (for example, four instruction cycles).
[0045]
The status monitoring register 51 includes registers CR1 to CR3, and holds various status data (such as flags) for the IO processor 5 to monitor various input / output statuses. For example, the state monitoring register 51 includes a state of the stream input unit 1 (start code detection flag in the MPEG stream), a state of the video output unit 12 (a flag indicating a horizontal blanking period, a frame data transfer completion flag), and an audio output unit. 13 (transfer completion flag of audio frame data) and the state of data transfer between them and the buffer memory 2, external memory 3 and FIFO memory 4 (number of data transfer, data request flag to FIFO memory 4) Holds state data indicating such as
[0046]
More specifically, it includes the following flags and the like.
-Start code detection flag (hereinafter also referred to as flag 1)
This flag is set when the stream input unit 1 detects a start code in an MPEG stream.
-Horizontal blanking flag (flag 2)
This flag indicates a horizontal blanking period, and is set by the video output unit 12. It is set at a period of about 60 microseconds.
• Transfer completion flag of video frame data (flag 3)
This flag is set by the DMAC 5a when one frame of decoded image data is transferred from the external memory 3 to the video output unit 12.
• Transfer completion flag of audio frame data (flag 4)
This flag is set by the DMAC 5a when one frame of decoded audio data is transferred from the external memory 3 to the audio output unit 13.
• Data transfer completion flag (flag 5)
This flag is set when the number of compressed image data designated by the IO processor 5 from the stream input unit 1 to the buffer memory 2 is DMA-transferred by the DMAC 5a (when the terminal count is reached).
• DMA request flag (flag 6)
This flag is a flag indicating that there is data to be subjected to DMA transfer of the compressed image data or compressed audio data in the buffer memory 2 to the external memory 3 and is set by the IO processor 5 (from task 1 to task 2 described later). Request).
-Data request flag to video FIFO (flag 7)
This flag is a flag for requesting data transfer from the external memory 3 to the video FIFO in the FIFO memory 4, and is set when the compressed video data of the video FIFO becomes less than a predetermined amount. This flag is set at a period of about 5 to 40 microseconds.
• Data request flag to audio FIFO (flag 8)
This flag is a flag for requesting data transfer from the external memory 3 to the audio FIFO in the FIFO memory 4, and is set when the compressed audio data of the audio FIFO becomes less than a predetermined amount. This flag is set at a period of about 15 to 60 microseconds.
• Decoder communication request flag (flag 9)
This flag is a flag for requesting communication from the decoding processing unit 1002 to the input / output processing unit 1001.
-Host communication request flag (flag 10)
This flag is a flag for requesting communication from the host processor to the input / output processing unit 1001.
[0047]
The above-mentioned flags are constantly monitored, not interrupted, by each task executed by the IO processor 5.
The instruction memory 52 stores a plurality of task programs sharing a large number of data input / output (data transfer) controls. In this embodiment, six task programs of tasks 0 to 5 are stored.
・ Task 0 (Host I / F task)
This task is a task for performing communication with the host computer, that is, communication processing with the host computer via the host I / F unit 14 when the flag 10 is set. For example, start, stop, fast-forward playback, reverse playback, and the like of the decoding of the MPEG stream from the host processor are received, and the decoding status (error or the like) is notified. This process uses the flag 10 as a trigger.
・ Task 1 (purging task)
When a start code is detected by the stream input unit 1 (the flag 1), the task analyzes (parsing) the MPEG data input from the stream input unit 1 and extracts individual elementary streams. This is a program for transferring the extracted elementary stream to the buffer memory 2 by DMA transfer (the first half of the transfer path (1)). The types of elementary streams extracted here include compressed video data (also called video elementary streams), compressed audio data (also called audio elementary streams), and private data. When the elementary stream is stored in the buffer memory 2, the flag 6 is set.
・ Task 2 (stream transfer / audio task)
This task is a program that controls the following transfers (a) to (c).
[0048]
(A) DMA transfer of each elementary stream from the buffer memory 2 to the external memory 3 (the latter half of the transfer path (1)). This transfer is triggered by the flags 1 and 3 described above.
(B) DMA transfer of the compressed audio data from the external memory 3 to the audio FIFO of the FIFO memory 4 according to the data size (remaining amount) of the compressed audio data held in the audio FIFO (in the transfer path (2)) Transfer to audio FIFO). This data transfer is performed when the data size of the compressed audio data held in the audio FIFO becomes smaller than a certain amount. This transfer is triggered by the flag 8 described above.
[0049]
(C) DMA transfer of the decoded audio data from the external memory 3 to the buffer memory 2 and from the buffer memory 2 to the audio output unit 13 (the transfer path (4)). This transfer is triggered by the flag 2 described above.
・ Task 3 (Video supply task)
This task performs DMA transfer of the compressed video data from the external memory 3 to the video FIFO of the FIFO memory 4 according to the data size (remaining amount) of the compressed video data held in the video FIFO (the transfer path (2) described above). (A transfer to the video FIFO). This data transfer is performed when the data size of the compressed video data held in the video FIFO becomes smaller than a certain amount. This transfer is triggered by the flag 7 described above.
・ Task 4 (video output task)
This task is a program for processing the DMA transfer (the transfer path (4)) of decoded video data from the external memory 3 to the buffer memory 2 and from the buffer memory 2 to the video output unit 12. This transfer is triggered by the flag 2 described above.
・ Task 5 (Decoder I / F task)
This task is a program that processes a command from the decode processing unit 1002 to the IO processor 5. The commands include “getAPTS”, “getVPTS”, “getSTC”, and the like. The getVPTS (Video Presentation Time Stamp) is a command by which the decode processing unit 1002 requests the IO processor 5 to acquire the VPTS added to the compressed video data. The getAPTS (Audio Presentation Time Stamp) is a command by which the decoding processing unit 1002 requests the IO processor 5 to acquire the APTS added to the compressed audio data. getSTC (System Time Clock) is a command by which the decoding processing unit 1002 requests the IO processor 5 to acquire an STC. The IO processor 5 receiving these commands notifies the decode processing unit 1002 of the STC, VPTS, and APTS. The STC, VPTS, and APTS are used by the decoding processing unit 1002 to synchronize the decoding of audio and video, and to adjust the degree of decoding in units of frames. This process uses the flag 9 as a trigger.
[0050]
The instruction reading circuit 53 includes a plurality of program counters (hereinafter, abbreviated as PCs) each indicating an instruction fetch address, reads an instruction from the instruction memory 52 using a PC designated by the task management unit 58, and stores the instruction in the instruction register 54. . Specifically, the instruction reading circuit 53 has PCs 0 to 5 corresponding to the above tasks 0 to 5, and when the designation of the PC by the task management unit 58 is changed.
It is configured to switch the PC at high speed by hardware. With this configuration, the IO processor 5 saves the PC value of the current task in the memory at the time of the task switch and is released from the process of restoring the PC value of the next task from the memory.
[0051]
The decoder 55 decodes the instruction read from the instruction memory 52 and stored in the instruction register 54, and controls the operation execution unit 56 to execute the instruction. In addition, the decoder 55 performs a pipeline control of the entire IO processor 5 including at least three stages of an instruction reading stage of the instruction reading circuit 53, a decoding stage of the decoder 55, and an execution stage of the operation execution unit 56.
[0052]
The operation execution unit 56 includes an ALU (Arithmetic Logical Unit), a multiplier, a BS (Barrel Shifter), and the like, and executes an operation specified by the instruction under the control of the decoder 55.
The general-purpose register set group 57 includes six register sets corresponding to tasks 0 to 5 (one register set includes four 32-bit registers and four 16-bit registers). A register set having a total of 24 32-bit registers and 24 16-bit registers and corresponding to the task being executed is used. As a result, the IO processor 5 saves all the current register data in the memory at the time of the task switch and is released from the process of restoring the register data of the next task from the memory.
[0053]
The task management unit 58 performs task switching by switching the PC of the instruction reading circuit 53 and the register set of the general-purpose register set group 57 every predetermined number of instruction cycles. In this embodiment, the predetermined number is four. In addition, since the IO processor 5 processes one instruction in one instruction cycle, the task management unit 58 switches the task every four instructions without generating the overhead. As a result, response delay to various input / output requests generated asynchronously is suppressed. In other words, a response delay to an input / output request results in a maximum of only 24 instruction cycles.
<1.3.1.1.1 Instruction reading circuit>
FIG. 12 is a block diagram showing a detailed configuration example of the instruction reading circuit 53.
[0054]
In the figure, the instruction reading circuit 53 includes a task-specific PC storage unit 53a, a current IFAR (Instruction Fetch Address Register) 53b, an incrementer 53c, a next IFAR 53d, a selector 53e, a selector 53f, and a DECAR (DECode Address Register) 53g. At the time of switching, the instruction read address is switched without overhead.
[0055]
The task-specific PC storage unit 53a has six address registers corresponding to tasks 0 to 5, and holds a program count value for each task. Each program count value is a restart address of the corresponding task. At the time of task switching, under the control of the task management unit 58 and the decoder 55, the program count value is read from the address register corresponding to the task to be executed next, and the program of the address register corresponding to the task currently being executed is read. The count value is updated to a new restart address. At this time, the task to be executed next and the current task are respectively specified by the task management unit 58 by a “nexttaskid (rd addr)” signal (hereinafter also referred to as a task ID) and a “taskid (wr addr)” signal.
[0056]
The program count values corresponding to tasks 0, 1, and 2 are shown in PC0, PC1, and PC2 of FIG. In the figure, (0-0) represents instruction 0 of task 0, and (1-4) represents instruction 4 of task 1. For example, PC0 is read when task 0 is restarted (instruction cycle t0), and is updated to the address of the instruction (0-4) when switching to the next task (instruction cycle t4).
[0057]
The loop circuit including the incrementer 53c, the next IFAR 53d, and the selector 53e is a circuit that updates the instruction read address selected by the selector 53e. The address output from the selector 53e is shown as IF1 in FIG. In the figure, for example, when switching from task 0 to task 1, the selector 53e selects the instruction (1-0) address read from the task-specific PC storage unit 53a in cycle t4, and selects the next address in cycles t5 to t7. Select the incremented instruction address from IFAR 53d.
[0058]
The current IFAR 53b holds the selected output IF1 of the selector 53e with a delay of one cycle, and outputs it to the instruction memory 52 as an instruction read address. In other words, it holds the instruction read address of the currently active task. The instruction read address of the current IFAR 53b is shown in IF2 of FIG. As shown in the figure, IF2 indicates an instruction address of a different task every four instruction cycles.
[0059]
The DECAR 53g holds the address of the instruction held in the instruction register 54. That is, it indicates the instruction being decoded. DEC in FIG. 13 shows the address held in DECAR 53g. EX in FIG. 13 indicates an instruction address being executed.
The selector 53f selects a branch address when a branch instruction is executed or an interrupt occurs, and otherwise selects the address of the next IFAR 53d.
[0060]
With the provision of such an instruction reading circuit 53, the IO processor 5 performs four-stage (IF1, IF2, DEC, EX) pipeline processing as shown in FIG. The IF1 stage is a stage for selecting and updating a plurality of program count values. The IF2 stage is a stage for reading an instruction. <1.3.2.1.2 Task management unit>
FIG. 14 is a block diagram showing a detailed configuration of the task management unit 58. In the figure, the task management unit 58 is broadly divided into a slot manager that manages task switching timing and a scheduler that manages the order of tasks.
[0061]
The slot manager has a counter 58a, a latch 58b, a comparator 58c, and a latch unit 58d, and outputs a task switching signal (chgtaskex) for instructing task switching to the instruction reading circuit 53 every four instruction cycles.
Specifically, the latch 58b is two flip-flop (FF) circuits that hold the lower two bits of the output of the counter 58a. The counter 58a outputs 3 bits obtained by incrementing the 2-bit output value of the latch 58b by +1 every clock indicating an instruction cycle. As a result, the counter 58a repeatedly outputs 1, 2, 3, and 4. The comparator 58c outputs a task switching signal (chgtaskex) to the instruction reading circuit 53 and the scheduler when the output value of the counter 58a matches the constant 4.
[0062]
The scheduler includes a task round management unit 58e, a priority encoder 58f, and a latch 58g. Each time a task switching signal (chgtaskex) is output, the scheduler updates a task id, and outputs a current task id and a task id to be executed next. Is output to the instruction reading circuit 53.
Specifically, both the latch unit 58d and the latch 58g hold the current task id in an encoded format (3 bits). The value of the encoded format represents the task id.
[0063]
When the task switching signal (chgtaskex) is input, the task round management unit 58e refers to the latch unit 58d and outputs the task id to be executed next in a decoded format (6 bits). In the decoded format (6 bits), one bit corresponds to one task, and the bit position represents the task id.
The priority encoder 58f converts the task id output from the task round management unit 58e from a decoded format to an encoded format. Both the latch unit 58d and the latch 58g hold the encoded task id one cycle later.
[0064]
With this configuration, when the task switching signal (chgtaskex) is output from the comparator 58c, the task round management unit 58e sets the id of the task to be executed next from the priority encoder 58f as the “nexttaskid (rd addr)” signal. The current task id is output from the latch 58e as a "taskid (wr addr)" signal.
<1.4 Description of operation>
The operation of the video / audio processing device 1000 according to the first embodiment configured as described above will be described.
[0065]
In the input / output processing unit 1001, an MPEG stream asynchronously input from the stream input unit 1 is temporarily stored in the external memory 3 via the buffer memory 2 and the memory controller 6 under the control of the input / output processor 5, and The data is stored in the FIFO memory 4 via the controller 6. At this time, the IO processor 5 supplies the compressed moving image data and the compressed audio data to the FIFO memory 4 by executing the above tasks 2 (b) and 3 according to the remaining amount. As a result, a fixed amount of compressed moving image data and compressed audio data are supplied to the FIFO memory 4 without excess and deficiency, so that the decoding processing unit 1002 can separate from asynchronous input / output and exclusively use the decoding processing. it can. The processing up to this point is performed by the input / output processing unit 1001 independently and in parallel with the decoding processing unit 1002.
[0066]
On the other hand, in the decoding processing unit 1002, the MPEG stream data held in the FIFO memory 4 is subsequently decoded by the processor 7, the code conversion unit 9, the pixel operation unit 10, and the pixel read / write unit 11. FIG. 15 is an explanatory diagram showing the decoding operation after the FIFO memory 4.
In the figure, the horizontal axis represents the time axis, and the header analysis of approximately one macroblock and the decoding of each block are shown. The vertical direction indicates that the decoding of each block is executed in a pipeline manner in each unit of the decoding processing unit 1002.
[0067]
As shown in the figure, the processor 7 repeats the header analysis of the compressed video data and the decoding process on the compressed audio data in a time-division manner. That is, the processor 7 analyzes the header of one macroblock, notifies the code conversion unit 9, the pixel operation unit 10, and the pixel read / write unit 11 of the analysis result, and then starts decoding the macroblock to the code conversion unit 9. Instruct. Thereafter, the processor 7 performs a decoding process on the compressed audio data until an interrupt signal is notified from the code conversion unit 9. The decoded audio data is temporarily held in the internal memory 8 and further DMA-transferred to the external memory 3 by the memory controller 6.
[0068]
Further, the code conversion section 9 receives a macroblock decoding start instruction from the processor 7 and stores the macroblock in the buffer 200 for each block in the macroblock. At this time, the code conversion unit 9 changes the order of the write address to the buffer 200 according to the scan type of the block notified when the processor 7 analyzes the header. That is, the order of the write addresses is changed between the zigzag scan and the alternate scan. Thus, the pixel operation unit 10 does not need to change the order of the read addresses, and can always read in the same order of the read addresses regardless of the scan type. The code conversion unit 9 repeats the above operation until the six blocks in the macro block have been subjected to the VLD processing, and writes the blocks into the buffer 200. When the VLD of six blocks is completed, an interrupt is generated in the processor 7. This interrupt signal is a macro block decode end signal End Of Macro Block (EOMB). The code conversion unit 9 generates the EOMB by detecting the block end signal End Of Block (EOB) of the sixth block.
[0069]
The pixel operation unit 10 performs IQ and IDCT on the block data stored in the buffer 200 in block units as shown in FIG. 9 in parallel with the code conversion unit 9, and stores the processing result in the buffer 201.
The pixel read / write unit 11, in parallel with the pixel operation unit 10, based on the block data in the buffer 201 and the motion vector notified by the header analysis by the processor 7 as shown in FIG. Of a rectangular area from the block and block synthesis. The result of block synthesis is stored in the external memory 3 via the FIFO memory 4.
[0070]
The above is the operation when the block is not a skip macro block. In the case of a skip macro block, the code conversion unit 9 and the pixel operation unit 10 do not operate, and only the pixel read / write unit 11 operates. If there is a skipped macroblock, the image is the same as the rectangular area in the reference frame, so that the image is copied to the external memory 3 by the pixel read / write unit 11 as a decoded image.
[0071]
In this case, an interrupt signal from the code converter 9 to the processor 7 is generated as follows. That is, a signal indicating that the processor 7 has transmitted a control signal for starting a motion compensation operation to the pixel reading / writing unit 11, a signal indicating that the pixel reading / writing unit 11 can perform the motion compensation operation, and a skip macro block. Is obtained, and an interrupt signal is input to the processor 7 as a logical sum of the logical product and the EOMB signal.
[0072]
As described above, according to the video / audio processing device of the first embodiment of the present invention, the MPEG stream input processing from the storage medium or the communication medium, and the display image data and the audio data to the display device and the audio output device are performed. The input / output processing unit 1001 shares output processing and processing for supplying a stream to the decoding processing unit 1002, and the decoding processing unit 1002 shares decoding processing of compressed video data and compressed audio data. As a result, the decoding processing unit 1002 is released from the processing that occurs asynchronously and can exclusively use the decoding processing. As a result, since a series of processing of inputting, decoding, and outputting the MPEG stream is efficiently executed, full decoding of the MPEG stream (without dropped frames) can be realized without using a high-speed operation clock.
[0073]
Further, it is desirable that the present video / audio processing device be integrated into an LSI on one chip. In this case, the full decoding can be performed with an operation clock of 100 MHz or less (actually, 54 MHz). In this regard, recent high-performance CPUs whose operation clocks exceed 100 MHz or 200 MHz enable the full decoding as long as the image size is small, but on the other hand, the manufacturing cost is high. On the other hand, the present video / audio processing apparatus is superior in terms of manufacturing cost and full decoding.
[0074]
Furthermore, the decoding processing unit 1002 of the present video / audio processing apparatus shares roles as follows.
In other words, the processor 7 is in charge of header analysis that requires a wide variety of condition judgments for both compressed video data and compressed audio data, and is also responsible for decoding audio compressed data. Since a large amount of routine calculation is required for block data of compressed video data, dedicated hardware (firmware) such as a code conversion unit 9, a pixel operation unit 10, and a pixel read / write unit 11 performs decoding processing. In charge of As shown in FIG. 15, the code conversion unit 9, the pixel operation unit 10, and the pixel read / write unit 11 are pipelined. In the pixel operation unit 10, IQ and IDCT can be processed in parallel. The pixel read / write unit 11 implements access to a reference frame in units of two blocks. As a result, the efficiency of the compressed audio decoding process is improved, so that the hardware dedicated to video decoding can obtain high processing performance without using a high-speed clock. More specifically, a processing ability equivalent to or higher than that of the related art was obtained with a clock of about 50 to 60 MHz without using a high-speed clock exceeding 100 MHz. Therefore, it is not necessary to use a high-speed element, and the manufacturing cost can be reduced.
[0075]
Further, since the basic unit of video decoding is a macroblock unit in the processor 7, a block in the code conversion unit 9 and the pixel operation unit 10, and two blocks in the pixel read / write unit 11, the capacity of the buffer buffer in video decoding is minimized. It becomes possible.
<2 Second Embodiment>
The video and audio processing apparatus according to the present embodiment is configured to perform a compression function (hereinafter, referred to as an encoding process) and a graphics function in addition to a decoding function of the compressed stream data.
<2.1 Configuration of video and audio processing device>
FIG. 16 is a block diagram illustrating a configuration of a video and audio processing device according to the second embodiment of the present invention.
[0076]
The video / audio processing device 2000 includes a stream input / output unit 21, a buffer memory 22, a FIFO memory 24, an input / output processor 25, a memory controller 26, a processor 27, an internal memory 28, a code conversion unit 29, a pixel operation unit 30, a pixel read / write It comprises a unit 31, a video output unit 12, an audio output unit 13, a buffer 200, and a buffer 201. The video / audio processing device 2000 has the following functions in addition to the functions of the video / audio processing device 1000 shown in FIG. That is, a compression function for video data and audio data and a graphics function for drawing polygon data are added.
[0077]
Therefore, in the video / audio processing apparatus 2000, the components having the same names as those in FIG. 4 have exactly the same functions, and further have a function of performing a compression function and a graphics function. Hereinafter, the description of the same points as in FIG. 4 will be omitted, and the description will focus on the different points.
The stream input / output unit 21 is different in that it is bidirectional. That is, when the MPEG data is transferred from the buffer memory 22 under the control of the input / output processor 25, the transferred parallel data is converted into serial data and output to the outside as an MPEG data stream.
[0078]
The difference is that the buffer memory 22 and the FIFO memory 24 are also bidirectional.
The input / output processor 25 controls the data transfer along the routes (5) to (8) in addition to controlling the data transfer along the routes (1) to (4) shown in the first embodiment. .
(1) Stream input / output unit 21 → buffer memory 22 → memory controller 26 → external memory 3
(2) External memory 3 → memory controller 26 → FIFO memory 24
(3) External memory 3 → memory controller 26 → buffer memory 22 → video output unit 12
(4) External memory 3 → memory controller 26 → buffer memory 22 → audio output unit 13
(5) External memory 3 → memory controller 26 → internal memory 28
(6) External memory 3 → memory controller 26 → pixel read / write unit 31
(7) FIFO memory 24 → memory controller 26 → external memory 3
(8) External memory 3 → memory controller 26 → buffer memory 22 → stream input / output unit 21
The paths (5) and (6) are the paths of the original data when the video data and the audio data are encoded, and (7) and (8) show the paths of the MPEG stream after compression.
[0079]
First, the encoding process will be described. It is assumed that data to be encoded is stored in the external memory 3. The video data in the external memory 3 is transferred to the pixel read / write unit 31 by controlling the memory controller 26 by the pixel read / write unit 31.
The pixel read / write unit 31 performs a process of writing video data to the second buffer 201 and a process of generating a difference image. The difference image generation processing includes motion detection (calculation of a motion vector) in units of blocks and generation of a difference image. Therefore, the pixel read / write unit 31 includes therein a motion detection circuit that detects a motion vector by searching in a rectangular area similar to the encoding target block and a reference frame. Instead of the motion detecting circuit, a motion estimating circuit for estimating a motion vector to be coded by using a motion vector of an already calculated block of an adjacent frame may be provided.
[0080]
The pixel operation unit 25 receives the difference image data in block units, and performs DCT, IDCT, quantization processing (hereinafter, Q processing), and IQ. The video data thus quantized is stored in the buffer 200.
The code conversion unit 29 receives the quantized data from the buffer 200 and performs variable length code processing (VLC). The variable-length coded data is stored in the first-in first-out memory 24, is stored in the external memory 3 through the memory controller 26, and the processor 27 adds header information for each macroblock.
[0081]
The video data in the external memory 3 is transferred to the internal memory 28 via the memory controller 26. The processor 27 compresses the audio data in the internal memory 28 by a process of adding header information for each macroblock and a time division.
As described above, the encoding process is performed in a path reverse to that of the first embodiment.
[0082]
Next, the graphics processing will be described. The graphics processing is a three-dimensional image generation processing performed by a combination of rectangular figures called polygons. In this apparatus, processing for generating pixel data inside the polygon from pixel data at the vertex coordinates of the polygon is performed.
First, the vertex data of the polygon is stored in the external memory 3.
[0083]
The vertex data is stored in the internal memory 28 by the processor 27 controlling the memory controller 26. The processor 27 reads the vertex data from the internal memory 28, performs preprocessing of DDA (Digital Difference Analysis), and writes the data to the FIFO memory 24.
The code conversion unit 29 reads vertex data from the FIFO memory 24 according to the instruction of the pixel operation unit 30 and transfers the data to the pixel operation unit 30.
[0084]
The pixel operation unit 30 performs a DDA process and transmits the result to the pixel read / write unit 31. The pixel read / write unit 31 performs a Z-buffer process or an α-blending process according to an instruction from the processor 27 and writes image data to the external memory 3 via the memory controller 26.
<2.1.1 Pixel operation unit>
FIG. 17 is a block diagram illustrating a configuration of the pixel operation unit 30.
[0085]
In the figure, the same components as those of the pixel operation unit 10 shown in FIG. 7 are denoted by the same reference numerals, and the description thereof will be omitted.
The difference is that the pixel operation unit 30 has three execution units (501a to 501c) in the pixel operation unit 10 shown in FIG. That is, an instruction register 309 and a distribution unit 310 are added.
[0086]
The reason why the execution units 501a to 501c have three surfaces is to improve the calculation performance. Specifically, in the graphics processing, the color images RGB are independently processed in parallel. In the IQ and Q processes, three multipliers 502 are used to increase the speed. In the IDCT, time is reduced by using a plurality of multipliers 502 and a plurality of adders / subtractors 503. In the IDCT, there is an operation called a butterfly operation, and since there is a dependency between all data that is the source of the operation, a data line 103 for performing inter-unit communication of the execution units 501a to 501c is provided.
[0087]
The first instruction memory 506 and the second instruction memory 507 store microprograms for DCT, Q processing, and DDA in addition to IDCT and IQ. FIG. 18 shows an example of the storage contents of the first instruction memory 506 and the second instruction memory 507. Compared to FIG. 8, a Q processing microprogram, a DCT microprogram, and a DDA microprogram are added.
[0088]
The instruction pointer holding units 308a to 308c are provided corresponding to the execution units 501a to 501c, and each have a conversion table that converts an address input from the first program counter and outputs the converted address to the instruction register unit 309. The converted address indicates the register number of the instruction register unit 309. Further, the instruction pointer holding units 308a to 308c hold the later-described modify flags and output the same to the instruction execution units 501a to 501c.
[0089]
Regarding the conversion table, the instruction pointer holding units 308a, 308b, and 308c, for example, when the input addresses are 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, and 12, respectively, The converted address is output.
Instruction pointer holding unit 308a: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12
Instruction pointer holding unit 308b: 2, 1, 4, 3, 6, 5, 8, 7, 10, 9, 12, 11
Instruction pointer holding unit 308c: 4, 3, 2, 1, 8, 7, 6, 5, 12, 11, 10, 9
As shown in FIG. 23, the instruction register unit 309 includes a plurality of registers for holding microinstructions, three selectors, and three output ports. The three selectors select the microinstruction of the register specified by the conversion address (register number) input from the instruction pointer units 308a, 308b, 308c. The three output ports are provided corresponding to the selectors, and output the microinstructions selected by the selectors to the execution units 501a to 501c via the distribution unit 310. Three selectors and output ports are provided to supply different micro-instructions to three adder / subtracters 503 (or three multipliers 502) at the same time. In this embodiment, it is assumed that the three output ports are selectively supplied to one of the three adders / subtractors 503 and the three multipliers 502 via the distribution unit 310.
[0090]
For example, the instruction register unit 309 includes registers R1 to R16 (register numbers 1 to 16). The microprogram stored in the registers R1 to R16 represents a matrix operation process required in DCT and IDCT, and is stored so as to perform the same process in any of the three register numbers. That is, in the microprogram having the above three execution orders, the order of some microinstructions whose execution order is interchangeable is changed. This is because the execution units 501a to 501c execute the microprogram in parallel, so that resource interference such as competition of register (not shown) access between the execution units 501a to 501c is avoided. The matrix operation processing includes multiplication, transposition, and transfer of an 8 × 8 matrix.
[0091]
Next, the microinstruction stored in each register of the instruction register unit 309 is in a mnemonic format.
"Op Ri, Rj, dest, (modify flag)"
Is written. However, the microinstruction of the instruction register unit 309 is only “op and Ri, Rj and (modify flag)”. The “dest” part is specified from the instruction memories 506 and 507. Designated from the partial instruction pointer holding units 308a to 308c of "(modify flag)".
[0092]
Here, “op” is an operation code indicating a multiplication instruction, an addition / subtraction instruction, a transfer instruction, and the like, and “Ri, Rj” is an operand. The multiplication instruction is an instruction executed by each of the multipliers 502 in the three execution units 501a to 501c. The addition instruction and the transfer instruction are instructions executed by each of the multipliers 502 in the three execution units 501a to 501c. It is.
“Dest” indicates the storage location of the operation result. This “dest” is specified not from the register of the instruction register unit 309 but from the instruction memory 506 (for a multiplication instruction) or the instruction memory 507 (for an addition / subtraction instruction or a transfer instruction). This is to make the microprogram of the instruction register unit 309 common to the execution units 501a to 501c. If the transfer destination is designated by a register, it is necessary to prepare a separate microprogram for each of the execution units 501a to 501c, and the capacity of the microprogram is increased several times.
[0093]
The "modify flag" is a flag indicating whether addition or subtraction is performed in an addition / subtraction instruction. The "modify flag" is separately specified not from the register of the instruction register unit 309 but from the instruction pointer holding units 308a to 308c. This is because the constant matrix used for the matrix operation in DCT and IDCT includes a row (or column) in which all elements are “1” and a row (or column) in which all elements are “−1”. By designating the "modify flag" from 308a to 308c, the same microprogram of the instruction register unit 309 can be shared.
[0094]
When the three micro-instructions input from the instruction register unit 309 are addition / subtraction instructions, the distribution unit 310 determines the “op and Ri, Rj” part and the “dest” input from the instruction memory 506. The part and the “modify flag” input from the instruction pointer units 308 a to 308 c are distributed to the three adders / subtractors 503, and the micro instruction of the instruction memory 506 is simultaneously distributed to the three multipliers 502. When the three micro-instructions input from the instruction register unit 309 are multiplication instructions, the distribution unit 310 compares the “op and Ri, Rj” portions with “dest” input from the instruction memory 506. Is distributed to the three multipliers 502, and the microinstruction in the instruction memory 507 is distributed to the three adder / subtracters 503. In other words, the micro instruction supplied to the three adder / subtractors 503 by the distributor 310 is one micro instruction supplied from the instruction memory 507 for the instruction common to the three adder / subtractors 503, and the three adder / subtractor For the different addition / subtraction instructions at 503, three microinstructions from the instruction register unit 309 are supplied respectively. Similarly, the microinstruction supplied to the three multipliers 502 is a microinstruction supplied from the instruction memory 506 for an instruction common to the three multipliers 502, and an instruction for a multiplication instruction different in the three multipliers 502. The micro instruction from the register unit 309 is supplied to each.
[0095]
According to such a configuration of the pixel operation unit 30, the storage capacity of the instruction memory 506 and the instruction memory 507 can be reduced.
Assuming that the pixel operation unit 30 does not include the instruction pointer holding units 308a to 308c, the instruction register unit 309, and the distribution unit 310, the instruction memory 506 and the instruction memory 507 are all stored in the three execution units 501a to 501c. To supply different microinstructions for it, three microinstructions must be stored in parallel.
[0096]
FIG. 22 shows an example of the contents stored in the instruction memory 506 and the instruction memory 507 in the case where the instruction pointer holding units 308a to 308, the instruction register unit 309, and the distribution unit 310 are not provided. In the figure, a 16-step microprogram is stored, and one microinstruction has a 16-bit length. In this case, the instruction memory 506 and the instruction memory 507 require a total storage capacity of 1536 bits (16 steps × 16 bits × 3 × 2) because three micro-instructions are recorded in parallel.
[0097]
On the other hand, FIG. 23 shows an example of the storage contents of the instruction pointer holding units 308a to 308c and the instruction register unit 309 in the pixel operation unit 30 of the present embodiment. Also in this figure, a 16-step microprogram is stored, and one microinstruction has 16 bits. In the figure, the instruction pointer holding units 308a to 308c store 16 register numbers (4-bit length), respectively, and the instruction register unit 309 stores 16 microinstructions. In this case, the storage capacity of the instruction pointer holding units 308a to 308c and the instruction register unit 309 may be 448 bits (16 steps × (12 + 16)). As described above, in the pixel operation unit 30, the storage capacity of the microprogram can be significantly reduced. Actually, since "dest" and "modify flag" are separately issued, a recording capacity or a circuit corresponding to that is required. The instruction memories 506 and 507 designate "dest" in the microinstructions and issue multiplication instructions and addition / subtraction instructions common to the execution units 501a to 501c. I didn't even delete it. If the instruction register unit 309 is provided with six output ports, the instruction memory 506 and the instruction memory 507 can be deleted.
[0098]
In FIG. 23, the instruction pointer holding units 308a to 308c output the conversion address (register number) when the value of the first program counter is 0 to 15, but the present invention is not limited to this. For example, the conversion address may be output when the value of the first program counter is 32 to 47. In this case, an appropriate offset value may be added to the value of the first program counter. As a result, an arbitrary address string indicated by the first program counter can be converted into a conversion address.
[0099]
With the above configuration, in the present embodiment, not only decoding processing of compressed video data and compressed audio data, but also encoding processing of video and audio data and graphics processing based on polygon data are possible. Further, the processing efficiency is improved by the parallel operation of the plurality of execution units. In addition, by changing the order of some of the micro-instructions in the instruction register units 308a to 308c, resource interference among a plurality of execution units can be avoided, so that the processing efficiency is further improved.
[0100]
In the above embodiment, the configuration having three execution units is shown because it is advantageous in that each of the RGB colors can be calculated independently. Furthermore, the number of execution units may be any number as long as it is three or more.
Further, in the above embodiment, it is desirable that each of the video and audio processing devices 1000 and 2000 be implemented as a one-chip LSI. Further, the external memory 3 has been described as being external to the chip, but it may be configured to be built in one chip.
[0101]
In the above embodiment, the stream input / output unit 1 (or the stream input / output unit 21) stores the MPEG stream (or the video / audio data) in the external memory. May be configured.
Furthermore, in the above embodiment, the IO processor 5 performs the task switching every four instruction cycles, but may perform the task switching every plural instruction cycles other than the four instruction cycles. Also, the number of instruction cycles for task switching may be weighted in advance for each task and set to a different number of instruction cycles. Also, the number of instruction cycles for each task may be weighted according to the priority and the urgency.
[0102]
【The invention's effect】
A video and audio processing apparatus according to the present invention is a video and audio processing apparatus that externally inputs and decodes a data stream including compressed audio data and compressed video data, and outputs the decoded data to an output device. Input / output processing means for performing input / output processing occurring in the memory; and decoding processing means for performing decoding processing mainly for decoding a data stream stored in a memory in parallel with the input / output processing; The video data decoded by the means and the decoded audio data are stored in a memory, and the input / output processing includes inputting the data stream asynchronously input from the outside, storing the data stream in a memory, and storing the data stream in a memory. Supplying the decoded data stream to the decoding processing means, an external display device and an audio output device, respectively. To match the output rate from the memory, and is configured to perform and outputting them as input and output processing.
[0103]
According to this configuration, in addition to the input / output processing means and the decoding processing means operating in parallel in a pipeline, the asynchronous processing and the decoding processing are shared between the input / output processing means and the decoding processing means. The processing means can be released from the processing that occurs asynchronously and can exclusively use the decoding processing. As a result, the video and audio processing apparatus efficiently executes a series of processing of stream data input, decode, and output, thereby enabling full decoding of stream data (without dropping frames) without using a high-speed operation clock. ing.
[0104]
Further, the decoding processing means performs a sequential processing mainly on a condition determination on the data stream, including a header analysis of the compressed audio data and the compressed video data and a decoding of the compressed audio data. The sequential processing means performs a standard processing in parallel with the sequential processing. The routine processing may include a routine processing means for decoding the compressed video data excluding the header analysis of the compressed video data.
[0105]
According to this configuration, the processing efficiency can be greatly improved by eliminating the coexistence of the sequential processing having different processing characteristics and the routine processing suitable for the parallel processing in one unit. In particular, the processing efficiency of the routine processing means can be improved. This is because, in the present video / audio processing apparatus, since the standard processing means is released from the asynchronous processing and the sequential processing described above, it can exclusively use various standard operations required for decoding the compressed video data. As a result, high processing performance can be obtained without using a high-speed operation clock.
[0106]
Further, the input / output processing means includes an input means for inputting an asynchronous data stream from outside, a video output means for outputting decoded video data to an external display device, and an audio data decoded to an external audio output device. And a processor for executing the first to fourth tasks stored in the instruction memory while switching, wherein the first task is a program for transferring a data stream from an input unit to the memory. The second task is a program for supplying a data stream from the memory to the decoding processing means; the third task is a program for outputting decoded video data from the memory to a video output unit; The task is a program that outputs decoded audio data from the memory to an audio output unit. It may be.
[0107]
Here, the processor includes a program counter unit having at least four program counters corresponding to the first to fourth tasks, and an instruction memory for storing each task program using an instruction address indicated by one program counter. An instruction fetch unit that fetches an instruction, an instruction execution unit that executes the instruction fetched by the instruction fetch unit, and controls the instruction fetch unit to sequentially switch the program counter every time a predetermined number of instruction cycles elapse. It may be configured to have a task control unit.
[0108]
According to this configuration, regardless of the range of the input rate and the input cycle of the stream data determined by the external device, and the output rate and the output cycle of the video data and the audio data determined by the external display device and the external audio output device, respectively. In addition, there is an effect that a response delay to an input / output request is extremely small.
Further, the video and audio processing apparatus of the present invention includes: an input unit for inputting a data stream including compressed audio data and compressed video data; and a sequential processing mainly based on condition determination for the data stream. A sequential processing means for analyzing header information added to a predetermined block unit in the inside and decoding compressed audio data in a data stream; and a routine processing mainly including routine operations, using a result of the header analysis. Routine processing means for decoding the compressed video data in the data stream in units of predetermined blocks in parallel with the sequential processing, wherein the sequential processing unit performs routine processing when header analysis of the predetermined block is completed. To start decoding of the predetermined block, and when receiving a notification of the end of decoding of the predetermined block from the routine processing means, the next predetermined block is notified. It may be configured to start the header analysis.
[0109]
According to this configuration, the sequential processing means is responsible for header analysis which requires a wide variety of condition judgments for both compressed video data and compressed audio data, and is also responsible for decoding audio compressed data. On the other hand, the routine processing means is responsible for a large amount of routine operation on block data of the compressed video data. Due to such role sharing, the sequential processing means performs overall audio decoding, which requires a smaller amount of computation than video decoding, analyzes the header of compressed video data, and controls the routine processing means. Under the control, the routine processing unit performs a routine operation exclusively, so that efficient processing without waste can be realized. Therefore, the processing capability can be obtained without operating at a high frequency, and the manufacturing cost can be reduced. In addition, the sequential processing means sequentially performs overall audio decoding, header analysis of the compressed video data, and control of the standard processing means, so that it can be constituted by one processor.
[0110]
Further, the routine processing means performs a variable length decoding of the compressed video data in the data stream in accordance with an instruction of the sequential processing means, and performs a predetermined operation on the video block obtained by the variable length decoding. Computing means for performing inverse quantization and inverse discrete cosine transform, and synthesizing means for restoring video data by performing motion compensation processing by synthesizing a video block after inverse discrete cosine transform and a decoded block. ,
The sequential processing unit includes an obtaining unit that obtains header information that has been subjected to variable-length decoding by the data conversion unit, an analyzing unit that analyzes the obtained header information, and a notification that notifies a parameter obtained as an analysis result to the standard processing unit. Means, an audio decoding means for decoding the compressed audio data in the data stream inputted by the input means, and an operation of the audio decoding means when receiving an interrupt signal notifying the completion of decoding of the predetermined block from the routine processing means. The information processing apparatus may be configured to include: a control unit that instructs the data conversion unit to start variable-length decoding of the video block when the acquisition unit is stopped and the acquisition unit is activated and the notification unit notifies the notification.
[0111]
According to this configuration, the sequential processing unit performs header decoding after performing header analysis on a predetermined block basis such as a macroblock, and starts decoding the header of the next block when decoding of the predetermined block is completed by the standard processing unit. . As described above, the sequential processing means repeats the header analysis and the audio decoding in a time-division manner, so that it can be realized at low cost by one processor. Further, since the routine processing means does not need to perform a wide variety of condition judgment processing, it can be made into dedicated hardware (or hardware and firmware) at low cost.
[0112]
Here, the arithmetic unit further includes a first buffer having a storage area corresponding to one block, and the data conversion unit includes a variable length decoding unit that performs variable length decoding of compressed video data in a data stream; First address table means for storing a first address sequence in which addresses of storage areas of one buffer are arranged in zigzag scan order; and second address storing means for storing a second address sequence in which addresses of storage areas of the first buffer are arranged in alternate scan order. The configuration may include an address table unit and a writing unit that writes block data obtained by variable length decoding of the variable length decoding unit into the first buffer according to one of the first address sequence and the second address sequence.
[0113]
According to this configuration, the writing unit can write the block data in the storage area of the first buffer in response to both the zigzag scan and the alternate scan. Therefore, when reading the block data from the storage area of the first buffer, the arithmetic means does not need to change the order of the read addresses, and can always read in the same order of the read addresses regardless of the scan type.
[0114]
Further, the analysis unit may calculate a quantization scale and a motion vector based on the header information, and the notifying unit may notify the calculation unit of the quantization scale and notify the synthesis vector of the motion vector. Good.
According to this configuration, the calculation of the motion vector can be assigned to the sequential processing means, and the synthesizing means can routinely perform the motion compensation processing using the calculated motion vector. .
[0115]
Further, the arithmetic means designates a first and a second control storage section for storing a microprogram, a first program counter for designating a first read address in the first control storage section, and a second read address, respectively. A first program counter, a selector for selecting one of the first read address and the second read address and outputting the selected read address to the second control storage unit, a multiplier and an adder; An execution unit for performing inverse quantization and inverse discrete cosine transform in block units by microprogram control by the unit;
May be provided.
[0116]
According to this configuration, the microprogram (firmware) does not need to perform a wide variety of condition determination processing, and only realizes routine processing, so that the program size is small and easy to create, which is suitable for cost reduction. . Moreover, the multiplier and the adder can be operated independently and in parallel using two program counters.
[0117]
Further, when the second read address is selected by the selector, the execution unit performs the processing using the multiplier and the processing using the adder independently and in parallel, and the first read address is selected by the selector. At this time, the processing using the multiplier and the processing using the adder may be performed in conjunction with each other.
According to this configuration, the idle time of the multiplier and the adder can be reduced, and the processing efficiency can be improved.
[0118]
Here, the arithmetic unit further includes a first buffer for storing the video block from the data conversion unit, and a second buffer for storing the block subjected to the inverse discrete cosine transform by the execution unit, and The storage unit stores a microprogram for performing an inverse quantization process and a microprogram for performing an inverse discrete cosine transform, and the second control storage unit includes a microprogram for performing an inverse discrete cosine transform, and a video block subjected to an inverse discrete cosine transform. Is transferred to the second buffer, and the execution means executes processing for transferring the inverse discrete cosine transformed video block to the second buffer and processing for inversely quantizing the next video block in parallel. And the process of performing inverse discrete cosine transform of the inversely quantized video block may be performed in conjunction with the multiplier and the adder.
[0119]
According to this configuration, since the inverse quantization process and the transfer process to the second buffer are performed in parallel, the processing efficiency can be improved.
The input means further inputs polygon data, the sequential processing means further analyzes the polygon data to calculate the vertex coordinates of the polygon and the inclination of the edge, and the routine processing means further calculates the polygon data. The image data of the polygon may be generated according to the vertex coordinates and the inclination.
[0120]
According to this configuration, the sequential processing unit is in charge of analyzing the polygon data, and the standard processing unit is in charge of the standard image data generation processing. The video / audio processing apparatus can perform graphics processing for efficiently generating image data from polygon data.
Here, the first and second control storage units further store a microprogram for performing scan conversion by the DDA algorithm, and the execution unit further performs the above based on the vertex coordinates and the inclination calculated by the sequential processing means. The scan conversion may be performed by microprogram control.
[0121]
According to this configuration, the generation of the image data can be easily realized by the scan conversion microprogram in the first and second control storage units.
Further, the synthesizing means generates a differential block representing a differential image from the video data to be further compressed, the second buffer holds the further generated differential image, and the first control storage unit further performs a discrete cosine transform. The second control storage unit further stores a microprogram for performing discrete cosine transform and a microprogram for transferring the video block subjected to discrete cosine transform to the first buffer. The means further performs discrete cosine transform and quantization on the difference block held in the second buffer and transfers the result to the first buffer. The data conversion means further performs variable length coding on the block in the first buffer. And the sequential processing means further performs header information on the predetermined block which has been variable-length coded by the data conversion means. The may be configured to add.
[0122]
According to this configuration, the routine processing unit is responsible for quantization and discrete cosine transform as routine processing, and the sequential processing unit is responsible for processing that requires a condition determination (addition of header information). In this case, the present video / audio processing apparatus can efficiently execute the encoding process from image data to compressed video data without using a high-speed clock.
Further, the arithmetic means designates a first and a second control storage section for storing a microprogram, a first program counter for designating a first read address in the first control storage section, and a second read address, respectively. A first program counter, a selector for selecting one of the first read address and the second read address and outputting the selected one to the second control storage unit, a multiplier and an adder; A plurality of execution units for performing inverse quantization and inverse discrete cosine transform in block units by microprogram control by a storage unit are provided, and each execution unit is configured to share and process partial blocks obtained by dividing blocks. May be.
[0123]
According to this configuration, since a plurality of execution units execute the operation instructions in parallel, a large amount of routine operations can be efficiently executed in parallel at the pixel level.
Further, the arithmetic means is further provided corresponding to each execution unit, each conversion table is a plurality of address conversion table holding a conversion address partially changed the address order corresponding to a predetermined address sequence, An instruction register group consisting of a plurality of registers for storing individual microinstructions constituting a microprogram for realizing a predetermined operation in association with a translation address; and a first and second control storage units and a plurality of execution units. A switching unit that switches a microinstruction output from the first control storage unit or selector to each execution unit to a microinstruction in an instruction register and outputs the microinstruction to a plurality of execution units. (2) If the read address is an address in the predetermined address string, the address is converted according to each of the address conversion tables. It is converted to. The instruction register group may be configured to output a micro instruction corresponding to each translation address output from the translation table.
[0124]
According to this configuration, while the plurality of execution units execute the microprogram in parallel, resource interference such as access competition between the execution units can be avoided, and the processing can be performed more efficiently.
Here, each of the conversion tables should be added or subtracted in accordance with a microinstruction output indicating addition and subtraction in the register while the first program counter outputs the first read address in the predetermined address string. Outputting a flag indicating a perception to the plurality of execution units, wherein each execution unit performs addition and subtraction according to the flag, and the flag is set according to a microinstruction of the second control storage unit. Is also good.
[0125]
According to this configuration, since the conversion table specifies whether to perform addition or subtraction by a microinstruction, the same microprogram can be shared in two ways, so that the total capacity of the microprogram can be further reduced, The hardware scale can be reduced, and the cost can be reduced.
The second control storage unit further stores a micro instruction execution result storage location along with the micro instruction output in the register while the first program counter outputs the first read address in the predetermined address sequence. May be output to the plurality of execution units, and each execution unit may store the execution result according to the storage location information.
[0126]
According to this configuration, since the storage destination information can be specified separately from the microprogram in the instruction register group, the microprogram can be shared in different processing, for example, partial processing of matrix operation. As a result, the total capacity of the microprogram can be further reduced, and the hardware scale can be reduced, and the cost can be reduced.
[Brief description of the drawings]
FIG. 1 is an explanatory diagram of a decoding process by a video / audio decoder according to a first conventional technique.
FIG. 2 is an explanatory diagram of a decoding process by a two-chip decoder according to a second conventional technique.
FIG. 3 is a block diagram illustrating a schematic configuration of an image processing apparatus according to the first embodiment of the present invention.
FIG. 4 is a block diagram illustrating a configuration of an image processing apparatus according to the first embodiment of the present invention.
FIG. 5 is a diagram showing an MPEG stream in a hierarchical manner and showing operation timing of each section of the image processing apparatus.
FIG. 6 is a diagram illustrating an analysis of a macroblock header by a processor 7 and control contents for other units.
FIG. 7 is a block diagram illustrating a configuration of a pixel operation unit 10.
FIG. 8 shows an example of a microprogram stored in a first instruction memory 506 and a second instruction memory 507.
FIG. 9 is a diagram showing operation timings of the pixel operation unit 10;
FIG. 10 is a block diagram showing a detailed configuration of a pixel read / write unit 11;
FIG. 11 is a block diagram showing a configuration of an IO processor 5.
FIG. 12 is a block diagram showing a detailed configuration example of an instruction reading circuit 53.
FIG. 13 is a time chart showing the operation timing of the IO processor 5;
FIG. 14 is a block diagram illustrating a configuration of a task management unit.
FIG. 15 is an explanatory diagram showing a decoding operation after the FIFO memory 4;
FIG. 16 is a block diagram illustrating a configuration of an image processing apparatus according to a second embodiment of the present invention.
FIG. 17 is a block diagram illustrating a configuration of a pixel operation unit 30.
FIG. 18 shows an example of contents stored in a first instruction memory 506 and a second instruction memory 507.
FIG. 19 is a block diagram illustrating a configuration of a code conversion unit 9;
FIG. 20 shows a block storage area for storing 8 × 8 spatial frequency data and a zigzag scan route.
FIG. 21 shows a block storage area for storing 8 × 8 spatial frequency data and a route of an alternate scan.
FIG. 22 shows an example of the contents stored in the instruction memory 506 and the instruction memory 507 when the instruction pointer holding units 308a to 308, the instruction register unit 309, and the distribution unit 310 are not provided.
FIG. 23 shows an example of storage contents of the instruction pointer holding units 308a to 308c and the instruction register unit 309.
[Explanation of symbols]
1 Stream input section
2 Buffer memory
3 External memory
4 FIFO memory
5 I / O processor
5a DMAC
6 Memory controller
7 processor
8 Internal memory
9 Code converter
10. Pixel operation unit
12 Video output section
13 Audio output unit
14 Host I / F section
1000 video / audio processing device
1001 Input / output processing unit
1002 decode processing unit
1003 sequential processing unit
1004 Standard processing unit

Claims (31)

A video processing apparatus that outputs data including video data generated by decoding a data stream including compressed video data input from the outside to the outside,
A processor that performs an input / output process that is affected by external factors, including a process of externally inputting the data stream and a process of outputting data including the generated video data to the outside,
Independently of the processor, in parallel with decoding processing means for generating data including video data by decoding a data stream supplied from the processor,
The processor comprises:
An instruction memory for storing a plurality of task programs representing the contents of the input / output processing;
A program counter unit having a program counter corresponding to each of the plurality of task programs;
An instruction fetch unit that fetches an instruction from the instruction memory based on an instruction address indicated by one program counter;
An instruction execution unit that executes the instruction fetched by the instruction fetch unit;
A task control unit that controls the instruction fetch unit to sequentially switch a program counter to be used every time a predetermined number of instruction cycles elapses,
The processor executes the plurality of task programs and a register set while switching the set,
The processor further executes a task program for analyzing the MPEG data to extract individual elementary streams,
The decoding processing means,
Routine processing mainly including routine operations, in which variable length decoding of header information added to each block in the data stream and compressed video data in the data stream, and compression of the variable length decoded video data in block units are performed. A standard processing means for performing a process of decoding
It is a sequential processing mainly based on a condition judgment, and comprises a sequential processing means for performing a time-divisional analysis processing of header information subjected to variable length decoding processing and a decoding processing of compressed audio data in a data stream,
A video processing device for decoding each of the extracted elementary streams. The processor inputs the data stream from the outside, stores the data stream in the memory, reads the data stream stored in the memory, and supplies the data stream to the decoding processing unit according to the progress of the decoding operation of the decoding processing unit,
The decoding processing means stores data including the generated video data in the memory,
The video processing device according to claim 1, wherein the processor outputs the video data stored in the memory to the outside. The routine processing means performs a variable length decoding process on the header information and the compressed video data, and after the variable length decoding process on the block of the compressed video data ends, analyzes the header information of the next block to the sequential processing unit. To start,
When the header information to be analyzed is variable-length coded, the sequential processing means instructs the fixed-form processing means to perform variable-length decoding of the header information, and obtains the variable-length-decoded header information. After the analysis and the acquisition of the header information of the block are completed, the fixed-length processing unit starts variable-length decoding of the compressed video data of the block, and uses the analyzed header information to perform the variable-length decoding. The video processing apparatus according to claim 2 , wherein an instruction is given to decode the compressed video data. The routine processing means,
Data conversion means for performing variable length decoding of the compressed video data in the data stream according to the instruction of the sequential processing means,
Arithmetic means for performing inverse quantization and inverse discrete cosine transform by performing a predetermined operation on block data obtained by variable-length decoding,
4. The image processing apparatus according to claim 3 , further comprising: synthesizing means for synthesizing the block data after the inverse discrete cosine transform and the rectangular image of the decoded frame stored in the memory to restore video data corresponding to the block. The video processing device according to the above. The arithmetic unit further includes a first buffer having a storage area corresponding to one block,
The data conversion means,
A variable-length decoding unit for performing variable-length decoding of the compressed video data in the data stream;
First address table means for storing a first address sequence in which addresses of the storage area of the first buffer are arranged in zigzag scan order;
Second address table means for storing a second address sequence in which addresses of the storage area of the first buffer are arranged in an alternate scan order;
5. The writing device according to claim 4 , further comprising a writing unit that writes block data obtained by the variable length decoding of the variable length decoding unit to the first buffer according to one of the first address sequence and the second address sequence. 6. Video processing device. The writing means,
Table address generating means for sequentially generating table addresses for the first address table means and the second address table means;
Of the addresses of the first address string and the second address string respectively output from the first table means and the second table means to which the table address is input,
Address selection means for selecting one,
6. The video processing apparatus according to claim 5 , further comprising address output means for outputting the selected address to said first buffer. The processor comprises:
An input means for inputting an asynchronous data stream from outside;
Video output means for outputting decoded video data to an external display device,
Audio output means for outputting decoded audio data to an external audio output device;
A processor that executes the first to fourth tasks stored in the instruction memory while switching the tasks,
The first task is a program for transferring a data stream from an input unit to the memory,
The second task is a program for supplying a data stream from the memory to a decoding processing unit,
The third task is a program that outputs decoded video data from the memory to a video output unit,
3. The video processing device according to claim 2, wherein the fourth task is a program that outputs decoded audio data from the memory to an audio output unit. The video processing device according to claim 7 , wherein the program counter unit has at least four program counters corresponding to the first to fourth tasks. The processor further includes:
A register unit having at least four register sets corresponding to the first to fourth tasks,
9. The video processing device according to claim 8 , wherein the task control unit switches a register set to be used by the instruction execution unit simultaneously with switching of a program counter. The task control unit includes:
A counter for counting the number of instruction cycles according to a clock signal each time the program counter is switched;
10. The video processing device according to claim 9 , further comprising: a switching instruction unit configured to control the instruction fetch unit to switch a program counter when the count value of the counter reaches the predetermined number. A video and audio processing device that inputs, decodes, and outputs a data stream including compressed audio data and compressed video data,
A processor that performs input / output processing for storing a data stream that is asynchronously input due to external factors in a memory,
A fixed-length process mainly including a fixed-size operation, the variable-length decoding process of the header information added to the block unit in the data stream stored in the memory and the compressed video data in the data stream, and the variable-length decoded compression Standard processing means for performing processing of decoding video data in block units;
Sequential processing means for performing sequential processing mainly on condition determination, and performing time-divisional analysis processing of header information subjected to variable-length decoding processing and decoding processing of compressed audio data in a data stream stored in a memory; With
The audio data decoded by the sequential processing means, and the video data decoded by the standard processing means are stored in the memory,
The output processing is further decoded audio data, reads the video data from the memory, respectively, include an external display device, the output process tailored to each output rate of the audio output device,
The processor comprises:
An instruction memory for storing a plurality of task programs representing the contents of the input / output processing;
A program counter unit having a program counter corresponding to each of the task programs,
An instruction fetch unit that fetches an instruction from the instruction memory based on an instruction address indicated by one program counter;
An instruction execution unit that executes the instruction fetched by the instruction fetch unit;
A task control unit that controls the instruction fetch unit to sequentially switch a program counter to be used every time a predetermined number of instruction cycles elapses,
The processor executes the plurality of task programs and a register set while switching the set,
The processor further executes a task program for analyzing the MPEG data to extract individual elementary streams,
The video / audio processing apparatus, wherein the decoding processing means decodes each of the extracted elementary streams. The routine processing means performs a variable length decoding process on the header information and the compressed video data, and after the variable length decoding process on the block of the compressed video data ends, analyzes the header information of the next block to the sequential processing unit. To start,
When the header information to be analyzed is variable-length coded, the sequential processing means instructs the fixed-form processing means to perform variable-length decoding of the header information, and obtains the variable-length-decoded header information. After the analysis and the acquisition of the header information of the block are completed, the fixed-length processing unit starts variable-length decoding of the compressed video data of the block, and uses the analyzed header information to perform the variable-length decoding. The video / audio processing apparatus according to claim 11 , wherein an instruction is given to decode the compressed video data. The routine processing means,
Data conversion means for performing variable length decoding of the compressed video data in the data stream according to the instruction of the sequential processing means,
Arithmetic means for performing inverse quantization and inverse discrete cosine transform by performing a predetermined operation on block data obtained by variable-length decoding,
13. The image processing apparatus according to claim 12 , further comprising: synthesizing means for restoring video data corresponding to the block by synthesizing the block data after the inverse discrete cosine transform and the rectangular image of the decoded frame stored in the memory. The video / audio processing apparatus according to the above. The arithmetic means further includes a first buffer having a storage area corresponding to one block,
The data conversion means,
A variable-length decoding unit for performing variable-length decoding of the compressed video data in the data stream;
First address table means for storing a first address sequence in which addresses of the storage area of the first buffer are arranged in zigzag scan order;
Second address table means for storing a second address sequence in which addresses of the storage area of the first buffer are arranged in an alternate scan order;
14. The writing device according to claim 13 , further comprising: a writing unit that writes block data obtained by variable length decoding of the variable length decoding unit into the first buffer according to one of a first address sequence and a second address sequence. Video and audio processing device. The writing means,
Table address generating means for sequentially generating table addresses for the first address table means and the second address table means;
Address selecting means for selecting one of an address of the first address string and an address of the second address string output from the first table means and the second table means to which the table address is input, respectively;
15. The video / audio processing apparatus according to claim 14 , further comprising address output means for outputting the selected address to said first buffer. The processor comprises:
An input means for inputting an asynchronous data stream from outside;
Video output means for outputting decoded video data to an external display device,
Audio output means for outputting decoded audio data to an external audio output device;
A processor that executes the first to fourth tasks stored in the instruction memory while switching the tasks,
The first task is a program for transferring a data stream from an input unit to the memory,
The second task is a program for supplying a data stream from the memory to the decoding processing means,
The third task is a program that outputs decoded video data from the memory to a video output unit,
12. The video / audio processing apparatus according to claim 11 , wherein the fourth task is a program for outputting decoded audio data from the memory to an audio output unit. 17. The video and audio processing device according to claim 16 , wherein the program counter unit has at least four program counters corresponding to the first to fourth tasks. The processor further includes a register unit having at least four register sets corresponding to the first to fourth tasks,
18. The video / audio processing apparatus according to claim 17 , wherein the task control unit switches a register set to be used by the instruction execution unit simultaneously with switching of a program counter. The task control unit includes:
A counter for counting the number of instruction cycles according to a clock signal each time the program counter is switched;
19. The video / audio processing apparatus according to claim 18 , further comprising: a switching instruction unit configured to control the instruction fetch unit to switch a program counter when a count value of the counter reaches the predetermined number. . The routine processing means,
Data conversion means for performing variable length decoding of the compressed video data in the data stream according to the instruction of the sequential processing means,
Arithmetic means for performing inverse quantization and inverse discrete cosine transform by performing a predetermined operation on block data obtained by variable-length decoding,
19. The image processing apparatus according to claim 18 , further comprising: synthesizing means for synthesizing the block data after the inverse discrete cosine transform and a rectangular image of the decoded frame stored in the memory to restore video data corresponding to the block. The video / audio processing apparatus according to the above. The arithmetic means further includes a first buffer having a storage area corresponding to one block,
The data conversion means,
A variable-length decoding unit for performing variable-length decoding of the compressed video data in the data stream;
First address table means for storing a first address sequence in which addresses of the storage area of the first buffer are arranged in zigzag scan order;
Second address table means for storing a second address sequence in which addresses of the storage area of the first buffer are arranged in an alternate scan order;
21. The writing device according to claim 20 , further comprising: a writing unit that writes block data obtained by variable-length decoding of the variable-length decoding unit into the first buffer according to one of a first address sequence and a second address sequence. Video and audio processing device. The writing means,
Table address generating means for sequentially generating table addresses for the first address table means and the second address table means;
Address selecting means for selecting one of an address of the first address string and an address of the second address string output from the first table means and the second table means to which the table address is input, respectively;
22. The video / audio processing apparatus according to claim 21 , further comprising address output means for outputting the selected address to said first buffer. The analysis means calculates a quantization scale and a motion vector based on the header information,
21. The video and audio processing apparatus according to claim 20 , wherein the notifying unit notifies the calculating unit of a quantization scale and the synthesizing unit of a motion vector. The calculating means includes:
A first control storage unit and a second control storage unit each storing a microprogram;
A first program counter for designating a first read address in the first control storage unit;
A second program counter for designating a second read address;
A selector for selecting one of a first read address and a second read address and outputting the selected one to the second control storage unit;
An execution unit having a multiplier and an adder, and having a block unit for performing inverse quantization and inverse discrete cosine transform by microprogram control by the first control storage unit and the second control storage unit. 24. The video / audio processing apparatus according to claim 23 , wherein: When the second read address is selected by the selector, the execution unit performs processing using a multiplier and processing using an adder independently and in parallel, and the first read address is selected by the selector. 25. The video / audio processing apparatus according to claim 24 , wherein the processing using the multiplier and the processing using the adder are performed in conjunction with each other. The calculating means further comprises:
A first buffer for holding a video block from the data conversion means,
A second buffer for holding the block subjected to the inverse discrete cosine transform by the execution unit,
The first control storage unit stores a microprogram for performing an inverse quantization process and a microprogram for performing an inverse discrete cosine transform,
The second control storage unit stores a microprogram for performing inverse discrete cosine transform and a microprogram for transferring the video block subjected to inverse discrete cosine transform to the second buffer,
The execution means executes a process of transferring the inverse discrete cosine transformed video block to the second buffer and a process of inversely quantizing a next video block in parallel, and executes the inversely quantized video block. 26. The video / audio processing apparatus according to claim 25 , wherein the process of performing the inverse discrete cosine transform is performed in conjunction with the multiplier and the adder. The synthesizing unit further generates a difference block representing a difference image from the video data to be compressed,
The second buffer holds a further generated difference image,
The first control storage unit further stores a microprogram for performing a discrete cosine transform and a microprogram for performing a quantization process,
The second control storage unit further stores a microprogram for performing discrete cosine conversion and a microprogram for transferring the video block subjected to discrete cosine conversion to the first buffer,
The execution means further executes discrete cosine transform and quantization on the difference block held in the second buffer, and transfers the result to the first buffer.
The data conversion unit further performs variable length coding on the block of the first buffer,
27. The video / audio processing apparatus according to claim 26 , wherein the sequential processing unit further adds header information to a predetermined block that has been subjected to variable-length coding by the data conversion unit. The calculating means includes:
A first control storage unit and a second control storage unit each storing a microprogram;
A first program counter for designating a first read address in the first control storage unit;
A second program counter for designating a second read address;
A selector for selecting one of a first read address and a second read address and outputting the selected one to the second control storage unit;
A plurality of execution units each including a multiplier and an adder, and performing inverse quantization and inverse discrete cosine transform in block units by microprogram control by the first control storage unit and the second control storage unit; Prepare,
24. The video / audio processing apparatus according to claim 23 , wherein each of the plurality of execution units shares and processes a partial block obtained by dividing the block. The calculating means further comprises:
A plurality of address conversion tables provided corresponding to each of the plurality of execution units, each conversion table holding a conversion address in which the address order is partially changed corresponding to a predetermined address sequence;
An instruction register group consisting of a plurality of registers for storing individual microinstructions constituting a microprogram for realizing a predetermined operation in association with a translation address;
A microinstruction provided between the first control storage unit and the second control storage unit and the plurality of execution units, and output from the first control storage unit or the selector to each of the plurality of execution units. A switching unit that switches to a microinstruction in an instruction register and outputs the microinstruction to the plurality of execution units,
When the first read address or the second read address is an address in the predetermined address string, the address is converted into a conversion address by each of the plurality of address conversion tables, and the instruction register group is a conversion table. 29. The video / audio processing apparatus according to claim 28 , wherein micro-instructions corresponding to the respective conversion addresses output from are output. Each of the conversion tables may further include whether to add or subtract while the first program counter outputs the first read address in the predetermined address string, with a microinstruction output indicating addition and subtraction in the register. Is output to the plurality of execution units,
Each of the plurality of execution units performs addition and subtraction according to the flag,
The video / audio processing apparatus according to claim 29 , wherein the flag is set in accordance with a micro instruction of the second control storage unit. The second control storage unit is further configured to store a microinstruction execution result in accordance with a microinstruction output in the register while the first program counter outputs a first read address in the predetermined address string. Output to the plurality of execution units,
30. The video / audio processing apparatus according to claim 29 , wherein each of the plurality of execution units stores an execution result according to storage destination information.

Images