JP2007165962A - Stream data processing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a stream data processing device capable of effectively utilizing hardware resources in stream data processing for executing multiple steps of processing. <P>SOLUTION: A data discard determining means 104 obtains a prediction presentation enable time to be presented as a video image or a sound for data to be processed in each item of processing in the multiple steps of processing of the stream data processing, compares the prediction presentation enable time with a time stamp, and discards the data determined to be not in time of presentation schedule. Thus, the hardware resources in the stream data processing can be effectively utilized. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a stream data processing apparatus for presenting video and audio. More specifically, the present invention determines whether or not presentation is in time for the scheduled presentation time, and discards the video and audio by discarding those that are not in time. The present invention relates to a stream data processing apparatus that can effectively use hardware resources necessary for presenting a message.

  For content included in stream data such as video, audio, and music content, the real-time nature of presentation time is regarded as important, so scheduled presentation that indicates when and where in the stream data should be presented in the stream data Time information (hereinafter referred to as “time stamp”) is included. For example, the time stamp may be expressed as a relative time from the reproduction start time of the stream data, or may be expressed as an absolute time based on reference time information transmitted from a GPS, a broadcast station, a base station, or the like. Hereinafter, processing for presenting stream data as video or audio is referred to as “stream data processing”.

  In general, in a system that presents stream data as video or audio (hereinafter referred to as “stream data processing device”), in order to present the stream data as video or audio, decoding / encoding / conversion processing or output timing control processing for the stream data is performed. To do. Examples of decoding, encoding, and conversion processing are described in H.264. This is conversion processing from H.264 video data to YUV video data, and conversion processing from AAC (Advanced Audio Codec) audio data to PCM (Pulse Code Modulation) audio data. Depending on the type of stream data, one or more decoding / encoding / conversion processes may be performed. For example, H.264 expressed in the PES (Packetized Elementary Stream) format of MPEG2 TS. H.264 data. In this case, from the PES data, the H.264 data H.264 video data conversion processing; Conversion processing from H.264 video data to YUV video data is performed. The output timing control process performs a process for presenting the designated data at the designated time based on the time stamp given to the stream data.

  In general, in a stream data processing apparatus, the multi-stage processing (decoding / encoding / conversion processing and output timing control processing) is often processed in parallel in a pipeline manner with respect to stream data. The stream data is divided into predetermined units, and multi-stage processing is sequentially performed on the data of each division unit. For example, certain stream data is divided into two division units 1 and 2. It is assumed that the division unit 2 exists after the division unit 1 in the stream data, and the time stamp given to the division unit 2 indicates a presentation time later than that given to the division unit 1. At this time, after performing decoding / encoding / conversion processing of division unit 1, output timing control processing of division unit 1 and decoding / encoding / conversion processing of division unit 2 are performed in parallel, and then output of division unit 2 is performed. Perform timing control processing. When there are two or more decoding / encoding / conversion processes (in order, decoding process 1 and decoding process 2), after performing decoding process 1 on division unit 1, decoding process 2 and division unit 2 on division unit 1 are performed. The decoding process 1 is performed in parallel. In this way, when each processing stage is performed in parallel, the processing time fluctuates due to variations in the time required to execute each processing, the use status of the hardware resources of the stream data processing device, and the content of each division unit. In order to cope with this, a temporary storage area is prepared after a certain process is performed on the division unit until the next process can be performed. Hereinafter, the division unit is referred to as “data”, the temporary storage area is referred to as “data queue”, and the stream data processing means this multi-stage parallel processing.

  Conventionally, in order to maintain the real-time property in the stream data processing device, when the data existing in the data queue corresponding to the presentation timing control device is not in time for the time described in the time stamp (underflow), the blue screen or immediately before The video was displayed and the data was discarded. Here, when discarding data, because the data discard determination is performed when an underflow occurs, each process such as encoding, decoding, conversion processing so far is performed on all data, and regarding the discarded data Hardware resources for performing each process such as encoding, decoding, and conversion processing are wasted, and hardware resources are not effectively used.

Therefore, according to Patent Document 1, moving image encoded data for each partial region of a moving image signal is decoded. From an input encoded data bit stream, PTS (reproduction output time management information) and the bit stream are used. The STC (reproduction reference time) of the separated moving image encoded data is extracted, and both time information are compared. If there is a margin in the decoding process, normal processing is performed. If there is no margin in the decoding process, processing is performed. A method for effectively utilizing hardware resources by omitting the symbol is disclosed.
JP 2004-289745 A

  However, in the conventional method, it is determined whether or not the data is to be decrypted based on the time information given to the data and a certain threshold value. Therefore, the calculation amount at each processing stage of the stream data processing is considered. However, the flexible processing according to the usage status of hardware resources and the contents of data is not always performed.

  The present invention solves the above-described conventional problems, and takes into account the amount of calculation of each process of the decoding process, and predicts that each process of the decoding process is completed for the data every time the data is processed By obtaining the presentable time, data determined to be in time for the presentation time based on the time stamp assigned to the data is discarded. Accordingly, an object of the present invention is to provide a stream data processing apparatus that can effectively use hardware resources in accordance with the use status of hardware resources and the contents of data.

  In order to achieve the above object, a stream data processing apparatus of the present invention is a stream data processing apparatus that performs multi-stage processing on stream data, and the data included in the stream data has a scheduled presentation time. A data processing unit that includes a time stamp, executes a stage of processing included in the multi-stage processing on the data, and outputs the processed data; and the data output from the data processing unit A data queue to be temporarily stored, a queue management unit for writing and reading data to and from the data queue, and notifying the data output from the data processing unit to the data processing unit of the next stage; A processing time tube for holding the processing time required for the data processing means to process the data for each data processing means. And a processing time for the processing time management means to hold an estimated time to complete the multi-stage processing for the data to be processed when the queue management means writes or reads data to or from the data queue. A data discard determination means for determining whether to discard the data to be processed based on the predicted presentable time and a time stamp of the processing target data; Data discarding means for discarding the data to be processed based on the determination result of the discard determination means is provided.

  The stream data processing apparatus according to the present invention obtains a predicted presentable time at which all subsequent processing is completed for the data at each processing of the multi-stage processing, for example, at each processing such as encoding, decoding, and conversion processing. By comparing the predicted presentable time of the data with the time stamp and appropriately discarding the data that is determined not to be in time for the presentation time, there is an effect that unnecessary hardware resources are not consumed. .

  FIG. 1 is a block diagram showing an example of the configuration of a stream data processing apparatus 101 according to an embodiment of the present invention.

  1, the stream data processing apparatus 101 includes a data processing unit 102, a queue management unit 103, a data discard determination unit 104, a data discard unit 105, a processing time management unit 106, a data queue 107, and a time management unit 108. Reference numeral 109 denotes an input device, and 110 denotes an output device. Here, the stream data refers to video and audio content expressed by a digital data string. In addition to video and audio, the stream data also includes other content such as subtitles and logos, such as character strings and still images in which the real time property of the presentation time is important. Furthermore, the stream data is multiplexed and expressed as one digital data string, for example, MPEG2 or MPEG4 TS (Transport Stream) that stores moving picture, audio, and caption data as one digital data string. Including.

  The data processing means 102 is realized by, for example, software or at least one or more hardware, or a combination of software and at least one or more hardware. One or more data processing means 102 exist in the stream data processing apparatus 101, and the number varies depending on the type of stream data to be processed. The data processing means 102 is identified by the processing order for the stream data. When there are n data processing means 102 in a certain stream data processing, the data processing means 102 is identified as the data processing means 1, the data processing means 2,. Is done. Each data processing unit 102 performs decoding, encoding, conversion processing, output timing control processing, or the like on data. As an example of the data processing means 102 in charge of decoding, encoding, and conversion processing, H.264 is used. H.264 decoder. The data processing means 102 sends the queue management means 103 to the H.264 data. H.264 data is received. The H.264 decoding process is performed, and the data converted into YUV data is output to the queue management means 103. When the data processing unit 102 outputs the processed data, information indicating itself is added as an output. An example of the data processing means 1 is a PES decoder. As the input device 109, for example, PES data is input as stream data from a transport decoder, the PES data is analyzed, and ES (Elementary Stream) data in the PES is converted into PES data. A PTS (Presentation Time Stamp) is added as a time stamp, and the processed data is output to the queue management unit 103. An example of data processing means in charge of output timing control processing is video output device control processing. The queue management means 105 acquires YUV data read from the data queue 107, and the time when the YUV data is added to the data. The data is output to the output device 110 at the time indicated by the stamp. Here, the output device 110 is, for example, a liquid crystal display.

The queue management unit 103 is realized by, for example, software, at least one or more hardware, or a combination of software and at least one hardware. The queue management unit 103 receives data from the data processing unit 102 and stores the data in the data discard determination unit (for example, a memory such as SRAM or DRAM, a hard disk, a writable optical disk, a writable DVD, a floppy (registered trademark) disk, an SD card) Etc.), a data write request is issued. After the data is stored, information indicating that the data storage is completed is returned to the data transmission source. The queue management means 103 can identify which data processing means 102 has transmitted from the information added to the data processing means 102, and write request to store it in the data queue 107 corresponding to the data processing means of the transmission source. Can be issued. Hereinafter, the data queue 107 that holds and stores the data transmission source transmitted from a certain data processing means k is referred to as “data queue k” (1 ≦ k ≦ n−1). That is, the destination to which the data transmitted from the data processing unit 1 is written is the data queue 1, the destination to which the data transmitted from the data processing unit 2 is written is the data queue 1, and the data transmitted from the data processing unit n-1 is the destination. The writing destination is the data queue n-1. Later, the number of data stored in one data queue k and Q _k. Further, the queue management unit 103 issues a request to read data stored in the storage to the data discard determination unit in response to a data acquisition request from the data processing unit 102. When the data is read, it is returned to the data processing means 102. The queue management unit 103 can identify to which data processing unit 102 the data is transmitted based on the information added to the data processing unit 102. When returning data to a certain data processing means m, the data stored first in the data queue m-1 is returned to the data processing means m (2 ≦ m ≦ n). That is, the data to be transmitted to the data processing means 2 is stored in the data queue 1, the data to be transmitted to the data processing means 3 is stored in the data queue 2,. The data queue n-1 is stored. When the queue management unit 103 receives a data read request from a certain data processing unit m, if no data is stored in the corresponding data queue m-1, the queue management unit 103 stores data in the data queue m-1. Is returned to the data processing means m.

Note that the number of data Q _k may be the total size of the data stored in the data queue k because it is sufficient to know the amount of data stored in the data queue k.

  The data discard determination unit 104 is realized by, for example, software, at least one hardware, or a combination of software and at least one hardware. The data discard determination unit 104 obtains a predicted presentable time for processing target data when a request for writing or reading data from the queue management unit 103 to the data queue 107 is received. The predicted presentable time is obtained using the processing time of each data processing means held by the processing time management means. In addition, the data discard determination unit 104 determines whether the data is in time for the presentation time based on the predicted presentable time and the time stamp of the processing target data, and notifies the data discard unit 105 of the determination result.

  Further, when the data discard determination unit 104 obtains the predicted presentable time, the time management unit 108 inquires the current time and the queue management unit 103 asks the number of data stored in each data queue 107, and the current time and the number of data. The predicted presentation time may be obtained using. Details regarding the data discard determination method of the data discard determination means 104 will be described later.

  The data discard unit 105 discards the data when a data discard notification is received from the data discard determination unit 104. In addition, when a data write processing notification to the data queue 107 is received, the data is written to the data queue 107, and when a read request notification to the queue management unit 103 is received, the queue management unit from the data queue 107 The data is read to 103.

  The data discard determination unit 104 may discard data, read data to the queue management unit 103, or write data to the data queue 107.

The processing time management means 106 is realized by, for example, software or at least one or more hardware, or a combination of software and at least one or more hardware. The number of processing times held by the processing time management means 106 changes according to the type of stream data to be processed by the encoder, decoder, etc., but holds the same number as the data processing means 102, that is, n processing times. The processing time held by the processing time management unit 106 corresponds to each data processing unit 102 on a one-to-one basis. That is, the processing time management unit 106 holds the processing time T _k (1 ≦ k ≦ n) for processing the data of the corresponding data processing unit k (1 ≦ k ≦ n). For example, when the corresponding data processing means 102 is realized by software, the maximum time required to process one data when 100% of CPU time is allocated to the data processing is measured and calculated in advance. Hold as time. When the corresponding data processing means is realized by at least one or more hardware, the maximum time required for processing by the hardware for data processing is measured and calculated in advance, and stored as processing time. When the corresponding data processing means is realized by a combination of software and at least one or more hardware, the maximum CPU time required for one data processing when 100% of the CPU time is allocated to the data processing, The sum of the time required for processing by the hardware for one data processing is measured and calculated in advance, and is stored as the processing time. The processing time T _k held by the processing time management means 106 is T _k = S, where S _k is the maximum time required for the data processing means k corresponding to the processing time T _k to process one data. _The value obtained by _k + S _{k + 1} + S _{k + 2} +... + S _n , that is, the sum of the time required for subsequent data processing of the data to be processed may be held as the processing time T _k . Further, the processing time management unit 106 returns the processing time to be held in response to the processing time acquisition request from the data discard determination unit 104. In addition to returning the processing time in response to a request from the data discard determination unit 104, the processing time management unit 106 may notify the data discard determination unit 104 of the processing time periodically. In this case, the processing time notification means determines the processing time obtained by the latest notification so far as the processing time of the processing time management means 106 until a new notification is received. Furthermore, as a method of holding the processing time, each corresponding data processing unit 102 may hold each one, or the data processing unit 102 and the processing time may be associated with each other and collectively managed. Good.

The time management unit 108 is realized by, for example, software or at least one or more hardware, or a combination of software and at least one or more hardware. Based on the request from the data discard determination unit 104, the time management unit 108 returns the current time _Tnow to the request source. The time management means 108 expresses the current time _Tnow as, for example, a 32-bit integer, and can handle both relative time and absolute time depending on the type of stream data to be handled. For example, when the stream data to be handled is digital broadcasting, the time management unit 108 holds the current time T _now based on PCR (Program Clock Reference) that is reference clock information transmitted from the broadcasting station. In addition to returning the current time _Tnow in response to a request from the data discard determination unit 104, the time management unit 108 may notify the data discard determination unit 104 of the current time _Tnow periodically. At this time, the side receiving the notification of the current time T _now until further notice, to determine the time obtained in the most recent notification of up to it as the current time T _now.

  FIG. 2 is a flowchart showing an operation at the time of a data storage request of the stream data processing apparatus according to the present embodiment. When a certain data processing means j (1 ≦ j ≦ n−1) finishes the data processing, it issues a data storage request to the queue management means 103 (step S201). When the queue management means 103 receives a data storage request from the data processing means j, it issues a data write request to the data discard determination means 104 (step S202). When the data discard determination unit 104 receives a data write request from the queue management unit 103, the data discard determination unit 104 obtains a predicted presentable time for the data to be processed (step S203), and performs a data discard determination (step S204).

Here, the data discard determination method of the data discard determination means 104, which is the processing of step S203 and step S204, will be described. The data discard determination unit 104 first obtains the predicted presentation time for the data to be processed, so that each data processing unit processed after the data processing unit j, that is, each of the data processing unit j + 1 to the data processing unit n, is processed. The processing time management unit 106 is requested for a processing time corresponding to the data processing unit 102. Next, the queue management unit 103 is requested for the number of data stored in the data queues from the data queue j to the data queue n−1, and the current time T _now is requested to the time management unit 108. And the prediction presentable time is calculated | required by the following Numerical formula (1). That is, the sum of the product of the processing time corresponding to each data processing means 102 from the data processing means j + 1 to the data processing means n and the number of data stored in the data queue and the current time obtained from the time management means 108. Is added as the predicted presentable time of the data (step S203). It should be noted that the processing time, the number of data, and the order of requesting the current time are not limited as long as the predicted presentable time is obtained. By using such a calculation formula, it is possible to obtain a predictable presentation time that dynamically changes at a certain timing during multi-stage processing, and more correctly determine whether the processing is in time or not in time. .

  In the process of step S204, the data discard determination unit 104 performs a data discard determination based on the predicted presentable time and the time stamp given to the data. When it is determined by the data discarding determination unit 104 that the predicted presentable time is later than the time stamp given to the data, that is, the scheduled presentation time (step S204), the data discarding unit 105 is notified of the discarding of the data. The data discarding means 105 that received the notification discards the data (step S205). If the data discarding determination unit 104 determines that the predicted presentable time is not later than the time stamp given to the data, that is, the scheduled presentation time (step S204), the data discarding unit 105 stores the data in the data queue j + 1. A notification is sent to write, and the data discarding means 105 that has received the notification writes the data in the data queue j + 1 (step S206). As described above, when the data processing unit 102 issues a data storage request, the stream data processing apparatus according to the present embodiment performs processing for determining whether or not to discard the data to be processed.

  FIG. 3 is a flowchart showing the operation at the time of data acquisition request of the stream data processing apparatus according to the present embodiment. When a data processing unit h (2 ≦ h ≦ n) issues a data acquisition request to the queue management unit 103 (step S301), the queue management unit 103 issues a data read request to the data discard determination unit 104 (step S302). ). When the data discarding determination unit 104 receives a data read request from the queue management unit 103, the data discarding unit 105 requests the data discarding unit 105 to read the data stored first in the data queue h-1. A predicted presentable time is obtained for the data read from h-1 (step S303), and a data discard determination is made (step S304).

Here, the data discard determination method of the data discard determination unit 104, which is the processing of step S303 and step S304, will be described. The data discard determination unit 104 first obtains the predicted presentation time for the data to be processed, so that each data processing unit processed after the data processing unit h, that is, each of the data processing unit h + 1 to the data processing unit n, is processed. The processing time management unit 106 is requested for a processing time corresponding to the data processing unit 102. Next, the queue management unit 103 is requested for the number of data stored in the data queues from the data queue h to the data queue n-1, and further the current time _Tnow is requested to the time management unit 108. And the prediction presentable time is calculated | required by the following Numerical formula (2). That is, the processing time period T _h processing time management unit 106 corresponding to the data processing means h holds, from the data processing unit h + 1 to the processing time and the data queue corresponding to each of the data processing means 102 to the data processing means n The sum of the product of the number of stored data and the current time obtained from the time management means 108 is used as the predicted presentable time of the data (step S303). It should be noted that the processing time, the number of data, and the order of requesting the current time are not limited as long as the predicted presentable time is obtained.

  In the process of step S304, the data discard determination unit 104 determines whether to discard the data based on the predicted presentable time and the time stamp given to the data. When the data discarding determination unit 104 determines that the predicted presentable time is later than the time stamp given to the data, that is, the scheduled presentation time (step S304), the data discarding unit 105 notifies the data discarding unit 105 of the discarding. The data discarding means 105 that received the notification discards the data (step S305). If the data discarding determination unit 104 determines that the predicted presentable time is not later than the time stamp given to the data, that is, the scheduled presentation time (step S304), the data discarding unit 105 stores the data in the queue management unit 103. The data discarding unit 105 transmits the data to the queue managing unit 103 (step S306). The queue management means 103 that has received the data transmits the data to the data processing means h (step S307). As described above, when the data processing unit 102 issues a data acquisition request, the stream data processing apparatus according to the present embodiment performs processing for determining whether to discard data for the processing target data.

  FIG. 2 shows the data discarding determination operation only when the data processing means 102 of the stream data processing apparatus 101 according to the embodiment of the present invention issues a data storage request, and FIG. 3 shows the operation according to the embodiment of the present invention. Although the data discarding determination operation is performed only when the data processing unit 102 of the stream data processing apparatus 101 issues a data acquisition request, in the stream data processing apparatus 101 according to the embodiment of the present invention, the data processing unit 102 includes Data discarding determination may be performed each time a data storage request and a data acquisition request are issued.

  As described above, according to the stream data processing apparatus 101 according to the embodiment of the present invention, it is possible to discard in advance data determined to be in time for the scheduled presentation time. By omitting, hardware resources can be used effectively. Also, video and audio quality can be improved by effectively using hardware resources.

  The data discard determination unit 104 identifies the data processing unit 102 that takes the longest processing time from the processing time of each data processing unit held by the processing time management unit 106, and the identified data processing unit 102 performs processing. The discard determination may be performed only on data to be processed or data to be output after processing. As a result, it is not necessary to make a data discard decision in each data processing means 102 by making a discard decision only for the data processing means 102 with the longest processing time, and hardware resources can be used more effectively. .

  Also, in any description relating to any embodiment, requests and notifications between components are implemented as follows, for example. For example, it is realized by a function call in software. In this case, for example, a request / notification source performs a function call, and various data are transmitted to the request / notification destination using function arguments. Further, in the case of a request, a response to the request is transmitted to the request source using a return value of the function, for example. In addition, when realized by message communication in software, for example, the request / notification source stores various data in the message and transmits the message to the request / notification destination. Further, in the case of a request, a response to the request is transmitted to the request source using a message, for example. When implemented by hardware, for example, a request line / notification / response to a request is transmitted via a control line, and various data are transmitted via a signal line.

  Each functional block constituting the stream data processing apparatus of the present invention may be realized as an LSI that is typically an integrated circuit. These may be individually made into one chip, or may be made into one chip so as to include a part or all of them. The name used here is LSI, but it may also be called IC, system LSI, super LSI, or ultra LSI depending on the degree of integration.

  Furthermore, the method of the integrated circuit is not limited to the LSI, and may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after manufacturing the LSI, or a reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI may be used.

  Further, if integrated circuit technology comes out to replace LSI's as a result of the advancement of semiconductor technology or a derivative other technology, it is naturally also possible to carry out function block integration using this technology. Biotechnology can be applied.

  The stream data processing apparatus according to the present invention can discard a buffer that is not in time for presentation of stream data at an early stage of processing. It is useful for information terminals, mobile phones equipped with a playback function, communication equipment, and the like.

The block diagram which shows an example of a structure of the stream data processing apparatus 101 which concerns on embodiment of this invention. The flowchart which shows the operation | movement at the time of the data storage request | requirement of the stream data processing apparatus concerning this Embodiment. The flowchart which shows the operation | movement at the time of the data acquisition request of the stream data processing apparatus concerning this Embodiment

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Stream data processing apparatus 102 Data processing means 103 Queue management means 104 Data discard determination means 105 Data discard means 106 Processing time management means 107 Data queue 108 Time management means 109 Input device 110 Output device

Claims (4)

A stream data processing device that performs multi-stage processing on stream data, wherein the data included in the stream data includes a scheduled presentation time as a time stamp,
Data processing means for executing a certain stage of processing included in the multi-stage processing on the data and outputting the processed data;
A data queue for temporarily storing data output from the data processing means;
Queue management means for writing and reading data to and from the data queue and notifying the data output from the data processing means to the data processing means of the next stage;
A processing time management means for holding a processing time required for the data processing means to process data for each data processing means;
When the queue management unit writes or reads data to or from the data queue, a prediction time to complete the multi-stage processing on the processing target data is predicted using a processing time held by the processing time management unit A data discard determination means for determining whether to discard the data to be processed based on the predicted presentable time and the time stamp of the data to be processed;
A stream data processing apparatus comprising: a data discarding unit that discards the processing target data based on a determination result of the data discarding determination unit. A time management means for managing the current time;
The queue management means holds the number of data stored in the data queue;
The predicted presentable time is obtained based on the number of data in the data queue held by the queue management unit, the processing time held by the processing time management unit, and the current time held by the time management unit. The stream data processing apparatus according to claim 1, wherein: The predicted presentable time is obtained based on the product of the number of data in the data queue held by the queue management means and the processing time held by the processing time management means and the current time held by the time management means. The stream data processing apparatus according to claim 2. The data discard determination unit identifies the data processing unit that takes the longest processing time based on the processing time held by the processing time management unit, and determines the data processed by the identified data processing unit or the output data after processing The stream data processing apparatus according to claim 1, wherein only the data discarding determination is performed.

Images