JP2009116881A - Data processing device and its method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method for constructing and executing stream data processing to a series of processing points to improve portability. <P>SOLUTION: This method comprises connecting processing points called peer modules in data processing by link modules to set a stream data path, and scheduling in a single execution context to attain processing of high performance, and attaining these in layer configuration to achieve high portability and easiness in development. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

The present invention relates to a method for processing data independent of a specific operating system (OS).

In recent years, as multimedia data handled by computers, moving images, sounds, and the like have been used for various purposes. For example, a device called a hard disk recorder that converts a video such as a TV broadcast in real time using a video codec and records it on a hard disk,
Players that display moving image content data recorded as digital data on an optical disc such as a VD on a video display device are widely used. In general, in these contents processing, contents data must be processed within a predetermined time at a specified bit rate. Also, in order to be able to use these multimedia contents in various apparatuses, it is important that program modules that execute these complicated stream processes can be easily developed. For this reason, there is a method of executing a program for performing content processing in an OS kernel that easily realizes high performance. A stream is a data string that needs to be processed in one direction in a predetermined order. For example, there are a data string of audio data and a data string of moving image data. Stream processing refers to processing data included in these streams.

A general OS includes a user space, which is a space for executing applications, and an OS.
There are two kernel spaces, which are spaces for executing the above functions. When processing is performed in the kernel space, high-speed execution is possible, but a program executed in the kernel space basically depends largely on the specification of the OS in which the kernel is mounted. This is because, for example, the interface used when the function is used is different for each OS, and the function use procedure is different depending on the OS. It is also known that the device driver for using the device is specific to the platform and cannot be used. As a method for executing processing in the kernel space, there is a method for performing content processing by connecting a plurality of device drivers that process data called filters (for example, Patent Document 1).

This is because a software module called a filter is generated in the kernel space from an application program executed in the user space, the input and output of multiple filters are connected to build a flow path for content data, and processing starts from the application program This is a method for executing content processing by requesting a stop or the like. Here, filters and device drivers are modules specialized for devices and data formats, and each has an interface for input / output and control, and the data processed by each filter is passed to other filters. Process data. In this method, all filters are OS
High performance can be realized because it operates in the kernel part of. However, in this method, priority processing when a plurality of content processes are performed is not considered. For example, when processing and playing back a plurality of contents at the same time, there may be a request for increasing the priority of recording, but it is difficult to cope with this.

JP-A-10-283195

As described above, in the conventional method, there has been an implementation for handling a stream in the OS kernel space as a driver whose performance is easily guaranteed. This is highly dependent on the OS and hardware configuration (hereinafter, this combination is referred to as a platform), the program cannot support different platforms, and each platform has its own content data processing method. Therefore, the development of a program for processing content has to be performed for each platform, and there has been a first problem that the development process is increased.

Further, in Patent Document 1, since there is no management function in content processing, for example, when processing a plurality of contents at the same time, it is impossible to specify the priority of each processing. For example, when performing two processes at the same time, acquiring data from a certain device and recording it in storage, and acquiring data from a certain device and displaying it on the display, for example, when you want to preferentially perform the former process However, since there is no function for managing and managing these processes, there is a second problem that it is difficult to appropriately change the priority of both processes.

Furthermore, in the above case, when multiple content processes proceed based on their own implementation, the hardware resources are not properly managed or allocated, so the CP required for the process
There was also a third problem that a shortage of U resources occurred.

  The present invention solves at least one of the above problems.

In order to solve the first problem, the present invention provides a multimedia content processing means capable of supporting various platforms.

In the present invention, by providing a layer for providing a generalized interface for processing multimedia contents by absorbing a difference for each platform, a configuration in which the upper software module does not depend on the platform is realized.

In order to solve the second to third problems, in the present invention, a plurality of stream processes are executed using one execution context, for example, a process or a thread, and how many resources are used for each process of the content process. By centrally managing whether to allocate resources, processing priority processing and resource management are realized.

According to the present invention, a program that uses content data can be used on different platforms. In addition, the execution of a plurality of content processes can be managed, and the amount of used resources can be reduced.

  Embodiments of the present invention will be described below.

The outline of the processing flow in the present invention will be described with reference to FIG. This obtains moving image data from the disc (101) as moving image content, and displays it on the display (171).
And a process of reproducing by the speaker (181). These processes are handled by the device (
101), a processing point (100) for obtaining data, a processing point (120) for separating audio data and video data, and a processing point for outputting video to the video decoder (170) (
150) and a processing point (160) for outputting the audio to the audio decoder (180). A processing point is a part necessary for executing a predetermined process in a program for realizing a processing flow. A software module that executes processing at this processing point is called a peer module. A detailed configuration of the peer module will be described later with reference to FIG. Further, the link module connects these peer modules to each other, for example, the link module (110) is a peer module (100) and a peer module (120).
Connect. The link module has a function of transmitting data from one connected peer module to the other. As a result, the data acquired by the peer module (100) is sent to the video decoder (170) and the audio decoder (180) through the link module and other peer modules, so that the video is displayed on the display (171). Sound is reproduced from the speaker (180).

FIG. 2 is a diagram showing a software configuration in the present embodiment. In this configuration (210), a stream manager (for controlling stream processing in which peer modules are connected to each other)
230), and a data processing module manager (230) for managing the data processing module (241) for performing data conversion and the like. The stream manager manages the peer module and manages the peer module (231), the peer module scheduler (233) used to execute the processing of the peer module according to the execution schedule, and the peer module It is composed of a link module manager (232) for recording and managing link modules.

Further, a system interface (2) provided by the OS to use the functions of the OS.
80), a general-purpose system interface (260) for using resource management functions such as memory management, thread management, and time operation, and a data access module (access to the device to acquire data and control the device) 271), a data access module manager (270) is provided.

The application (200) can control the stream manager and the data processing module manager using the control interface (240), and can instruct a desired stream processing. Control that can be instructed will be described later with reference to FIG.

An interface layer (250) is formed by the layer including the general-purpose system interface (260) and the data access module manager (270). This interface layer provides functions provided by the system interface (280) provided by the OS to a higher layer using a generalized interface. This generalized interface is used to share a difference in protocol and instruction method resulting from a difference in platform and provide it to an upper layer. For example, there is a socket interface used for network communication. The functions executed by this interface are the same regardless of the platform, but the procedure for using this is different for each platform. For example, when a general Unix (registered trademark) including Linux is used to create a socket communicating with TCP, a socket function, a bind function, a listen function, and an accept function may be executed in order. On the other hand, Windows (registered trademark) requires different procedures according to different specifications. If these are handled by processing on the application program side as in the prior art, it is necessary to change all the processing with different procedures on the platform for each platform to be used. Therefore, in the general-purpose interface, for example, a generalized interface such as makeServerTCPsock is provided as a general-purpose function for performing processing for creating a server socket at a specified port. For example, in the general-purpose system interface for Linux, this mak
In the eServerTCPsock function, the process is executed according to the Linux procedure described above. Meanwhile Windo
In the ws version of the general-purpose system interface, the process is executed in the Windows procedure. Next, the general-purpose system interface for Linux is used when the software module of the present application is used in Linux, and the general-purpose system interface for Windows is used when used in Windows. Next, in the application program, when creating a socket,
When makeServerTCPSock is called, the general-purpose system interface executes a procedure corresponding to the platform, so that it is not necessary to change the application for each platform. Such a method makes it possible to generalize functions provided by the platform and access methods to devices. As described above, when the software module of FIG. 2 is migrated to a different platform, a part of the software module included in the layer (250) that uses a function or a usage procedure that differs depending on the platform is as described above. Changes are required, but higher layers, ie, control interface (220), data processing module manager (240), stream manager (23
0) and the application (200) can be migrated to another OS without changing the processing of the program.
16, FIG. 17, and FIG. 18 will be used to explain how the processing described in FIG. 1 is realized with the configuration in FIG.

FIG. 18 shows a table (2000) for recording the stream processing configuration, which is managed by the stream manager. One row of the table corresponds to one stream process. The stream process includes a stream process ID (2001), a peer module ID (2002) of a peer module constituting the stream process, a link module ID (2003) of a link module constituting the stream process, and a state (2004) of the stream process. .

FIG. 16 shows a table (1800) for recording the configuration of the peer module, which is managed by the peer module manager. One row of the table corresponds to one peer module. One peer module includes a peer module ID (1801), a stream processing ID (1802), a data processing module ID (1803), a data access module ID (1804), an input link module ID (1805), and an output link module ID (1806). ), An execution waiting data object matrix (1807), and a post-processing waiting data object matrix (1808). Execution pending data object matrix (18
07) and a post-processing waiting data object matrix (1808) are matrices in which data can be input and output, and are implemented as arrays, queues, hash tables, lists, or the like. Although the figure shows an example showing whether or not a matrix exists in the table, when a matrix exists in the peer module manager, the existing addresses are indicated in columns (1807) and (1808).
) May be recorded. A value of −1 in the link module ID indicates the peer module (100
) Indicates that the link module is not connected.

FIG. 17 shows a table (1900) for recording the configuration of the link module.
Managed by the link module manager. One row of the table corresponds to one link module. One link module has a link module ID (1901).
, Stream processing ID (1902), start point peer module ID (1903), end point peer module ID (1904), and buffer matrix (1905). Buffer matrix (19
05) is a matrix that can input and output data, and is implemented as an array, a queue, a hash table, or a list. Although an example showing whether or not a matrix exists in the table is shown in the figure, when a matrix exists in the link module manager, the existing address may be recorded in a column (1905).

FIG. 3 shows a hardware configuration used to implement this embodiment. C
Memory such as PU (300), RAM (310) or ROM (320), clock (3
30), a disk device (350) such as an HDD, a tuner (360) for receiving a broadcast of content data, a multimedia codec processor (370) for encoding / decoding multimedia data, and a built-in speaker TV receiver display (
380) are connected by a system bus (340). 2 is loaded into the memory and executed by the CPU. Further, this hardware configuration is used in, for example, an HDD recorder, a DVD recorder, a PC, or a surveillance camera.

FIG. 5 shows an example of commands issued to the control interface in order for the application of this embodiment to issue an instruction to the stream manager (230). FIG. 5 defines an instruction type (501) to the stream manager (230), its parameters (502), and a return value (503). These control interfaces are realized as functions or modules when the present embodiment is developed in a programming language such as C or C ++.

Note that it is possible to generate a data processing flow using the data access module ID, the data processing module ID, and the instructions specified in FIG. 5 and start the processing. Details will be described later with reference to FIG.

Instructions for creating and controlling a data processing flow will be described with reference to FIG. For example, when a stream process generation command (500) is executed, a stream process is generated. In particular,
A unique stream processing ID is assigned and unused 1 in FIG. 18 (2000).
The stream processing ID is recorded in one row, and the stream processing generates. That value is returned as the return value. Here, it is assumed that 1 is returned to the application.

Next, when the application executes the peer module generation instruction (520) using the stream processing ID, the application generates a peer module. Specifically, a new unique peer module ID is generated, the peer module ID is recorded in one unused line in FIG. 16 (1800), and the stream processing ID passed as a parameter is recorded. A peer module is generated by recording the peer module ID in the field of the peer module ID (2002) of the stream process corresponding to the stream process ID specified as a parameter in the table (2000). The assigned peer module ID is returned to the application as a return value. Here, it is assumed that 100 is returned.

Then, another peer module (120) (shown at 1820 in FIG. 16) is created in a similar manner. The module ID of the peer module is 120. Next, by using a link module generation instruction (540), these peer modules are coupled by the link module. Specifically, a new unique link module ID is assigned, and FIG.
00) is recorded in one unused line, and the values passed as parameters are recorded as the start point peer module ID and the end point peer module ID, respectively. In the example, they are 100 and 120, respectively. Next, in the table (1800), 110 is recorded in the output link module ID of the row (1810) where the peer module ID is 100, and then 110 is recorded in the input link module ID of the row (1820) where the peer module ID is 120. To do. Next, the link module ID is registered in the link module ID (2003) in the row (2010) where the stream processing ID is 1 in the table (2000). With these processes, generation and setting of the link module for coupling the peer modules are completed.

Next, the data processing module control instruction (
560) and a data access control instruction (570). For details of the data processing module ID in the parameter of the data processing module control instruction (560), refer to FIG.
The data access module ID in the parameter of the data access module control instruction will be described later with reference to FIG. For example, when a process with a data access module ID of 1 is assigned to a peer module with a peer module ID of 100, a data access module control instruction (570) is used. As a result, the data access module ID 1 is recorded in the data access module ID (1804) in the row where the peer module ID is 100 in the table (1800). Similarly, the data processing module is set using the data processing module control instruction (560). Details of the data processing module and the data access module will be described later with reference to FIG.

Further, when 580 data processing control command instructions are used, it is possible to control and set the data processing module using the “command” in the parameter. For example, in the case of a data processing module that performs network processing, the packet size in the data processing module can be set for changing processing using a “packet size setting command”. The data access module control command (570) can also be used to specify a file name when reading data from a file, for example, and to specify a port number when inputting and outputting data from a network.

The execution of data processing is controlled using a data processing command instruction (580) that issues a start command and a stop command, and can control the scheduling of each peer module related to the stream processing. This is because the stream processing table (2000)
Since the link module and the peer module related to the stream processing are registered, if the stream processing ID is designated, it is possible to change all the settings and operations of the related link module and the peer module.

The application receives the data processing module (40) via the event management command (590).
1), an event such as an error can be passed, and data processing can be terminated. Data processing is discarded by a stream processing discard instruction (510).
Or a peer module discard instruction or a link module discard instruction (530, 55).
0) can be selectively discarded. Deletion in this case corresponds to deletion of the contents of the registered table.

FIG. 13 shows a table (1300) for managing the correspondence between the data access module ID and the data access module. The table exists in the data access module manager (270). The table (1300) has a data access module ID (
1301), the corresponding address (1302) to the input / output execution processing unit, the address (1303) to the input / output completion processing unit, and the description of the function outline (1304, 1305). In the example shown in the figure, when the data access module ID is 1, (13
10), an I / O execution processing unit having a function of “data read request issuance to hard disk” existing at address 0xF98000000, and an I / O completion processing unit having a function of “acquiring data from hard disk” existing at address 0xF9808000 Is recorded.

About correspondence of data access module ID and processing that module actually performs,
For example, it is described in a manual or the like, and the user obtains a data access module ID that realizes a necessary function. Alternatively, a necessary item is selected from the contents described in the function column (1304, 1305).

FIG. 14 shows a table for managing the correspondence between the data processing module ID and the data processing module. The table exists in the data processing module manager (240). In this table (1500), a data processing module ID (1501), a corresponding address (1502) to the pre-processing unit, an address (1503) to the post-processing unit, and a description of these functions are recorded. Yes. In the example shown in the figure, the data processing module ID is 100.
In the case of (1510), a pre-processing unit having a function of “instructing reading of a specified file. Specify next scheduling time” existing at address 0xFA8000000, and 0xFA
A post-processing unit having a function of “outputting acquired data to an output link” existing at address 808000 is recorded. The correspondence between the data processing module ID and the processing actually performed by the module is described in a manual, for example, and the user obtains a data processing module ID that realizes a necessary function. Alternatively, a necessary item is selected from the contents described in the function column (1504, 1505).

FIG. 15 shows a procedure in which the application uses the instructions shown in FIG. 5 to combine the peer module and the link module to construct a data processing program in order to perform the data processing shown in FIG.

First, the application generates a data processing flow using the stream processing generation instruction (500) (step 1500), and obtains a stream processing ID.

Next, using the stream processing ID, a file read peer module is generated using a peer module generation instruction (520) to obtain a peer module ID. Next, the data processing module control instruction (56
0) using the data processing module ID (step 1511), and assigning the data access module to the data access module control command (570) using the data access module ID (step 1512). Similarly, a multiplexer peer module is created (step 1520) and a data processing module is assigned (step 152).
1). Thereafter, similarly, a video peer module is created (step 1530), and a data processing module and a data access module are allocated (steps 1531 and 1532).
The voice peer module is created (step 1540), and the data processing module and the data access module are allocated (steps 1541 and 1542). Next, a link module connecting the multiplexer peer module from the file read peer module is created using the link module generation instruction (540) (step 1550) to obtain the link module ID, and the video peer module is similarly obtained from the multiplexer peer module. A link module to be connected and a link module to connect a voice peer are created (step 156).
0, 1570). By processing from the application in this way, peer modules to be used are related to each other, and a data processing module and a data access module to be used for processing are assigned.

As described above, the generation and setting of the peer module, the link module, and the like are performed by the application via the control command. Generate necessary peer modules based on instructions executed from the application, connect these peer modules using a link module and create a data processing flow, then information and processing necessary for processing such as bit rate By setting the data access module and data processing module necessary for the peer module and instructing the start of data processing, it is possible to execute predetermined processing.

By providing such a function, the description of an application that performs stream processing is greatly reduced. Because stream processing is highly real-time processing where it is important to process data at a specified bit rate, development of programs that realize such processing uses functions provided by the platform, such as time management. However, advanced scheduling technology is required such that a part of the CPU time is occupied and the flow of processing is not delayed. However, if the method of this embodiment is used, scheduling is performed by the stream manager. An application that uses a manager can create a peer module and link the link module, create the necessary data access module and data processing module, and link this to the peer module. The above scheduling function is implemented. Because there is no need to do.

In addition, since these modules only need to use an interface independent of the platform provided by the stream manager, the application can be used without changing on any platform.

FIG. 4 is a diagram showing a logical configuration of the peer module described with reference to FIG. The peer module (400) includes a data processing module (401) that executes data processing depending on the data format, and a data access module that executes data processing using a specific hardware device via the OS system interface (280). (405). The peer module is identified in the stream manager by a unique identifier (peer module ID) of the peer module.

The data processing module (401) is identified by a unique identifier (data processing module ID), and further includes a pre-processing unit (402) and a post-processing unit (403). Which peer module is associated with which data processing module is a column (180) in the table (1800).
3). The data access module (405) is also identified by a unique identifier (data access module ID), and includes an input / output execution processing unit (406) and an input / output completion processing unit (407). Which data access module is associated with which peer module is recorded in the column (1804) of the table (1800).

The peer module is connected to other peer modules by link modules (410, 420). The link module comprises a buffer matrix (411). Link module (
When 410) is connected to the input side of the data processing module, the data stored in the buffer in the link module can be output to the preprocessing unit of the data processing module. In the case of the output link module (420) connected to the output side of the data processing module,
Data stored in the buffer in the link module can be input from the post-processing unit of the data processing module.

For example, data processed by another peer module is written into a buffer and input to the input link module (410). This data is received by the preprocessing unit (402), converted into a data object (408) by the preprocessing unit, and placed in the data object matrix (412) waiting for execution. This is acquired and processed by the data access module (405). Details of the data object (408) will be described later with reference to FIG.

The data access module accesses the device driver (430) via the system interface (280) in the OS, and inputs / outputs data to / from the hardware device (440). The input / output execution processing unit (406) of the data access module performs processing on the data input / output request or the data passed from the preprocessing unit. Examples of the processing include data conversion processing such as an instruction for reading a file stored in the hardware device (440), an instruction for encryption processing in the hardware for encryption processing of data (408) in the data object, and the like. Can be considered. The processing result is acquired by the input / output completion processing unit (407), and input as a data object (409) to the post-processing waiting processing object matrix (413). The post-processing unit outputs the data in the data object extracted from the post-processing waiting process object matrix (413) to the output link module (402), and finishes the data processing of the peer module.

These operations are executed by the peer module scheduler (233) with reference to the table (1800) and the table (1900). The execution timing is executed according to the information described in the scheduling table (1100) of FIG. Details of FIG. 11 will be described later.

The peer module is connected to one or more input link modules and one or more output link modules, and two or more peer modules are connected via the link module to realize a series of data processing flows. be able to. This allows data from multiple peer modules to be collected in a single peer module to multiplex multiple data,
It is also possible to send a single data flow to several different peer modules. This can be used, for example, for separation of multiplexed data as shown in FIG. 1 or multiplexing of a plurality of data.

Input / output processing execution unit (406) and input / output completion processing unit (4) of the data access module
In 07), when the hardware devices provided in the platform are different, the implementation method can be changed. By modifying the data access module, a module in a higher layer than the data access module can execute processing without depending on the system interface.

FIG. 12 shows the structure of the data object. The data object (1200) includes a data list (1210), a command code (1212), and a command argument (1213). The data list further includes an address (1221) to the data and a size (1
222) are stored as a list. The data object holds data to be processed in a data list (1210), and further records a command (1211) and a parameter (1213) for the data. The command and data need not necessarily be recorded in pairs. The data may be empty and only the command may be stored, or conversely, only the data may be stored.

FIG. 11 shows the configuration of the schedule table in the peer module scheduler. The schedule table (1100) manages when which peer module the peer module scheduler executes. The table includes schedule time (1101) and peer module ID (1102) as items. Here, the schedule time in the table (1100) is a reserved time when the processing of the data access module and the data processing module associated with a certain peer module is executed next. In addition, when a plurality of processes must be performed, the schedule is to determine and record which process is to be performed in what order. Also, schedule execution or scheduling is
This refers to executing processing according to a predetermined schedule.

In the example of FIG. 11, for example, the peer module IDs are 10 and 1 at a time of 250 milliseconds.
It is shown in lines 1110, 1120 that three peer modules are executed. That means
There may be a plurality of peer modules to be executed at the same time. However, even if the schedule time is the same time, the actual process may be performed immediately after the completion of one process.

In the present embodiment, it is the peer module scheduler that executes processing according to the contents recorded in the schedule table. That is, in the above-described processing, the peer module scheduler sequentially performs necessary processing while referring to the table (1800), the table (1900), and the table (2000) at the above-described timing. That is, the processing of the data processing module and the data access module of each peer module is executed by the peer module scheduler. In other words, the processing of the peer module peer module scheduler, the pre-processing unit, the post-processing unit, the input / output execution processing unit, and the input / output completion processing unit are performed in one execution context such as a thread. is there.

The peer module scheduler, for example, periodically starts, refers to the table, and if there is a peer module with a schedule time earlier than the current time, performs processing of this peer module. Update the table (1100) if necessary,
Wait for the next startup.

  The execution procedure of the peer module will be described with reference to FIG.

First, the peer module scheduler refers to the column (2004) of the stream table (2000) to check whether there is a stream process that has already been started (step 600). If there is no stream process that has been started, the process waits until a new stream process is executed (step 601). If there is a stream process that has been started, the schedule table (1110) is referred to, and the process waits for the schedule time that is the earliest to be executed from the current time (step 603). The method of waiting for the time may be determined by periodically looking at the clock, may use a function that stops execution for a specified time, or if this is a platform that can use time interruption, use this. Also good. Immediately after the stream processing is started, the processing start time is not yet reserved in the schedule manager by the peer module, so the stream manager registers the current time as the scheduling time of the peer module that constitutes the stream processing. Make the first run happen. When the standby operation is completed, the peer module I of the schedule table (1100)
The peer module scheduler executes the preprocessing unit of the data processing module of the peer module described in D (1102) (step 604). In this execution process, the pre-processing unit checks whether data exists in the input link module, and if it exists, converts it into a data object and inputs it to the data object matrix (412) waiting for execution. Alternatively, in a peer module that is the starting point of stream processing, such as a peer module that reads data from FIG. 1 from a hard disk, a data read request is input to the matrix as a data object.

Next, the time when the own module should be scheduled next is calculated, and the time is registered in the peer module scheduler (step 605). Set the next schedule time to the pre-processing unit (4
02) is determined because the time required for processing such as the size of the data (411) obtained from the link module (410) can be calculated. For example, assume that a peer module that reads a file such as the peer module (100) in FIG. 1 has a bit rate of 1024 kilobytes per second and a data block size to be read at 256 kilobytes at a time. This designation is designated by the application using the data processing command command (580). In such a case, if the file reading peer is, the reading time interval can be determined by a value defined by (data block size read at one time) / (bit rate). For example, if the block size is 1024 kilobytes per second and the block size is 256 kilobytes, it can be understood that the specified bit rate can be realized if reading is performed every 0.25 seconds. That is, after the pre-processing unit makes a data request of 256 kilobytes to the input / output processing unit, the next processing should be performed after 0.25 seconds, so the next scheduling time is determined to be 250 milliseconds later. .

In the downstream peer module, the demultiplexer peer module and the video peer module in FIG. 10, these scheduling times are calculated from the bit rate set in the same manner and the size of the data obtained via the link module. It may be decided, but it is better to set the scheduling time to a relatively short time and execute it frequently.
This is because the uppermost file reading peer controls the bit rate of the data in the above-described manner, and in the lower peer module, the data is input to the input link object without particularly controlling the bit rate. If this is taken out and processed immediately, and output to the output link module, a predetermined bit rate can be realized. In this case, in the downstream peer object, it is better to check whether data is frequently input to the link object in order to prevent a processing delay from occurring and a bit rate from decreasing. For this purpose, a low scheduling time should be reserved.

When the data object is output from the preprocessing unit to the data access module (step 606), the peer module scheduler executes the input / output execution processing unit of the data access module to start the processing instructed by the data object. (Step 6
07) This corresponds to, for example, an operation for requesting a hardware device to process or acquire data.

If there is no data access module in the pending data object queue and no new data object has been output from the data access module to the post-processing pending data processing queue, the peer module scheduler proceeds to step 611 (step 608).

When the processing of the input / output execution processing unit that has been executed previously is completed and there is data as a processing result, for example, this is because the data processing or acquisition processing requested for the hardware device is completed, If the resulting data exists, the peer module scheduler executes the input / output completion processing unit of the data access module (step 609) and inputs the data as a data object to the post-processing waiting data object matrix (413). (Step 610). If the data object is in the post-processing data object queue (413), the peer module scheduler executes the post-processing unit and passes only the data list in the data object to the output link module. When the processing of the post-processing unit is completed, the peer module scheduler returns to the start state for the schedule of the next peer module.

In this execution method, for example, even when a plurality of stream processes are executed, the process of the peer module existing in each stream process can be performed in one execution context of the peer module scheduler. Here, the execution context is an execution unit having information necessary for executing a program, and a thread or the like corresponds to this. because,
By using the above-described method, the time for scheduling the processing of each peer module is specified by each peer module even in the case of a plurality of stream processing, so the peer module scheduler only needs to execute at that time. Because. When processing is performed in one execution context in this way, there is no need to perform synchronization processing that occurs when processing a plurality of stream processes in a plurality of execution contexts. Since the synchronization processing for exclusively accessing the device has a high load, the performance degradation of the stream processing can be prevented by omitting this processing.

Also, in this method, for example, if the execution priority is increased, if the reservation time of the peer module determined by the pre-processing unit of the peer module is shortened, the processing of the peer module is scheduled earlier. More CPU time will be spent processing the peer module. Further, when a plurality of processes are executed, it is possible to specify the priority of each process by adjusting the reservation time. For example, when performing two processes at the same time, the process of acquiring data from a certain device and recording it in the storage, and the process of acquiring data from a certain device and displaying it on the display, if the reservation time of the former process is set short Therefore, it is possible to set a higher priority for recording processing.

In addition, by executing multiple data processing in one execution context, it is not necessary to allocate multiple buffers to multiple execution contexts, and in multiple execution contexts such as semaphores used in parallel execution of multiple contexts When one resource is used, it is not necessary to use a synchronization function that is a mechanism for guaranteeing exclusive access, and it is possible to reduce resources used.

Further, by executing processing in one context, the peer module scheduler (233) can set efficient scheduling. This is because, in the conventional scheduling method, for example, the round robin method, the processing is unconditionally switched every fixed time. In some cases, real-time performance is reduced, and unnecessary CPU time is consumed. On the other hand, in this method, the time required for the next processing is calculated, and the required time is registered in the scheduling table and scheduling is performed. Therefore, unnecessary scheduling is not performed, and the time is allocated to the redundant time in the conventional round robin method or the like. Since it is possible to allocate CPU processing to other processing, the use efficiency of CPU time, which is one of the resources of the platform as a whole, is improved, and it is scheduled correctly at the time when processing is required. It becomes possible to improve the real-time property.

In FIG. 7, as an example of data processing using the peer module, link module, and peer module scheduler according to the present embodiment, a file in which audio and video are multiplexed and recorded is read from the hard disk and separated again. A data flow in the case of reproducing as a moving image from the display and the speaker will be described. As a method of recording audio and video multiplexed, there is MPEG (Motion Picture Expert Group) and the like. In this process, the file read peer module (700)
730) and the link module (720) and the demultiplexer peer module (
730) and the video peer module (760) are connected by a link module (740), and an audio peer module (780) is further connected by a link module (750).

First, the file reading peer module accesses the hard disk through the system interface (709) and the device driver (710) to acquire data. The demultiplexer peer module (730) demultiplexes the acquired data. The video data obtained by this processing is sent to the video peer module (760) and sent to the hardware video decoder (770) through the device driver (769), and the audio data is sent to the audio peer module (780) and the device. It is sent to the hardware audio decoder (790) through the driver (789) and reproduced.

FIG. 10 shows a detailed configuration of the processing example shown in FIG. The processing procedure will be described using this.

First, the peer module scheduler starts executing the data processing module of the file reading peer module. A pre-processing unit (702) of the file reading module which is the data processing module (701) is a data object which is a file reading input / output processing to the input / output execution processing unit (705) of the file access module which is the data access module (704). This is requested by inputting (707) into the input / output waiting data object matrix. This read processing is performed by the input / output processing unit (705) by the system interface (7).
09) and the storage device adapter device driver (710). Further, the peer module scheduler checks whether or not the reading process executed before the execution of the process is completed. If the process is completed, the input / output completion processing unit (706) is executed, and the data is stored as data. An object (708) is put into the post-processing data object queue.

Next, when the post-processing unit (703) is executed, the data object (708) stored in the post-processing waiting data object matrix is acquired, and the data list in the data object is stored in the buffer of the link module (720). To the queue.

In these processes, each process is searched as follows. Since the peer ID of the file read peer module is 100, the peer module scheduler determines from this value:
With reference to the table (1800), the data processing module ID assigned as the data processing module is obtained. In the example of FIG. Next, the table (15
00), a row with a data processing module ID of 100 is searched. As a result, the pre-processing unit to be executed is at address 0xFA800000, and the post-processing unit to be executed is 0xFA808.
Since it can be seen that it exists at address 000, these are executed. In this case, the content of the pre-processing unit stored at this address is the processing for designating the next scheduling time after instructing the reading of the specified file, and the content of the processing of the post-processing unit is acquired. This process is to output the processed data to the output link. Similarly, for the data access module, the table (1800) is referred to, and the data access module ID assigned as the data access module is obtained. In the example of FIG. Next, the table (1300
), A row having a data access module ID of 1 is searched. As a result, the address of the input / output execution processing unit to be executed and the address of the input / output completion processing unit are obtained and executed. In this case, the processing content of the input / output execution processing unit stored at this address is a data read request issuance to the hard disk, and the processing content of the input / output completion processing unit is data acquisition processing from the hard disk. .

Demultiplexer processing module (73) of demultiplexer peer module (730)
When 1) is executed by the peer module scheduler, the pre-processing unit (732) is executed, and the data list is retrieved from the queue (721) of the link module (720). If there is no data in the queue, the pre-processing unit ends the processing and proceeds to the processing of the post-processing unit.

Note that the processing in the pre-processing unit is the same in the processing of the peer module connected from the start point to the second and subsequent points in the data processing flow. Also, the pre-processing unit of all peer modules reserves the next time that it should be executed before ending the processing. This reserved time depends on the bit rate of the stream processing and the required real time property. For example, in the case of high bit rate data processing or processing that requires high real-time performance, the data must be processed quickly and passed to the next peer module, so the data is stored in the buffer of the input link module. It is necessary to check frequently. On the contrary, if the bit rate is low, the check can be made at a relatively large interval.

The processing of the demultiplexer peer module is a data format conversion, which can be executed by a program and does not need to access I / O hardware such as an HDD, so the data access module must define it. Rather, the data processed by the pre-processing unit is passed to the post-processing unit (733). In this case, the data format conversion process is as follows:
The preprocessing unit (732) of the multiplexer peer module analyzes the given data in advance,
The result is a process of writing the result into two audio buffers and a video buffer. The post-processing unit extracts the video data from the video buffer, and queues (741) of the link module (740).
Similarly, the audio data is extracted from the audio buffer and input to the queue (751) of the link module (750).

Next, when the preprocessing unit (762) of the video writing module (761), which is the data processing module of the video peer module (760), is executed, the data list waiting in the queue (741) is taken out. Application can control data processing module (
560) according to the bit rate determined by using the video decoder access module (76).
4), the data is output to the device driver (769) of the hardware video decoder (770) by the input / output execution processing unit (765), and output to the hardware video decoder (770). As a video. The data that has been output is input / output completion processing section of the video decoder access module (data access module)
766) and is placed in the data object queue waiting for termination processing as a data object (768). Since the video peer module is the end of the data processing and there is no output link module, the video data written in the device is discarded by the post-processing unit (763).

The processing in the audio peer module (780) is the same as the processing in the video peer described above. The preprocessing unit (782) of the voice writing module (781), which is the data processing module of the voice peer, is executed, and the waiting voice data list is extracted from the queue (751). In accordance with the bit rate of the extracted data, it is sent to the audio decoder access module (784), which is a data access module, and an input / output execution processing unit (78
The process is started by 5). The audio data is sent to the hardware audio decoder (790) through the device driver (789) and reproduced. The data that has been output is detected by the input / output completion processing unit (786) of the data access module, and the completed data is placed as a data object (788) in a post-processing waiting data object matrix,
Thereafter, it is discarded by the post-processing unit (783).

FIG. 8 shows an example in which the configuration of FIG. 2 is implemented in user space. When mounted in user space, all modules are mounted in user space, so that there is an effect that development is easy.
This is because development tools used when developing application programs, such as a debugger, can be used, and the memory management function provided by the OS can be used, so interference from other applications can be prevented, and reliability and availability can be improved. This is because it can be increased.

FIG. 9 shows an example in which most of the configuration of FIG. 2 is configured by an OS. For example, in the case of a configuration implemented in a kernel space, an OS in which modules in the OS execute preferentially over application programs has an advantage in performance over implementation in a user space. This is particularly effective when content processing is performed at a high bit rate that requires real-time performance.

In these two configurations, the general system interfaces (920, 931) and the data access module manager (932) are dependent on the platform, and other program modules belonging to the present embodiment do not depend on the platform. This is because the module absorbs differences in the function usage procedures and device access methods due to differences in hardware and OS, and provides a flat home by using the generalized interface provided by the module. It is because it is possible to use the function to do.

Which configuration is to be adopted should be determined in consideration of the application, hardware configuration, development man-hours, and the like. For example, when implementation on a platform with low CPU performance or high performance is required, debugging is somewhat difficult, but implementation in the kernel space of FIG. 9 is relatively easy to achieve high performance. Can be considered. On the other hand, when the CPU can sufficiently use execution resources such as a memory sufficiently quickly or when high performance is not required, it can be considered to have a configuration like the user space of FIG.

The present technology can be used for streaming processing such as moving image and audio processing. For example,
It can be applied to broadcasting equipment, receivers, recording / recording equipment, surveillance cameras, and the like.

Configuration diagram showing the processing flow of data processing configured as a data flow between processing points The block diagram which showed the whole structure of the element of a present Example Configuration diagram showing hardware configuration The block diagram which showed the definition of the processing point of a present Example The figure which showed the control interface of a present Example The flowchart which showed the processing step of the scheduling function of the processing point of a present Example Configuration outline diagram showing an example of a flow of data processing for presenting a video filter stored in a storage device and decoded using a hardware decoder Configuration diagram showing an example of implementation of the present embodiment for execution at the application level of the operating system Configuration diagram showing an example of implementation of this embodiment for execution at the kernel level of the operating system The block diagram which showed the example of the flow of the data processing for showing the video filter memorize | stored in the memory | storage device and decoded using the hardware decoder Figure showing the configuration of the table that manages the schedule time Diagram showing the structure of the data object The figure which showed the structure of the table which manages a response | compatibility with the address of data access module ID and data access module management information The figure which showed the structure of the table which manages a response | compatibility of data processing module ID and the address of data processing module management information Flow showing the procedure for building stream processing Diagram showing peer module configuration Diagram showing link module configuration Diagram showing stream processing configuration

Explanation of symbols

100 camera, 110 microphone, 120 step of retrieving data, 121
Points for accessing video data using a camera, 122 points for accessing audio data using a microphone, 130 steps for synchronizing and multiplexing video data and audio data, 131 single processing points, 140 steps for presenting data , 14
1 single processing point, 150 steps to record data, 160 hardware data decoder, 170 display, 180 speakers, 190 storage device

Claims (18)

Data processing method in data processing apparatus having data input device, data output device, storage unit storing each processing module, instruction issuing unit for issuing command, and arithmetic processing unit for executing processing using said processing module Because
Identifying at least one or more processes necessary to process the data based on a first instruction issued from the instruction issuing unit;
A processing module necessary for the specified processing is generated in the storage unit,
Connecting the processing modules using a link module, generating a processing flow for executing the processing in the storage unit,
Based on the second command issued from the command issuing unit, data input from the input device is input to the start of the processing flow, and the processing flow is executed by the arithmetic processing unit,
A data processing method, wherein a processing result obtained by executing the processing flow is output from the output device. A data processing method according to claim 1, wherein
The processing module includes: a data access module that inputs data from the input device or outputs data to the output device; and a data that has been processed after executing a predetermined process on the data input via the data access module And a data processing module for outputting the data to another processing module. A data processing method according to claim 2, comprising:
The data processing module receives data from the link module, and executes a predetermined process, and then outputs the post-execution data to the data access module;
A second for outputting the data input from the data access module to another processing module;
A data processing method. A data processing method according to claim 3, wherein
The data access module includes: an input / output execution processing unit that transmits an input / output request for data transmitted from the first processing unit to the input device or the output device; and the data acquired by the input / output request A data processing method comprising: an input / output completion processing unit for outputting to the processing unit. A data processing method according to claim 3, wherein
A data processing management module and a stream table that defines a processing order;
The first processing unit includes:
Based on the bit rate notified from the command issuing unit and the block size, which is a unit for processing in the first processing unit, calculates the time when the processing module is activated next and registers it in the stream table A data processing method characterized by the above. A data processing method according to claim 5, wherein
The data processing management module refers to the stream table,
A data processing method comprising: executing processing registered in the stream table in one execution context. A data processing device,
A data input device;
A data output device;
A storage module storing a processing module, a link module connecting the processing module, and a data processing management module;
An instruction issuing unit for issuing instructions;
An arithmetic processing unit that executes processing using the processing module;
The data processing management module includes:
Based on the first instruction issued from the instruction issuing unit, at least one process necessary for processing the data is specified, and a processing module required for the specified process is generated in the storage unit And
Connecting the processing modules using the link module, generating a processing flow for executing the processing in the storage unit;
Based on the second command issued from the command issuing unit, data is input from the input device, the processing flow is executed by the arithmetic processing unit, and a processing result obtained by executing the processing flow is sent to the output device. A data processing apparatus that outputs the data. The data processing apparatus according to claim 7, wherein
The processing module includes: a data access module that inputs data from the input device or outputs data to the output device; and a data that has been processed after executing a predetermined process on the data input via the data access module And a data processing module for outputting the data to another processing module. The data processing apparatus according to claim 8, wherein
The data processing module receives data from the link module, and executes a predetermined process, and then outputs the post-execution data to the data access module;
A second for outputting the data input from the data access module to another processing module;
A data processing apparatus. The data processing apparatus according to claim 9, wherein
The data access module includes: an input / output execution processing unit that transmits an input / output request for data transmitted from the first processing unit to the input device or the output device; and the data acquired by the input / output request And an input / output completion processing unit for outputting to the processing unit. The data processing apparatus according to claim 9, wherein
A data processing management module and a stream table that defines a processing order;
The first processing unit includes:
Based on the bit rate notified from the command issuing unit and the block size, which is a unit for processing in the first processing unit, calculates the time when the processing module is activated next and registers it in the stream table A data processing apparatus. A data processing apparatus according to claim 11, comprising:
The data processing management module refers to the stream table,
A data processing apparatus, wherein processing registered in the stream table is executed in one execution context. A data processing program in a computer having an instruction issuing unit for issuing an instruction, a data input device, and an output device,
Identifying at least one or more processes required to process the data based on a first instruction issued from the instruction issuing unit;
Generating a processing module necessary for the specified processing in the storage unit;
Connecting the processing modules using a link module and generating a processing flow for executing the processing in the storage unit;
Based on a second command issued from the command issuing unit, data input from the input device is input to the start of the processing flow, and the processing flow is executed by the arithmetic processing unit;
The data processing program for performing the process that the process result by which the said process flow was performed is output from the said output device. A data processing program according to claim 13,
The processing module includes: a data access module that inputs data from the input device or outputs data to the output device; and a data that has been processed after executing a predetermined process on the data input via the data access module And a data processing module for outputting the data to another processing module. A data processing program according to claim 14, wherein
The data processing module receives data from the link module, and executes a predetermined process, and then outputs the post-execution data to the data access module;
A second for outputting the data input from the data access module to another processing module;
A data processing program. A data processing program according to claim 15,
The data access module includes: an input / output execution processing unit that transmits an input / output request for data transmitted from the first processing unit to the input device or the output device; and the data acquired by the input / output request A data processing program comprising an input / output completion processing unit for outputting to the processing unit. A data processing program according to claim 15,
A data processing management module and a stream table that defines a processing order;
The first processing unit includes:
Based on the bit rate notified from the command issuing unit and the block size, which is a unit for processing in the first processing unit, calculates the time when the processing module is activated next and registers it in the stream table A data processing program. A data processing program according to claim 17,
The data processing management module refers to the stream table,
A data processing program for executing processing registered in the stream table in one execution context.

Images