JP2005236466A - Information processor and processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform both synchronous processing and asynchronous processing of a plurality of information signals smoothly without disturbing integrity. <P>SOLUTION: In a processing controller section 3, a processing time estimating section 371 estimates the required time of an interrupting asynchronous processing, and an idle time calculating section 372 calculates the idle time of each processor section. A searching section 373 searches a processor section having a longest idle time calculated at the idle time calculating section 372, and searches a processor section having a longest idle time. At the processor section having a longest idle time thus searched, a decision is made whether asynchronous processing requiring the estimated time can be executed or not and a processor for interrupting the asynchronous processing is determined. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention breaks down synchronization processing for a plurality of information signals such as simultaneous playback and simultaneous recording of a plurality of TV broadcast programs, multi-point video conference, and arbitrary viewpoint video playback using a multi-view video / audio stream. The present invention relates to an apparatus and a method for enabling interruption of asynchronous processing.

  For example, in Patent Document 1 described later, a plurality of TV signal input units, a plurality of recording units, a plurality of playback units, and a plurality of TV signal output units are provided, so that simultaneous recording and playback of a plurality of TV broadcast channels are performed. A technique for performing the above is disclosed. Thus, in the conventional broadcast receiving apparatus, in order to simultaneously view and record a plurality of programs, it is necessary to install a plurality of tuners, and since this hurdle was high, there was little practical feasibility. . Even in BS (Broadcasting Satellite) digital broadcasting and terrestrial digital broadcasting, there are restrictions on the number of tuners installed, and even though 100-channel broadcasting is not possible, simultaneous viewing and recording of 100 channels is not impossible, but can be used easily. It is not a thing.

  However, in streaming broadcasting using a network, this restriction does not exist, and simultaneous viewing / recording of a plurality of channels according to the network bandwidth can be realized. That is, since a stream is transmitted and received through a closed transmission path called a network, there is no need for a plurality of tuners for receiving and selecting a plurality of broadcast channels simultaneously. Currently, applications that perform real-time video and audio transmission using a network include a TV (Television) conference application and a streaming application.

  In general, in real-time video / audio transmission, RTP (real-time transport protocol) defined in RFC (request for comments) -1889 using UDP / IP (user datagram protocol / internet protocol) with low reliability is used. The packet loss handling process on the network is application dependent. The arrival interval of the real-time video / audio packet is affected by the transmission delay on the network, but the transmission delay on the network is not constant and includes jitter. Further, when congestion occurs on the network path, a packet loss that causes packet loss occurs. The recovery processing for packet loss is in a trade-off relationship with real-time characteristics, and the time required for recovery processing is very severe, and good response is required.

  In IP telephones and the like that have become widespread in recent years, the Y.S. standard in ITU-T (International Telecommunication Union Telecommunication Standardization Sector). The quality target value of the Internet protocol communication service defined by the title “Internet Protocol Communication Service-Performance Objectives” is adopted as 1541, and is also adopted by the TTC (Telecommunication Technology Committee) in Japan. Here, the end-to-end delay (one-way delay) is defined as 100 msec (milliseconds) or less for class A (same as a fixed telephone) and 150 msec or less for class B (same as a mobile phone). Yes.

  Considering a TV phone (TV conference) based on this rule, in the case of NTSC (National Television System Committee) video signal, the frame period is about 33 msec, and at least one frame delay in compression encoding and decoding. Therefore, 66 msec is used. Since a transmission delay in the network path (about several tens of msec for domestic ADSL (asymmetric digital subscriber line) lines) is added to this, it is clear that the processing time constraint for network packet processing is on the order of several msec.

  Here, the network bandwidth used by the current application is about several Mbps (megabits / second), about 2 Mbps at the maximum for video conferencing and about 3 Mbps at the maximum for streaming. In the video conference, there was a history of being an ISDN (integrated services digital network) main body, and it was centered on a low bit rate, but in recent years, a shift to a LAN (local area network) is being performed.

  While streaming is also mainly ADSL, it seems to be centered on several Mbps, but it is considered that multi-channeling will also progress with the spread of fiber to the home (FTTH). If a stream of about several channels is to be handled, for example, as in Patent Document 2 described later, a method of negotiating with a communication partner so as to use an available resource among hardware mounted resources is also conceivable. .

As described above, an environment has been established in which more real-time information such as video information and audio information is transmitted and used through various networks.
JP 2001-008144 A JP 2002-152307 A

  By the way, the increase in the bandwidth of the infrastructure is very fast, and it is expected that the available bandwidth of the network will be in the order of Gbps (gigabit / second) in a few years. In this case, simultaneous broadcasting of 100 channels is possible by streaming, and there is a possibility that a processing amount of 100 times or more as the current video / audio packet is generated. The transmitted video / audio data (the synchronized video data and audio data (so-called AV (Audio / Visual) data)) is usually compressed and encoded, and each received video / audio data is present. It is decoded every fixed unit (frame).

  In the decoding process in units of frames, there is a restriction called a decoding time stamp (DTS) and a restriction called a display time (PTS). If these constraint conditions are not satisfied, the reproduced video / audio is disturbed, and noise is added to the audio, or the video frame is skipped and the video is jerky. For example, when asynchronous interrupt processing that does not synchronize with a video / audio frame occurs frequently, there is a possibility that the time constraint such as decoding processing time / display time (DTS / PTS) cannot be maintained due to processing delays that occur in the processing.

  For example, as shown in FIG. 26, in each of the processing devices (1) to (n), a decoding process (decoding process) or an encoding process (encoding process) of compression encoding is performed within the frame period T. think of. Here, processing is started at the timing of the frame synchronization signal Fd, and display is performed at the timing of the next frame signal. Therefore, the decoding process and the encoding process have a time constraint that the process must be completed by the timing of the next frame signal.

  In FIG. 26, the processing device (1) performs MPEG (Moving Picture Experts Group) 2-HD decoding processing, and the processing device (2) performs MPEG2-SD decoding processing. ) Shows a case where the MPEG4 decoding process is performed, and the processing apparatus (n) performs the MPEG2 encoding process. FIG. 27 shows a time margin for the processing of each of the processing devices (1) to (n) shown in FIG. 26 as a free time in the frame. As shown in FIG. 27, the processing device (1) and the processing device (n) have little free time, and the processing device (2) and the processing device (3) have relatively much free time. You can see that it is in a state.

  FIG. 28 shows an example in which asynchronous packet processing from the network is performed by a statically associated processing device. The example of FIG. 28 shows a case where an asynchronous packet interrupt process is performed by the processing device (1) with a small free time. In this case, as shown in FIG. 28, in the second and third frame periods (frame periods), the decoding process cannot satisfy the time constraint condition, and the video decoding process fails. Is shown.

  In the case of the example shown in FIG. 28, it is assumed that the compression-coded video / audio packet from the network is a 4 Mbps IP packet. Here, if the IP packet is 1500 bytes, the number of packets per second is 4000000/1500/8 = 333.33 (pps: pulse per second), and if the frame period is 33 msec, it is about 10 on average. Video / audio packet processing of packets / frame (10 packets per frame) occurs.

  Network packets are truly asynchronous packets, and the amount of information is constantly increasing. For this reason, it is considered that the processing does not complete within the frame period T as described with reference to FIG. In this case, for example, as disclosed in Japanese Patent Application Laid-Open No. 11-177953, priority is given to a plurality of streams input from the network, and processing for all the streams is not in time, It is conceivable to use a technology that preferentially processes high priority items and prevents video quality degradation.

  In this case, it means that a stream with low priority cannot be processed with high quality. In particular, real-time video and audio packets are required to have high responsiveness from the viewpoint of recent quality emphasis and quality of service (QoS). I can't answer my demands enough.

  In view of the above, the present invention eliminates the above-mentioned problems, disturbs the consistency of both synchronous processing and asynchronous processing for a plurality of information signals while sufficiently answering the demands from the viewpoint of quality-oriented and QoS. An object of the present invention is to provide an apparatus and a method that enable smooth processing.

In order to solve the above problem, an information processing apparatus according to claim 1 is provided.
A plurality of processing device units capable of executing synchronous processing and asynchronous processing, and a processing control device unit for controlling each of the plurality of processing device units,
The processing control unit is
Estimating means for estimating the time required for the asynchronous processing when the asynchronous processing is interrupted to the processing unit;
Free time calculating means for calculating the free time of each processing unit,
Search means for searching for a processing device unit having the maximum free time based on free times of the plurality of processing device units from the free time calculating means;
And a determination unit configured to determine a processing device unit that interrupts the asynchronous process from a search result of the search unit and a required time estimated from the estimation unit.

  According to the information processing apparatus of the first aspect of the present invention, in the processing control unit, the time required for the asynchronous processing to be interrupted is estimated by the estimating unit, and each processing is performed by the free time calculating unit. The free time for each device unit is calculated. Then, the search unit searches for the processor unit having the longest free time calculated by the free time calculation unit. Then, in the processing device unit having the longest vacant time that has been searched, it is determined by the determining means whether the asynchronous processing that takes the estimated required time can be executed, and the processing device that interrupts the asynchronous processing is determined.

  As described above, since the processing control unit has a process scheduling function in each processing unit, it is possible to allocate a processing unit that is more suitable for interrupting asynchronous processing. Then, the asynchronous process can be interrupted and processed without breaking the synchronous process in each processing unit.

An information processing apparatus according to a second aspect of the present invention is the information processing apparatus according to the first aspect,
The synchronization processing executed in the processing unit is one or both of compression encoding processing and compression decoding processing.

  According to the information processing apparatus of the second aspect of the present invention, the synchronization processing performed in each processing unit is, for example, compression encoding processing and compression encoding processing for various data such as video data and audio data. One or both of them, the synchronization processing can be surely ended within a predetermined period without causing the compression encoding processing or the compression encoding processing to fail.

An information processing apparatus according to a third aspect of the present invention is the information processing apparatus according to the first or second aspect,
The plurality of processing units are connected by a ring-shaped bus.

  According to the information processing apparatus of the third aspect of the present invention, since each processing unit is connected by a ring-shaped bus, it is possible to execute pipeline processing of asynchronous processing and efficiently execute processing. Can be.

An information processing apparatus according to a fourth aspect of the present invention is the information processing apparatus according to the first, second, or third aspect,
The asynchronous processing is characterized in that one or both of video data and audio data is processed data.

  According to the information processing apparatus of the fourth aspect of the present invention, asynchronous processing using video data and audio data as processing target data is performed as interrupt processing. In this case, asynchronous processing for processing video data and audio data can be processed at high speed as interrupt processing, and synchronous processing executed in each processing unit does not fail.

An information processing apparatus according to claim 5 is the information processing apparatus according to claim 4,
The processing target data is provided using RTP (real-time transport protocol),
The processing control unit is
Dividing means for dividing the data to be processed into layers of RTP / UDP (user datagram protocol) / IP (internet protocol);
And distribution means for supplying the target data of each layer divided by the dividing means to different processing device units.

  According to the information processing apparatus of the fifth aspect of the invention, the processing target data is a video packet or an audio packet provided using the RTP protocol, and the processing target packet is RTP / UDP processed by the dividing means. / IP is divided into each layer, and the packet of each layer is supplied to a different processing unit by the distributing means so as to be processed.

  As a result, the time required for asynchronous processing can be estimated in detail, the processing can be subdivided, the load on the processing device unit can be distributed, and the asynchronous processing to be interrupted can be processed efficiently, and each processing device unit In this way, the synchronization processing performed in step 1 is not broken.

  According to the present invention, a scheduling function for performing asynchronous processing is provided, and a more suitable processing device can be assigned to perform asynchronous processing.

  In addition, when performing either compression encoding processing or compression decoding processing or a combination thereof as synchronous processing, it becomes possible to assign a more suitable processing device to asynchronous processing in order to comply with the display time constraint, and video This has the effect of preventing voice failures.

  Further, by connecting the processing device units with a ring-shaped bus, asynchronous processing pipeline processing can be performed, and asynchronous processing can be performed more efficiently. Then, when the synchronization processing is performed in each processing unit, the asynchronous processing can be interrupted and executed without breaking the synchronization processing.

  Even if the asynchronous processing is video data, audio data, or the like, an environment for processing them at high speed can be prepared. For example, video / audio packet processing input via the network can be processed at high speed as interrupt processing while keeping the time restriction of the synchronization processing.

  In addition, when performing either compression encoding processing or compression decoding processing, or a combination thereof as synchronization processing, for example, video / audio packet processing using RTP input via a network is performed while keeping the time constraints. In addition to processing at high speed as interrupt processing, RTP / UDP / IP packet processing is divided for each layer and processed by different processing units, thereby making it possible to perform more efficient control and processing.

  Hereinafter, an embodiment of an apparatus and a method according to the present invention will be described with reference to the drawings. In the embodiments described below, a case where the apparatus and method according to the present invention are applied to a device that uses simultaneous decoding of a plurality of compressed encoded video and audio streams as an application will be described as an example.

  Note that there are various applications such as simultaneous viewing / recording of a plurality of TV broadcast programs and multi-point TV conferences as applications for simultaneous decoding of a plurality of compressed encoded video / audio streams. In the embodiment, a case where the apparatus and method according to the present invention are applied to an arbitrary viewpoint video reproduction apparatus using a multi-view video / audio stream will be described as an example.

[Arbitrary viewpoint video multi-camera system]
First, an arbitrary viewpoint video multi-camera system (hereinafter simply referred to as a multi-camera system) that forms and transmits a multi-view video / audio stream used in an arbitrary viewpoint video playback apparatus to which an embodiment of the present invention is applied. ).

  1 and 2 are diagrams for explaining an example of a multi-camera system as a multi-view video / audio stream generation / transmission apparatus. As shown in FIG. 1, the multi-camera system of this example is configured by providing a plurality of cameras that are video / audio recorders for photographing a stadium (here, basket court Bct).

  In the case of this example, as shown in FIG. 1A, eight cameras 5 (1) to 5 (8) are provided so as to surround the circumference of the basket coat Bct, and as shown in FIG. 1B, the basket coat Bct is provided. Is provided with three cameras 5 (9) to 5 (11). Therefore, in the case of the multi-camera system of this example, the basket court Bct can be photographed simultaneously from 11 different points (viewpoints).

  As described above, each of the cameras 5 (1) to 5 (11) is a video / audio recorder, and can record not only video but also audio. They can be recorded at the same time and processed in synchronization.

  Then, the video signal and the audio signal recorded in each of the cameras 5 (1) to 5 (11) correspond to each of the cameras 5 (1) to 5 (11) as shown in FIG. The video / audio compression encoding units 6 (1) to 6 (11) provided are subjected to compression encoding processing.

  The video / audio stream including the video data and the audio data compression-encoded in each of the video / audio compression encoding units 6 (1) to 6 (11) is supplied to the multiplexing unit 7. The multiplexing unit 7 multiplexes the video / audio streams from the video / audio compression / encoding units 6 (1) to 6 (11) supplied thereto so as to be separable for each camera (specifically, IP packet) to form a multi-view video / audio stream, which is sent to the network.

  Note that as compression encoding methods used in the video / audio compression encoding units 6 (1) to 6 (11), for video, for example, MPEG2, MPEG4, H.264, and the like. H.264, MJPEG2000, etc. are conceivable. Examples of audio include MPEG2-AAC (MPEG2-advanced audio coding), AC3 (Dolby digital Audio code number 3), and ATRAC-3 (adaptive transform acoustic coding 3).

  Further, as a multiplexing method used in the multiplexing unit 7, there are MPEG2-TS (MPEG2-transport stream), MPEG2-PS (MPEG2-program stream), and the like. Further, as a transmission protocol on the network, for example, there is a protocol based on “RTP: A Transport Protocol for Real-Time Applications” defined in RFC1889, and when transmitting MPEG2 using RTP, “RTP Payload Format for MPEG1 / MPEG2 Video” defined in RFC2038 is used.

  Further, here, the case where the cameras 5 (1) to 5 (11) are used has been described in order to simplify the description. However, the number of cameras to be installed is not limited to this example, and any number can be provided.

  In this way, a multi-view video / audio stream (hereinafter simply referred to as a video / audio stream) in which video and audio simultaneously recorded by a plurality of cameras are multiplexed is distributed through the network, and an arbitrary-view video playback device to be described later. The video at an arbitrary viewpoint can be played back.

[Arbitrary viewpoint video playback system]
Next, an outline of the arbitrary viewpoint video reproduction method will be described. FIG. 3 is a diagram for explaining an arbitrary viewpoint video reproduction method. Here, for the sake of simplicity of explanation, as shown in FIG. 3, a case where a video / audio stream from a multi-camera system in which four cameras (A) to (D) are arranged in a grid is used will be described as an example. .

  As shown in FIG. 3, the camera position (installation position) of the camera (A) is the position (i, j), the camera position of the camera (B) is the position (i + 1, j), and the camera (C) Assume that the camera position is position (i + 1, j + 1), and the camera position of camera (D) is position (i, j + 1). An arbitrary viewpoint position is defined as a position (x, y).

As shown below, the arbitrary viewpoint video playback system can be broadly divided into:
(1) Use the camera image closest to the arbitrary viewpoint position (x, y).
(2) A video at an arbitrary viewpoint position (x, y) is synthesized from neighboring camera videos.
(3) Generate a stereoscopic image by a plurality of cameras.
There are three possible methods.

  In the case of (1), it is necessary to simultaneously decode a plurality of videos from the camera at the camera position that may be switched. In the case of (2), it is necessary to simultaneously decode a plurality of videos from a plurality of cameras at nearby camera positions for composition. In the case of (3), it is necessary to simultaneously decode videos from a plurality of cameras necessary for generating a stereoscopic video. Thus, it can be seen that in any of the methods (1) to (3), it is necessary to simultaneously decode a plurality of adjacent compressed and encoded video / audio streams.

  Any one of the above (1) to (3) is used, and a video when viewed from an arbitrary viewpoint position (x, y) is formed and reproduced and provided to the user. Then, the multi-view video / audio stream from the multi-camera system configured with many cameras as described with reference to FIGS. 1 and 2 is processed to reproduce the video when viewed from the arbitrary viewpoint position. In this case, the number of video data to be processed simultaneously increases depending on the number of cameras used, and it can be seen that the processing becomes enormous.

[Arbitrary viewpoint video playback device]
Next, an arbitrary viewpoint video reproduction apparatus according to this embodiment to which the apparatus and method according to the present invention are applied will be described. Here, for example, as described with reference to FIGS. 1 and 2, n video and audio streams from a multi-camera system including, for example, n (n is an integer of 2 or more) cameras are multiplexed. An arbitrary viewpoint video reproduction apparatus capable of processing a multi-view video / audio stream will be described as an example.

  FIG. 4 is a diagram for explaining an arbitrary viewpoint video reproduction apparatus according to this embodiment. As shown in FIG. 4, the arbitrary viewpoint video reproduction apparatus of this embodiment includes n processing device units 1 (1) to 1 (n) and a data bus (ring bus in this example) 2 connecting them. A processing control device unit 3 and a network interface 4 are provided.

  Here, the network interface 4 takes in the multi-view video / audio stream provided through the network and supplies it to the processing control unit 3. The processing control unit 3 controls each unit of the arbitrary viewpoint video reproduction device according to this embodiment, and includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. It is a configured microcomputer.

  Then, the processing control device unit 3 allocates a video / audio stream to be decoded to each of the processing device units 1 (1) to 1 (n), or each processing device unit 1 (1) to 1 (n). ), And a process of instructing a specific arbitrary viewpoint video reproduction method.

  Each of the processing device units 1 (1) to 1 (n) performs decoding (compression decoding) processing of the allocated video / audio stream and uses FIG. 3 according to control from the processing control device unit 3. Based on one of the described methods, video data at an arbitrary viewpoint is reproduced in consideration of video data after decoding processing from another processing unit.

  And the process control apparatus part 3 can control not only the synchronous process performed in each process apparatus part 1 (1) -1 (n) but an asynchronous process. FIG. 5 is a configuration example of control information that is formed and managed in the control processing unit 3 in order to perform synchronous processing and asynchronous processing in each of the processing units 1 (1) to 1 (n). It is a figure for demonstrating.

  In this embodiment, the synchronization processing performed in each of the processing device units 1 (1) to 1 (n) is a decoding processing of a video / audio stream assigned to the own device that needs to comply with frame synchronization. Yes, asynchronous processing corresponds to various other processing.

  In this embodiment, the control information includes management information (FIG. 5A), synchronous processing information (FIG. 5B (1), FIG. 5B (2)), asynchronous processing information (FIG. 5C (1), FIG. 5C). (2)). The management information (FIG. 5A) includes a processing device number, a situation, a pointer to synchronous processing information, and a pointer to asynchronous processing information. Which processing device unit is to execute what processing. Information for management.

  The synchronization process information (FIG. 5B (1), FIG. 5B (2)) includes a synchronization process number, a situation, a pointer to the next process, a processing device number, a required processing time, a completion restriction time, a scheduled start time, a start time, and a completion. This information includes a scheduled time and a completion time, and is information for managing the synchronization processing by the completion restriction time and managing the status of the synchronization processing. Asynchronous processing information (FIG. 5C (1), FIG. 5C (2)) includes an asynchronous processing number, a status, a pointer to the next processing, a processing device number, a reception time, a scheduled start time, a start time, a scheduled completion time, This is information for managing the status of asynchronous processing.

  However, as shown in FIG. 5, there is no room for managing execution of synchronous processing and asynchronous processing in each processing unit 1 (1) to 1 (n) using synchronous processing information and asynchronous processing information. If execution of asynchronous processing is assigned to a processing device unit that is executing synchronous processing, the synchronous processing in the processing device unit fails. Therefore, in the arbitrary viewpoint video reproduction device of this embodiment, the processing control device unit 3 performs the scheduling of the processing executed in each processing device unit 1 (1) to 1 (n) in detail, thereby synchronizing processing. To prevent bankruptcies.

[Configuration of Processing Control Unit 3]
FIG. 6 is a block diagram for explaining the processing control unit 3. As shown in FIG. 6, the processing control unit 3 includes a control unit 30 in which a CPU 31, a ROM 32, a RAM 33, and an EEPROM (Electrically Erasable Programmable ROM) are connected via a CPU bus 35, and each processing device The network interface 4 includes an interface (hereinafter abbreviated as I / F) 36 for communication with the units 1 (1) to 1 (n), a scheduling unit 37, and a clock circuit 38. Have been to.

  Then, as shown in FIG. 6, the scheduling unit 37 includes a processing time estimation unit 371 for estimating (predicting) a processing time in each processing device unit, and a free time calculation for calculating a free time in each processing device unit. And a search selection unit 373 that searches for and selects (determines) a processing device unit to which a target asynchronous process is assigned.

  In FIG. 6, the functions of the processing time estimation unit 371, the free time calculation unit 372, and the search selection unit 373 enclosed by double lines are realized by a program (software) executed by the CPU 31 of the control unit 30. It is also possible. That is, the function of the scheduling unit 37 can also be realized by a program executed by the CPU 31.

  Then, as will be described in detail below, the processing control device unit 3 estimates (predicts) the processing time of the processing when executing the asynchronous processing by the function of the scheduling unit 37, and each processing device unit 1 (1 ) To 1 (n) are calculated, a processing unit having sufficient free time to execute the target asynchronous processing is searched, and a processing unit that executes the asynchronous processing is selected ( Decision).

  Note that the video / audio stream is assigned to each processing unit by the function of the control unit 30. Of course, it is also possible to provide a processing unit that allocates the video / audio stream to each processing unit. The clock circuit 38 provides the current time.

  Then, in order to perform the arbitrary viewpoint video reproduction process by the function of the scheduling unit 37 as described above, the processing unit 1 (1) to 1 (n) performs the video / audio stream decoding process as the synchronization process. Even in this case, asynchronous processing can be interrupted and executed promptly and appropriately without breaking the synchronous processing in each processing unit.

  Hereinafter, the operation of the processing control unit 3 of the arbitrary viewpoint video reproduction apparatus of this embodiment will be described in detail. First, the synchronization processing will be described by taking an example of processing a video / audio stream in the MPEG2-PES (MPEG2-packetized elementary stream) format.

  In the video signal in the MPEG2-PES format video / audio stream, the decoding timing and the display timing of the video signal are controlled using DTS (Decode Time Stamp) and PTS (Presentation Time Stamp) included in the video signal. In MPEG2, when a B picture (Bidirectionally predictive picture) exists, DTS and PTS do not match. This is because it is necessary to replace frames for bidirectional prediction.

PTS is a display frame period, and at the display time indicated by PTS, it is required that decoding of the frame is completed and display is possible. In order to express this time constraint, the following variables are defined.
(A) Video frame data display time: PTS
(B) Frame decoding processing scheduled completion time: FDTS
(C) Frame decoding process start time: SDTS
(D) Current time: CTS.

  The estimated frame decoding processing completion time (FDTS) is estimated by statistically processing previous frame processing time information, macro block position information currently being processed, and the like. For example, a result obtained by multiplying the processing time information of previously processed frames having the same frame type by the number of unprocessed macro blocks and dividing by the total number of macro blocks in the frame is used. The calculation of the frame decoding process scheduled completion time (FDTS) is performed by the processing time estimation unit 371 shown in FIG.

  The video frame data display time (PTS) uses the PTS added to the video signal to be processed, and the processing control device unit 3 has the frame decoding processing start time (SDTS) and the current time (CTS). It can be obtained from the clock circuit 38. Acquisition of the video frame data display time (PTS), frame decoding processing start time (SDTS), and current time (CTS) is performed by the control unit 30 or the processing time estimation unit 371.

  Then, the time required for asynchronous processing (time required for asynchronous processing) can be obtained from the frame decoding processing start time (SDTS) and the frame decoding processing scheduled completion time (FDTS). The time required for the asynchronous processing is estimated in the processing time estimation unit.

The processing margin for the display time constraint can be expressed as a processing margin X as shown in the following (Equation 1).
Processing margin X = PTS−FDTS (Formula 1)
By this (Equation 1), the processing for obtaining the processing margin X is performed in the free time calculation unit 372 shown in FIG.

  As shown in (Equation 1), the asynchronous interrupt processing is performed by obtaining the difference between the determined video frame data display time (PTS) and the calculated frame decoding processing completion scheduled time (FDTS). The processing margin X can be obtained as the time (free time) that can be used. Therefore, it can be seen that the time available for asynchronous interrupt processing is less than or equal to the processing margin X.

  Then, when the n processing device units 1 (1) to 1 (n) are respectively performing frame decoding processing (compression decoding processing), the processing control device unit 3 is configured to process each processing device unit 1 (1 ) To 1 (n), processing margins X (1) to X (n) are calculated. The processing control unit 3 performs processing margins X (1) to X (n) obtained for each processing unit 1 (1) to 1 (n) and processing time for processing units that execute asynchronous processing by interruption. The selection is made based on the time required for the asynchronous process calculated based on the estimated frame decoding process completion time FDTS and the frame decoding process start time (SDTS) calculated by the estimation unit 371.

  Specifically, for example, the processing device unit having the largest processing margin at that time is searched, and if the processing margin X of the processing device unit is equal to or longer than the time required for asynchronous processing, the processing device unit is processed asynchronously. Select (determine) the processor unit to be interrupted. The search selection unit 373 performs a process of searching for a processing unit having the largest processing margin and selecting a processing unit having a sufficient processing margin to interrupt asynchronous processing.

  In this way, even if asynchronous processing is interrupted, it is possible to minimize the possibility that the synchronous processing already executed will fail.

  By the way, it is considered that the operation state of the processing unit is between the frame decoding process and the next frame decoding process, and may be in an idle state. In such a case, if the asynchronous interrupt process is easily executed, the restriction condition of the next frame decoding process may not be observed. Therefore, it is necessary to consider a similar margin in this case.

In this case, the next frame decoding scheduled start time: NSDTS is further defined as a variable. The value of NSDTS is a DTS accompanying the frame data to be decoded next. If the next frame data has not yet been acquired, it is determined based on the past DTS value interval or the like. In this case, the processing margin X (1) for the first processing unit can be obtained as in the following (Expression 2).
Processing margin X (1) = NSDTS (1) −CTS (Expression 2)
In this case, the time available for asynchronous processing (asynchronous interrupt processing) to be interrupted is equal to or less than the processing margin X (1). Processing margins X (2) to X (n) for the second to n-th processing device units can be obtained in the same manner.

Furthermore, if it is possible to estimate the frame decoding process completion scheduled time in the next frame that has not yet started decoding, it is possible to add the margin. The processing margin X (1) in this case can be obtained as in the following (Equation 3).
Processing margin X (1) = (NSDTS (1) −CTS)
+ (NPTS (1) -NFDTS (1)) (Formula 3)
In this case, the time available for asynchronous processing (asynchronous interrupt processing) to be interrupted is equal to or less than the processing margin X (1) obtained by (Equation 3).

  Note that NPTS and NFDTS in (Equation 3) mean PTS and FDTS for the next frame, NPTS is the video frame data display time of the next frame, and NFDTS is frame decoding in the next frame. The scheduled processing completion time is shown.

  Each normal video / audio packet constituting the video / audio stream has an IP packet length of about 1500 bytes in the case of an RTP packet, for example, and it is relatively easy to predict the processing time. For this reason, using the above-described (Equation 3), the asynchronous processing can be interrupted even to the processing unit that is between the frame decoding process and the next frame decoding process and is in the idle state. It can be determined whether or not it is possible.

  However, when exceptional processing occurs or when the packet to be processed is a variable-length packet, there are cases where the above-described mechanism cannot be used. A case where such exceptional processing occurs and a case where the packet to be processed is a variable-length packet will be described.

[When exceptional processing occurs]
For example, consider a case where packet loss occurs in the received RTP video stream. In this case, it is necessary to identify the lost packet and perform some kind of restoration process. For the restoration process, error correction by FEC (forward error correction), packet retransmission request, or the like can be considered. When such exceptional processing occurs, it is necessary to perform scheduling again at that time, and a processing device unit that executes the processing is selected using a processing margin.

  When different processing device units are used for restoration processing, it is necessary to exchange data between the processing device units. Therefore, it is necessary to consider the overhead of data transfer as a processing amount. For example, as shown in FIG. 4, when each processing apparatus is connected by a ring bus, the data transfer overhead to the adjacent processing apparatus section is smaller than that for other processing apparatuses. Therefore, the data transfer overhead is also used as a parameter when selecting the processor unit.

  In this way, even when packet loss occurs, the restoration process is taken into consideration, and as described above, by performing rescheduling of asynchronous processing for each processing device unit, a predetermined period of time can be obtained. It is possible to surely prevent the failure of the synchronization processing that needs to be completed within the system.

[For variable-length packets]
As described above, the packet of the video / audio stream received through the network may be a variable length packet. Therefore, the estimation error of the packet processing time can be a problem. For example, the difference in processing time is significant between a 64-byte packet and a 1500-byte packet.

  In this case, if the video / audio stream uses RTP as a protocol, RTP uses UDP / IP, so that the packet to be processed is IP (internet Protocol) / UDP (user datagram protocol) / RTP ( By dividing into processes for each layer, such as real-time transport protocol, more accurate control is possible by reducing the estimation error of processing time and making the scheduling control unit finer.

  Further, as shown in FIG. 7, the processing control device 4 allocates processing of each layer such as IP / UDP / RTP to each processing device unit 1 (1) to 1 (n), and a data bus (ring bus). ) 2 is configured to perform pipeline processing, it is possible to perform high-speed parallel processing by pipeline processing while minimizing the required program memory size for each processing unit 1 (1) to 1 (n).

  In the case of the example shown in FIG. 7, IP layer processing is assigned to the processing device unit 1 (1), UDP layer processing is assigned to the processing device unit 1 (2), and processing Processing is performed by assigning RTP layer processing to the device unit 1 (3), and performing high-speed parallel processing as processing in the processing device units 1 (1), 1 (2), and 1 (3) as pipeline processing. Even if the target packet is a variable-length packet, quick processing is possible.

[When each synchronization timing is different for each processing unit]
In some cases, the synchronization timings of the synchronization processes executed in the processing device units 1 (1) to 1 (n) are different from each other. Even in such a case, each processing device unit 1 (1) to 1 (n) that needs to be terminated within a predetermined period by scheduling asynchronous processing in consideration of synchronous processing in each processing device unit. It is possible to reliably prevent the synchronization processing in the system from failing.

  FIG. 8 is a diagram for explaining synchronization processing in the processing device units 1 (1), 1 (2), and 1 (3). As shown in FIG. 8, the synchronization process executed in the processing device 1 (2) is different in phase from the synchronization process executed in the processing device unit 1 (1). Further, as shown in FIG. 8, the synchronization processing executed in the processing device 1 (3) is different from the synchronization processing executed in the processing device units 1 (1) and 1 (2) in both phase and period. It is different.

  As described above, even if the synchronization processing executed in each of the processing device units 1 (1) to 1 (n) is different in phase and cycle between the processing devices, the processing control device unit 3 As described with reference to FIG. 4, since the operation states of the processing device units 1 (1) to 1 (n) are grasped, the operation states of the processing device units 1 (1) to 1 (n). Depending on the situation, it is possible to schedule appropriately.

[Configuration example of processing unit]
As described above, a configuration example of the processing device units 1 (1) to 1 (n) in which the processing control unit 3 manages the processing schedule of the synchronous processing and the asynchronous processing will be described. FIG. 9 is a block diagram for explaining a configuration example of the processing device unit 1.

  As shown in FIG. 9, the processing unit 1 of this embodiment includes a CPU 11, a ROM 12, a RAM 13, and an EEPROM 14 connected through a CPU bus 15, and a control unit 10 configured as a microcomputer, and a data bus 2. A communication interface (in FIG. 9, described as a communication I / F) 16 and an interface for transmitting / receiving various information to / from the processing control unit 3 (in FIG. 9, , I / F.), A video signal processing unit 18, a video signal output end 19, an audio signal processing unit 20, and an audio signal output end 21. Note that the functions of the video signal processing unit 18 and the audio signal processing unit 20 can also be realized by a program (software) executed by the CPU of the control unit 10.

  Then, as described above, the control unit 10 receives the supply of the video / audio stream and the control signal supplied through the processing control unit 3 through the I / F 17, and the control unit 10 receives the video signal processing unit 198 and the audio signal processing unit 20. Are controlled to perform video data decoding processing and audio data decoding processing so that video and audio can be reproduced.

  Then, the processing in each of the processing device units 1 (1) to 1 (n) is scheduled by the processing control device unit 3 so that the synchronous processing will not fail even if asynchronous processing is interrupted. Therefore, it is possible to realize a highly reliable arbitrary viewpoint video reproduction device including a plurality of processing device units 1 (1) to 1 (n).

  Note that processing for reproducing video at an arbitrary viewpoint based on the above-described arbitrary viewpoint video reproduction method is performed by the processing device units 1 (1) to 1 (n) in accordance with control of the processing control device unit 3. Any one of them may be performed by receiving a reproduction signal from each processing device unit, or may be performed by the processing control device unit 3 by receiving a reproduction signal from each processing device unit. it can. Alternatively, an arbitrary viewpoint video reproduction device unit that receives a reproduction signal from each processing device unit and reproduces an image at an arbitrary viewpoint may be provided.

[Summary of Operation of Processing Control Device 2]
Next, the operation of the processing control unit 3 of this embodiment will be summarized with reference to the flowchart of FIG. The process shown in FIG. 11 is a process mainly performed in the scheduling unit 37 under the control of the control unit 30 of the process control device unit 3 shown in FIG.

  The control unit 30 of the processing control device unit 3 starts the processing shown in FIG. 11 when, for example, a video / audio stream to be asynchronously processed is received through the network. First, the control unit 30 controls the processing time estimation unit 371 to calculate a frame decoding processing scheduled completion time (FDTS), and the frame decoding processing scheduled completion time (FDTS) and the frame decoding processing start time. The required time Ta for asynchronous packet processing is estimated from (SDTS) (step S1).

  Further, the control unit 30 controls the idle time calculation unit 372 to determine from the video frame data display time (PTS) of each processing device unit and the frame decoding processing scheduled completion time (FDTS) obtained in step S1. A processing margin (free time) X (i) for the i-th processing unit 1 (i) is calculated for the processing units 1 (1) to 1 (n) (step S2). Then, the control unit 30 controls the search selection unit 373 to search for the processing device unit Pm having the free time maximum value Max (X (i)) (step S3), and to search for the free time maximum value Max (X (i )) Determines whether or not it is longer than the required time Ta required for asynchronous packet processing (step S4).

  If it is determined in step S4 that the free time maximum value Max (X (i)) is greater than the required time Ta, the processing unit Pm having the free time maximum value Max (X (i)) is executed. (Step S5), and the free time for the processing unit Pm is updated (Step S6). That is, the control unit 30 of the processing control unit 3 manages the management information of the processing unit Pm that interrupted asynchronous processing among the management information of the processing units 1 (1) to 1 (n) managed by itself. Update.

  Then, the control unit 30 determines whether or not processing is performed for each layer, that is, whether or not processing is performed for each layer of IP / UDP / RTP (step S7). If it is determined in step S7 that the processing is not for each layer, the processing shown in FIG. 11 is terminated.

  When it is determined in step S7 that the processing for each layer is performed, the control unit 30 determines whether the processing for all layers is completed (step S8), and the processing for all layers is performed. When it is determined that the process is not completed, the process from step S1 is repeated. If it is determined in step S8 that all layers have been processed, the processing shown in FIG. 11 is terminated.

  Further, in the determination process of step S4, when it is determined that the free time maximum value Max (X (i)) is not greater than the required time Ta required for asynchronous packet processing, it is determined whether or not the process is for each layer (step S4). Step S9). If it is determined in step S9 that the process is for each layer, the processor unit having sufficient free time maximum value Max (X (i)) for interrupting asynchronous processing even if the processing is broken down into layers. 11 does not exist, a process when there is no compatible processing unit, for example, a message is displayed (step S10), and the process shown in FIG. 11 is terminated.

  If it is determined in step S9 that the processing is not for each layer, the control unit 30 separates the processing target packet into each layer of IP / UDP / RTP and performs the processing for each layer. (Step S11), the process from Step S1 is repeated.

  As a result, as described above, the processing device unit that interrupts the asynchronous processing is accurately selected without causing the synchronous processing in each of the plurality of processing device units 1 (1) to 1 (n) to fail. In addition, asynchronous processing of interrupts can be processed reliably and efficiently.

  The division of each layer is performed by the processing device unit 30, and the distribution of the divided layers to the respective processing device units may be performed by the control unit 30 of the control unit 30 controlling the I / F 36. . Of course, a division processing unit that performs division processing of each layer and a distribution control unit that performs distribution control may be provided.

  Further, in the case of the arbitrary viewpoint video reproduction device described above, the processing device units 1 (1) to 1 (n) have been described as performing the video / audio stream decoding processing. However, it is not limited to this. The present invention can also be applied to an apparatus for encoding video / audio data. That is, in the multi-camera system shown in FIG. 1 and FIG. 2, either the video / audio compression / encoding unit 6 functions as a processing control device unit or a separate processing control device unit is provided. As a result, asynchronous processing can be appropriately interrupted without causing the synchronous processing to fail.

  In other words, as the synchronization process, there may be one or both of a video / audio decoding (compression decoding) process and an encoding (compression encoding) process. In addition to the processing related to the video signal and the audio signal, various processing that must be performed in synchronization with other data processing is handled as the synchronization processing in the above case. Of course you can.

  Therefore, the present invention can be applied not only to an information processing apparatus that processes video and audio but also to various information processing apparatuses that process various data. For example, the present invention can be applied to personal computers, various game machines, and other various information processing apparatuses.

[Other configuration examples of information processing apparatus]
Recently, grid computing has attracted attention. Grid computing is a technology that realizes high computing performance by cooperating a plurality of information processing devices connected to a network. A multiprocessor system may be used in various information processing apparatuses that are formed using grid computing technology and are used in a network system that performs distributed processing among a plurality of information processing apparatuses. The present invention can be applied to an information processing apparatus or a network system formed by using such an information processing apparatus.

[Basic configuration of network system and information processing apparatus: FIGS. 11 to 14]
FIG. 11 shows an example of a network system configured by providing a plurality of information processing apparatuses. A plurality of information processing apparatuses 100, 200, 300, and 400 are connected via a network 90.

(Information processing apparatus and information processing controller)
Each of the information processing apparatuses 100, 200, 300, and 400 can be configured as various AV (Audio and Visual) devices and portable devices as described later, and is used as the multi-viewpoint video playback device according to the above-described embodiment. It can be configured. In addition, a network system constituted by a plurality of information processing devices can be configured as a multi-viewpoint video playback device.

  As will be described below, each of the information processing apparatuses 100, 200, 300, and 400 includes a main processor 121 as the processing control apparatus unit 3 and a plurality of sub-processors 123 as the processing apparatus unit 1. Is.

  As for the information processing apparatus 100, the information processing apparatus 100 includes an information processing controller 111 as a computer function unit. The information processing controller 111 includes a main processor 121-1, sub-processors 123-1, 123-2, 123-3, a DMAC (direct memory access controller) 125-1 and a DC (disk controller) 127-1.

  The main processor 121-1 performs schedule management of program execution (data processing) by the sub-processors 123-1, 123-2, and 123-3, and general management of the information processing controller 111 (information processing apparatus 100). . However, the main processor 121-1 can be configured such that a program other than the management program operates. In that case, the main processor 121-1 also functions as a sub processor. The main processor 121-1 has an LS (local storage) 122-1.

  There may be one sub-processor, but preferably there are a plurality of sub-processors. This example is a plurality of cases.

  Each of the sub-processors 123-1, 123-2, and 123-3 executes a program and processes data in parallel and independently under the control of the main processor 121-1. Further, in some cases, the program in the main processor 121-1 can be configured to operate in cooperation with the programs in the sub-processors 123-1, 123-2, and 123-3. A function program described later is also a program that operates in the main processor 121-1. Each of the sub-processors 123-1, 123-2, and 123-3 also includes LS (local storage) 124-1, 124-2, and 124-3.

  The DMAC 125-1 accesses a program and data stored in a main memory 126-1 including a DRAM (dynamic RAM) connected to the information processing controller 111, and the DC 127-1 is used for the information processing controller 111. The external recording units 128-1 and 128-2 connected to are accessed.

  The external recording units 128-1 and 128-2 may be fixed disks (hard disks) or removable disks, and optical disks such as MO, CD ± RW, DVD ± RW, memory disks, SRAM (static RAM), ROM, and the like. Various types can be used. Therefore, the DC 127-1 is called a disk controller, but is an external recording unit controller.

  As in the example of FIG. 11, the information processing controller 111 can be configured so that a plurality of external recording units 128 can be connected to the information processing controller 111.

  The main processor 121-1, each of the sub processors 123-1, 123-2, 123-3, the DMAC 125-1 and the DC 127-1 are connected by a bus 129-1.

  An identifier that can uniquely identify the information processing apparatus 100 including the information processing controller 111 throughout the entire network is assigned to the information processing controller 111 as an information processing apparatus ID.

  Similarly, identifiers that can identify the main processor 121-1 and the sub-processors 123-1, 123-2, and 123-3 are assigned as main processor IDs and sub-processor IDs.

  The information processing controller 111 is preferably configured as a one-chip IC (integrated circuit).

  The other information processing apparatuses 200, 300, and 400 are configured similarly. Here, even if the unit having the same parent number has a different branch number, the same function is assumed unless otherwise noted. Further, in the following description, when the branch number is omitted, it is assumed that the difference in the branch number does not occur.

(Access to main memory from each sub processor)
As described above, each sub-processor 123 in one information processing controller independently executes a program and processes data, but different sub-processors simultaneously read or write to the same area in the main memory 126. If done, data inconsistencies can occur. Therefore, access from the sub-processor 123 to the main memory 126 is performed according to the following procedure.

  As shown in FIG. 12A, the main memory 126 is configured by memory locations that can specify a plurality of addresses. Each memory location is allocated an additional segment for storing information indicating the state of the data. The additional segment includes an F / E bit, a sub processor ID, and an LS address (local storage address). Each memory location is also assigned an access key to be described later. The F / E bit is defined as follows.

  The F / E bit = 0 indicates that data being processed being read by the sub-processor 123 or invalid data that is not the latest data because it is empty, and cannot be read. The F / E bit = 0 indicates that data can be written to the memory location, and is set to 1 after writing.

  The F / E bit = 1 indicates that the data at the memory location has not been read by the sub-processor 123 and is the latest unprocessed data. The data in the memory location can be read and set to 0 after being read by the sub-processor 123. Further, the F / E bit = 1 indicates that the memory location cannot write data.

  Furthermore, it is possible to set a read reservation for the memory location in the state where the F / E bit = 0 (read disabled / write enabled). When a read reservation is made for a memory location with the F / E bit = 0, the sub processor 123 sets the sub processor ID and LS of the sub processor 123 as read reservation information in an additional segment of the memory location where the read reservation is made. Write the address.

  Thereafter, when data is written to the memory location reserved for reading by the sub-processor 123 on the data writing side and the F / E bit is set to 1 (readable / not writable), it is added to the additional segment as read reservation information in advance. Read to the written sub-processor ID and LS address.

  When it is necessary to process data in multiple stages by a plurality of sub-processors, the sub-processor 123 that performs the process in the previous stage controls the read / write of the data in each memory location in this way. Immediately after the data is written to a predetermined address on the main memory 126, another sub-processor 123 that performs the subsequent processing can read the data after the preprocessing.

  As shown in FIG. 12B, the LS 124 in each sub-processor 123 is also configured by memory locations that can specify a plurality of addresses. An additional segment is similarly allocated for each memory location. The additional segment includes a busy bit.

  When the sub-processor 123 reads the data in the main memory 126 to the memory location of its own LS 124, it reserves by setting the corresponding busy bit to 1. No other data can be stored in the memory location where the busy bit is 1. After reading to the memory location of the LS 124, the busy bit becomes 0 and can be used for any purpose.

  As shown in FIG. 12A, the main memory 126 connected to each information processing controller further includes a plurality of sandboxes. The sandbox defines an area in the main memory 126, and each sandbox is assigned to each sub processor 123 and can be used exclusively by the sub processor. That is, each sub-processor 123 can use the sandbox assigned to itself, but cannot access data beyond this area.

  The main memory 126 is composed of a plurality of memory locations, and the sandbox is a collection of these memory locations.

  Further, in order to realize exclusive control of the main memory 126, a key management table as shown in FIG. 12C is used. The key management table is stored in a relatively high-speed memory such as SRAM in the information processing controller, and is associated with the DMAC 125. Each entry in the key management table includes a sub processor ID, a sub processor key, and a key mask.

  The process when the sub processor 123 uses the main memory 126 is as follows. First, the sub processor 123 outputs a read or write command to the DMAC 125. This command includes its own sub-processor ID and the address of the main memory 126 that is the use request destination.

  Before executing this command, the DMAC 125 refers to the key management table and checks the sub processor key of the sub processor of the use request source. Next, the DMAC 125 compares the examined sub-processor key of the use request source with the access key allocated to the memory location shown in FIG. Execute the above command only when two keys match.

  In the key mask on the key management table shown in FIG. 12C, when the arbitrary bit becomes 1, the corresponding bit of the sub-processor key associated with the key mask may become 0 or 1. it can.

  For example, assume that the sub-processor key is 1010. Normally, this sub-processor key only allows access to a sandbox with 1010 access keys. However, if the key mask associated with this sub-processor key is set to 0001, the match determination between the sub-processor key and the access key is masked only for the digit whose key mask bit is set to 1. This sub-processor key 1010 enables access to a sandbox having an access key whose access key is either 1010 or 1011.

  As described above, the sandbox exclusivity of the main memory 126 is realized. That is, when it is necessary to process data in multiple stages by a plurality of sub-processors in one information processing controller, by configuring as described above, the sub-processor that performs the process in the previous stage and the process in the subsequent stage are processed. Only the executing sub processor can access a predetermined address of the main memory 126, and data can be protected.

  For example, it can be used as follows. First, immediately after the information processing apparatus is activated, the values of the key masks are all zero. It is assumed that a program in the main processor is executed and operates in cooperation with a program in the sub processor. When the processing result data output by the first sub-processor is temporarily stored in the main memory and desired to be input to the second sub-processor, the corresponding main memory area must naturally be accessible from either sub-processor. is there. In such a case, the program in the main processor appropriately changes the value of the key mask and provides a main memory area that can be accessed from a plurality of sub processors, thereby enabling multi-stage processing by the sub processors. .

More specifically, it is a multi-step process in the order of data from another information processing apparatus → processing by the first sub processor → first main memory area → processing by the second sub processor → second main memory area. When processing is done,
Sub-processor key of the first sub-processor: 0100
First main memory area access key: 0100,
Sub-processor key of the second sub-processor: 0101,
Access key for second main memory area: 0101
If the setting is kept as described above, the second sub-processor cannot access the first main memory area. Therefore, by setting the key mask of the second sub processor to 0001, it is possible to allow the second sub processor to access the first main memory area.

(Software cell generation and configuration)
In the network system in FIG. 11, software cells are transmitted between the information processing apparatuses 100, 200, 300, and 400 for distributed processing between the information processing apparatuses 100, 200, 300, and 400. That is, the main processor 121 included in the information processing controller in a certain information processing apparatus generates a software cell including a command, a program, and data, and transmits it to another information processing apparatus via the network 90 to perform processing. Can be dispersed.

  FIG. 13 shows an example of the configuration of the software cell. The software cell in this example is composed of a transmission source ID, a transmission destination ID, a response destination ID, a cell interface, a DMA command, a program, and data as a whole.

  The transmission source ID includes the network address of the information processing device that is the transmission source of the software cell, the information processing device ID of the information processing controller in the device, and the main processor 121 included in the information processing controller in the information processing device, and The identifier (main processor ID and sub processor ID) of each sub processor 123 is included.

  The transmission destination ID and the response destination ID include the same information about the information processing device that is the transmission destination of the software cell and the information processing device that is the response destination of the execution result of the software cell, respectively.

  The cell interface is information necessary for using the software cell, and includes a global ID, necessary sub-processor information, a sandbox size, and a previous software cell ID.

  The global ID can uniquely identify the software cell throughout the network, and is created based on the transmission source ID and the date and time (date and time) of creation or transmission of the software cell.

  In the necessary sub-processor information, the number of sub-processors necessary for executing the software cell is set. The sandbox size is set with the amount of memory in the main memory 126 and the LS 124 of the sub processor 123 necessary for executing the software cell.

  The previous software cell ID is an identifier of the previous software cell in a group of software cells that request sequential execution of streaming data or the like.

  The execution section of the software cell is composed of DMA commands, programs and data. The DMA command includes a series of DMA commands necessary for starting the program, and the program includes a sub processor program executed by the sub processor 123. The data here is data processed by a program including the sub processor program.

  Further, the DMA command includes a load command, a kick command, a function program execution command, a status request command, and a status return command.

  The load command is a command for loading information in the main memory 126 to the LS 124 in the sub processor 123, and includes a main memory address, a sub processor ID, and an LS address in addition to the load command itself. The main memory address indicates an address of a predetermined area in the main memory 126 from which information is loaded. The sub processor ID and the LS address indicate the identifier of the sub processor 123 to which the information is loaded and the address of the LS 124.

  The kick command is a command for starting execution of the program, and includes a sub processor ID and a program counter in addition to the kick command itself. The sub processor ID identifies the sub processor 123 to be kicked, and the program counter gives an address for the program execution program counter.

  As will be described later, the function program execution command is a command for requesting execution of a function program from another information processing apparatus to another information processing apparatus. The information processing controller in the information processing apparatus that has received the function program execution command identifies a function program to be activated by a function program ID described later.

  The status request command is a command for requesting transmission of device information related to the current operation state (situation) of the information processing device indicated by the transmission destination ID to the information processing device indicated by the response destination ID. Although the function program will be described later, it is a program categorized into the function program in the software configuration diagram stored in the main memory 126 of the information processing controller shown in FIG. The function program is loaded into the main memory 126 and executed by the main processor 121.

  The status reply command is a command in which the information processing apparatus that has received the status request command responds to the information processing apparatus indicated by the response destination ID included in the status request command with its own apparatus information. The status reply command stores device information in the data area of the execution section.

  FIG. 14 shows the structure of the data area of the software cell when the DMA command is a status return command.

  The information processing device ID is an identifier for identifying the information processing device including the information processing controller, and indicates the ID of the information processing device that transmits the status reply command. The information processing apparatus ID is included in the information processing controller in the information processing apparatus by the main processor 121 included in the information processing controller in the information processing apparatus when the power is turned on. The number of sub processors 123 to be generated is generated.

  The information processing device type ID includes a value representing the characteristics of the information processing device. The characteristics of the information processing apparatus include, for example, a hard disk recorder, a PDA (Personal Digital Assistant), a portable CD (Compact Disc) player, and the like, which will be described later. The information processing device type ID may represent a function of the information processing device such as video / audio recording or video / audio reproduction. It is assumed that values representing the characteristics and functions of the information processing apparatus are determined in advance, and it is possible to grasp the characteristics and functions of the information processing apparatus by reading the information processing apparatus type ID.

  The MS (master / slave) status indicates whether the information processing apparatus is operating as a master apparatus or a slave apparatus, as will be described later. When this is set to 0, it operates as a master apparatus. If it is set to 1, it indicates that it is operating as a slave device.

  The main processor operating frequency represents the operating frequency of the main processor 121 in the information processing controller. The main processor usage rate represents the usage rate in the main processor 121 for all programs currently running on the main processor 121. The main processor usage rate is a value representing the ratio of the processing capacity in use to the total processing capacity of the target main processor. For example, the main processor usage rate is calculated by using MIPS, which is a unit for evaluating the processor processing capacity, or per unit time. Calculated based on processor usage time. The same applies to the sub-processor usage rate described later.

  The number of sub-processors represents the number of sub-processors 123 included in the information processing controller. The sub processor ID is an identifier for identifying each sub processor 123 in the information processing controller.

  The sub processor status represents the state of each sub processor 123, and there are states such as “unused”, “reserved”, and “busy”. “unused” indicates that the sub-processor is not currently used and is not reserved for use. “reserved” indicates a reserved state that is not currently used. Busy indicates that it is currently in use.

  The sub-processor usage rate represents the usage rate of the sub-processor for a program that is currently being executed by the sub-processor or that is reserved for execution by the sub-processor. That is, the sub processor usage rate indicates the current usage rate when the sub processor status is busy, and indicates the estimated usage rate that is to be used later when the sub processor status is reserved.

  One set of sub processor ID, sub processor status, and sub processor usage rate is set for one sub processor 123, and the number of sets corresponding to the sub processor 123 in one information processing controller is set.

  The main memory total capacity and the main memory usage represent the total capacity and the capacity currently in use of the main memory 126 connected to the information processing controller, respectively.

  The number of external recording units represents the number of external recording units 128 connected to the information processing controller. The external recording unit ID is information for uniquely identifying the external recording unit 128 connected to the information processing controller. The external recording unit type ID represents the type of the external recording unit (for example, hard disk, CD ± RW, DVD ± RW, memory disk, SRAM, ROM, etc.).

  The external recording unit total capacity and the external recording unit usage amount represent the total capacity and the currently used capacity of the external recording unit 128 identified by the external recording unit ID, respectively.

  The external recording unit ID, the external recording unit type ID, the external recording unit total capacity, and the external recording unit usage amount are set for one external recording unit 128 and connected to the information processing controller. The number of external recording units 128 is set to the number of sets. That is, when a plurality of external recording units are connected to one information processing controller, a different external recording unit ID is assigned to each external recording unit, the external recording unit type ID, the external recording unit total capacity, and the external recording unit Department usage is also managed separately.

(Execute software cell)
The main processor 121 included in the information processing controller in a certain information processing apparatus generates a software cell having the above configuration and transmits the software cell to another information processing apparatus and the information processing controller in the apparatus via the network 90. . The transmission source information processing device, the transmission destination information processing device, the response destination information processing device, and the information processing controller in each device are identified by the transmission source ID, the transmission destination ID, and the response destination ID, respectively. .

  The main processor 121 included in the information processing controller in the information processing apparatus that has received the software cell stores the software cell in the main memory 126. Further, the transmission destination main processor 121 reads the software cell and processes the DMA command included therein.

  Specifically, the transmission destination main processor 121 first executes a load command. As a result, information is loaded from the main memory address instructed by the load command into a predetermined area of the LS 124 in the sub processor specified by the sub processor ID and the LS address included in the load command. The information loaded here is a sub-processor program or data included in the received software cell, or other designated data.

  Next, the main processor 121 outputs the kick command together with the program counter included in the kick command to the sub processor indicated by the sub processor ID included therein.

  The instructed sub processor executes the sub processor program according to the kick command and the program counter. Then, after the execution result is stored in the main memory 126, the main processor 121 is notified that the execution has been completed.

  Note that the processor that executes the software cell in the information processing controller in the information processing apparatus of the transmission destination is not limited to the sub-processor 123, and the main processor 121 stores a main memory program such as a function program included in the software cell. It can also be specified to execute.

  In this case, the transmission source information processing apparatus includes a main memory program and data processed by the main memory program instead of the sub processor program, and the DMA command is sent to the transmission destination information processing apparatus. A software cell as a load command is transmitted, and the main memory 126 stores the main memory program and data processed thereby. Next, the transmission source information processing apparatus identifies the main processor ID, the main memory address, and the main memory program for the information processing controller in the transmission destination information processing apparatus for the transmission destination information processing apparatus. A software cell that includes an identifier such as a function program ID (to be described later) and a program counter and whose DMA command is a kick command or a function program execution command is transmitted to cause the main processor 121 to execute the main memory program.

  As described above, in the network system of the present invention, the transmission source information processing apparatus transmits the sub processor program or the main memory program to the transmission destination information processing apparatus by the software cell, and transmits the sub processor program to the transmission destination. It is possible to load the sub processor 123 included in the information processing controller in the information processing apparatus, and cause the transmission destination information processing apparatus to execute the sub processor program or the main memory program.

  When the program included in the received software cell is a sub processor program, the information processing controller in the transmission destination information processing apparatus loads the sub processor program to the designated sub processor. Then, the sub processor program or the main memory program included in the software cell is executed.

  Therefore, even if the user does not operate the transmission destination information processing apparatus, the sub processor program or the main memory program can be automatically executed by the information processing controller in the transmission destination information processing apparatus.

  In this way, when the information processing controller in its own device does not have a main memory program such as a sub processor program or a function program, the information processing device can receive information from other information processing devices connected to the network. Can be obtained. Furthermore, data transfer is performed between each sub-processor by the DMA method, and the above-described sandbox is used, so that even when it is necessary to process data in multiple stages within one information processing controller, high speed and high security are achieved. The process can be executed.

[Distributed processing as a network system: FIGS. 15 to 25]
As a result of distributed processing using software cells, a plurality of information processing apparatuses 100, 200, 300, and 400 connected to the network 90 as shown in the upper part of FIG. It operates as a single information processing apparatus 700. However, for this purpose, the following processing needs to be executed by the following configuration.

(System software configuration and program loading)
FIG. 16 shows the configuration of software stored in the main memory 126 of each information processing controller. These software (programs) are recorded in the external recording unit 128 connected to the information processing controller before the information processing apparatus is turned on.

  Each program is categorized into a control program, a function program, and a device driver by function or feature.

  The control program is the same for each information processing controller, and is executed by the main processor 121 of each information processing controller, and includes an MS (master / slave) manager and a capability exchange program described later.

  The function program is executed by the main processor 121, and is provided for each information processing controller, such as for recording, for reproduction, or for material search, according to the information processing apparatus.

  Device drivers are used for input / output (transmission / reception) of an information processing controller (information processing device), such as broadcast reception, monitor output, bit stream input / output, network input / output, etc. Provided.

  When the information processing apparatus is physically connected to the network 90 by plugging in a cable or the like, the main power supply is turned on, and the information processing apparatus is electrically and functionally connected to the network 90. The main processor 121 of the information processing controller of the information processing apparatus loads each program belonging to the control program and each program belonging to the device driver into the main memory 126.

  As a load procedure, the main processor 121 first reads a program from the external recording unit 128 by causing the DC 127 to execute a read command, and then causes the DMAC 125 to execute a write command to load the program into the main memory 126. Write to.

  As for each program belonging to the function program, it may be configured to load only the necessary program when necessary, or like the programs belonging to other categories, each program is loaded immediately after the main power is turned on. You may comprise as follows.

  Here, each program belonging to the function program does not need to be recorded in the external recording unit 128 of all information processing apparatuses connected to the network, but is recorded in the external recording unit 128 of any one information processing apparatus. If so, it can be loaded from another information processing apparatus by the above-described method. As a result, the function program is executed as one virtual information processing apparatus 700 as shown in the lower part of FIG. Can do.

  Here, as described above, the function program processed by the main processor 121 may operate in cooperation with the sub processor program processed by the sub processor 123. Therefore, when the main processor 121 reads out the function program from the external recording unit 128 and writes it into the main memory 126, when there is a sub processor program that operates in cooperation with the target function program, the sub processor program also includes the same main program. It is assumed that data is written in the memory 126.

  In this case, there may be one or more sub-processor programs that operate in cooperation with each other. If there are a plurality of sub-processor programs, the sub-processor programs that operate in cooperation with each other are written in the main memory 126. The sub processor program written in the main memory 126 is then written in the LS 124 in the sub processor 123 and operates in cooperation with the function program processed by the main processor 121.

  As shown in the software cell in FIG. 13, an identifier that can uniquely identify a program for each program is assigned to the function program as a function program ID. The function program ID is determined from the creation date and time, the information processing apparatus ID, and the like at the stage of creating the function program.

  A sub processor program ID is also assigned to the sub processor program, whereby the sub processor program can be uniquely identified. The assigned sub processor program ID is an identifier related to the function program ID of the function program that is the partner of the cooperative operation, for example, the function program ID as a parent number and a branch number added at the end. There may be an identifier that is not related to the function program ID of the function program that is the partner of the cooperative operation.

  In any case, when the function program and the sub processor program operate in cooperation, it is necessary to store the program ID which is the identifier of the other party in the own program. Even when the function program operates in cooperation with a plurality of sub processor programs, the function program stores the sub processor program IDs of all the sub processor programs.

  The main processor 121 secures an area in the main memory 126 for storing device information (information regarding the operation state) of the information processing device on which the main processor 121 operates, and records the information as a device information table of the own device. The device information here is each piece of information below the information processing device ID shown in FIG.

(Determination of master / slave in the system)
In the network system described above, when the main power supply to a certain information processing apparatus is turned on, the main processor 121 of the information processing controller of the information processing apparatus loads a master / slave manager (hereinafter referred to as MS manager) into the main memory 126 and executes it. To do.

  When the MS manager detects that the information processing apparatus on which it operates is connected to the network 90, the MS manager confirms the existence of another information processing apparatus connected to the same network 90. As used herein, “connection” or “existence” means that the information processing apparatus is not only physically connected to the network 90 but also electrically and functionally connected to the network 90. Indicates.

  In addition, an information processing apparatus in which the device operates is referred to as a self device, and another information processing device is referred to as another device. The apparatus also indicates the information processing apparatus.

  A method in which the MS manager confirms the existence of another information processing apparatus connected to the same network 90 will be described below.

  The MS manager generates a software cell in which the DMA command is a status request command, the transmission source ID and the response destination ID are the information processing apparatus, and the transmission destination ID is not specified, and the network manager is connected to the information processing apparatus. To set a timer for network connection confirmation. The timeout time of the timer is, for example, 10 minutes.

  When another information processing apparatus is connected to the network system, the other apparatus receives the software cell of the status request command, and sends it to the information processing apparatus that has issued the status request command specified by the response destination ID. On the other hand, the DMA command is a status return command, and a software cell including device information of itself (other device) is transmitted as data. The software cell of this status reply command includes at least information for identifying the other device (information processing device ID, information on the main processor, information on the sub processor, etc.) and the MS status of the other device.

  The MS manager of the information processing apparatus that has issued the status request command monitors the reception of the software cell of the status reply command transmitted from another apparatus on the network until the timer for network connection confirmation times out. As a result, when the status reply command indicating the MS status = 0 (master device) is received, the MS status in the device information table of the own device is set to 1. Thus, the device becomes a slave device.

  On the other hand, if no status reply command is received before the network connection confirmation timer times out, or if no status reply command indicating MS status = 0 (master device) is received, The MS status in the device information table of the own device is set to 0. This makes the device a master device.

  That is, when any device is not connected to the network 90 or when a new information processing device is connected to the network 90 in a state where no master device exists on the network 90, the device automatically becomes the master device. Set as On the other hand, when a new information processing apparatus is connected to the network 90 in a state where a master apparatus already exists on the network 90, the apparatus is automatically set as a slave apparatus.

  For both the master device and the slave device, the MS manager periodically monitors the status of the other device by sending a status request command to the other device on the network 90 and inquiring status information. As a result, the main power supply of the information processing apparatus connected to the network 90 is cut off or the information processing apparatus is disconnected from the network 90, so that the status from a specific other apparatus is determined within a predetermined period set in advance for determination. When there is a change in the connection state of the network 90, such as when a reply command is not returned or when a new information processing apparatus is connected to the network 90, the information is notified to the ability exchange program described later. .

(Acquisition of device information in master device and slave device)
When the main processor 121 receives an inquiry from the MS manager about other devices on the network 90 and notification of completion of setting of the MS status of the own device, the main processor 121 executes the capability exchange program.

  When the own device is the master device, the capability exchange program acquires device information of all other devices connected to the network 90, that is, device information of each slave device.

  As described above, the device information of another device is generated by generating a software cell in which the DMA command is a status request command and transmitting it to the other device. Thereafter, the DMA command is a status return command and data of the other device. This is possible by receiving a software cell containing device information from another device.

  The capability exchange program stores an area for storing device information of all other devices (each slave device) connected to the network 90 in the main memory of the own device, similarly to the device information table of the own device which is the master device. This information is recorded in a device information table of another device (slave device).

  That is, the device information of all information processing devices connected to the network 90 including the device itself is recorded in the main memory 126 of the master device as a device information table.

  On the other hand, when the own device is a slave device, the capability exchange program acquires device information of all other devices connected to the network 90, that is, device information of each slave device other than the master device and the own device. The information processing device ID and the MS status included in the device information are recorded in the main memory 126 of the own device.

  That is, the device information of the own device is recorded as a device information table in the main memory 126 of the slave device, and the master device connected to the network 90 other than the own device and the information processing device ID for each slave device. And the MS status are recorded as another device information table.

  Further, in both the master device and the slave device, when the capability exchange program is notified from the MS manager that the information processing device is newly connected to the network 90 as described above, the device of the information processing device Information is acquired and recorded in the main memory 126 as described above.

  Note that the MS manager and the capability exchange program are not limited to being executed by the main processor 121, but may be executed by any of the sub processors 123. The MS manager and the capability exchange program are preferably resident programs that always operate while the main power supply of the information processing apparatus is turned on.

(When the information processing device is disconnected from the network)
For both the master device and the slave device, the capability exchange program causes the MS manager to cut off the main power supply of the information processing device connected to the network 90 or disconnect the information processing device from the network 90 as described above. When it is notified, the device information table of the information processing device is deleted from the main memory 126 of the own device.

  Furthermore, when the information processing apparatus disconnected from the network 90 is a master apparatus, a new master apparatus is determined by the following method.

  Specifically, for example, each of the information processing apparatuses that are not disconnected from the network 90 replaces the information processing apparatus ID of the own apparatus and the other apparatus with a numerical value, and sets the information processing apparatus ID of the own apparatus to the information processing apparatus of the other apparatus. If the information processing device ID of the own device is the smallest among the information processing devices not disconnected from the network 90, the slave device transitions to the master device and sets the MS status to 0. As described above, device information of all other devices (each slave device) connected to the network 90 is acquired as a master device and recorded in the main memory 126.

(Distributed processing based on device information)
As shown in the lower part of FIG. 15, in order for a plurality of information processing apparatuses 100, 200, 300, and 400 connected to the network 90 to operate as a single virtual information processing apparatus 700, the master apparatus is a user It is necessary to understand the operation of the slave device and the operating state of the slave device.

  FIG. 17 shows a state in which four information processing apparatuses operate as one virtual information processing apparatus 700. It is assumed that the information processing apparatus 100 is operating as a master apparatus and the information processing apparatuses 200, 300, and 400 are operating as slave apparatuses A, B, and C.

  When the user operates an information processing device connected to the network 90, if the operation target is the master device 100, the operation information is directly grasped in the master device 100, and if the operation target is a slave device, The operation information is transmitted from the operated slave device to the master device 100. That is, regardless of whether the user's operation target is the master device 100 or the slave device, the operation information is always grasped by the master device 100. The operation information is transmitted, for example, by a software cell whose DMA command is an operation information transmission command.

  Then, the main processor 121-1 included in the information processing controller 111 in the master device 100 selects a function program to be executed according to the operation information. At that time, if necessary, the main processor 121-1 included in the information processing controller 111 in the master device 100 may transfer the main memory 126-1 from the external recording units 128-1 and 128-2 of the own device by the above method. However, another information processing apparatus (slave apparatus) may transmit the function program to the master apparatus 100.

  In the function program, information processing apparatus type IDs represented as information shown in FIG. 14, main processor or sub-processor processing capacity, main memory usage, and conditions related to the external recording unit are required for each execution unit. The required specifications regarding the device are defined.

  The main processor 121-1 included in the information processing controller 111 in the master device 100 reads out the required specifications necessary for each function program. Further, the device information table of each information processing device is read by referring to the device information table recorded in the main memory 126-1 in advance by the capability exchange program. The device information here indicates each piece of information below the information processing device ID shown in FIG. 14, and is information on the main processor, sub processor, main memory, and external recording unit.

  The main processor 121-1 included in the information processing controller 111 in the master device 100 sequentially compares the device information of each information processing device connected to the network 90 and the required specifications necessary for executing the function program. To do.

  For example, when the function program requires a recording function, only the information processing apparatus having the recording function is specified and extracted based on the information processing apparatus type ID. Furthermore, a slave device that can secure the conditions regarding the processing capability of the main processor or sub processor, the amount of main memory used, and the external recording unit necessary for executing the function program is specified as an execution request candidate device. Here, when a plurality of execution request candidate devices are specified, one execution request candidate device is specified and selected from the candidate devices.

  When the slave device to be requested for execution is specified, the main processor 121-1 included in the information processing controller 111 in the master device 100 determines the main memory included in the information processing controller 111 in the own device for the specified slave device. The device information table of the slave device recorded in 126-1 is updated.

  Further, the main processor 121-1 included in the information processing controller 111 in the master device 100 generates a software cell whose DMA command is a function program execution command, and the cell interface of the software cell needs a function program related to the function program. The sub processor information and the sandbox size (see FIG. 13) are set and transmitted to the slave device requested to execute.

  The slave device requested to execute the function program executes the function program and updates the device information table of the own device. At that time, if necessary, the main processor 121 included in the information processing controller in the slave device transfers the function program from the external recording unit 128 of the own device to the main memory 126 and operates in cooperation with the function program by the above method. Load the processor program.

  When the required function program or the sub processor program that operates in cooperation with the function program is not recorded in the external recording unit 128 of the slave device requested to execute the function program, the other information processing apparatus What is necessary is just to comprise a system so that a program or a sub processor program may be transmitted to the slave apparatus of the function program execution request destination.

  The sub-processor program can be executed by another information processing apparatus using the aforementioned load command and kick command.

  After the execution of the function program, the main processor 121 included in the information processing controller in the slave device that has executed the function program transmits an end notification to the main processor 121-1 included in the information processing controller 111 in the master device 100. At the same time, the device information table of the own device is updated. The main processor 121-1 included in the information processing controller 111 in the master device 100 receives the end notification, and updates the device information table of the slave device that has executed the function program.

  The main processor 121-1 included in the information processing controller 111 in the master device 100 identifies itself as an information processing device that can execute the function program from the reference result of the device information table of the own device and the other device. There is also a case of selecting. In that case, the master device 100 executes the function program.

  In the example of FIG. 17, the case where the user operates slave device A (information processing device 200) and another slave device B (information processing device 300) executes a function program corresponding to the operation will be described with reference to FIG. An example of distributed processing is shown.

  In the example of FIG. 18, when the user operates the slave device A, the distributed processing of the entire network system including the slave device A starts. First, in step 81, the slave device A transmits the operation information to the master device. To 100.

  The master device 100 receives the operation information in step 72, and further proceeds to step 73, where each information processing is performed from the device information table of the own device and the other device recorded in the main memory 126-1 of the own device. The operating state of the apparatus is checked, and an information processing apparatus that can execute a function program corresponding to the received operation information is selected. This example is a case where the slave device B is selected.

  Next, in step 74, the master device 100 requests the selected slave device B to execute the function program.

  In step 95, the slave device B receives the execution request, and further proceeds to step 96 to execute the function program requested to be executed.

  As described above, the user operates only one information processing apparatus, and operates a plurality of information processing apparatuses 100, 200, 300, and 400 without operating other information processing apparatuses. The information processing apparatus 700 can be operated.

(Specific examples of information processing devices and systems)
Any of the information processing apparatuses 100, 200, 300, and 400 connected to each other via the network 90 basically performs information processing by the information processing controllers 111, 112, 113, and 114 as described above. An example is shown in FIG.

  In this example, the information processing apparatus 100 including the information processing controller 111 is a hard disk recorder. As shown in FIG. 20, the hardware configuration includes a hard disk as the external recording unit 128-1 shown in FIG. As the external recording unit 128-2 shown in FIG. 11, an optical disk such as DVD ± R / RW, CD ± R / RW, Blu-ray Disc (registered trademark) can be mounted, and the information processing controller 111 The bus 131-1 connected to the bus 129-1 is connected to the broadcast receiving unit 132-1, the video input unit 133-1, the audio input unit 134-1, the video output unit 135-1, the audio output unit 136-1, The operation panel unit 137-1, the remote control light receiving unit 138-1, and the network connection unit 139-1 are connected.

  The broadcast receiving unit 132-1, the video input unit 133-1 and the audio input unit 134-1 receive a broadcast signal or input a video signal and an audio signal from the outside of the information processing apparatus 100, and each has a digital format of a predetermined format. This is converted into data and sent to the bus 131-1 for processing by the information processing controller 111. The video output unit 135-1 and the audio output unit 136-1 are transmitted from the information processing controller 111 to the bus 131-. The video data and the audio data sent to 1 are processed, converted into digital data or converted into analog signals, and sent to the outside of the information processing apparatus 100. The remote control light receiving unit 138-1 is a remote controller. A remote control (remote operation) infrared signal is received from the transmitter 143-1.

  As shown in FIG. 19 and FIG. 20, a monitor display device 141 and a speaker device 142 are connected to the video output unit 135-1 and the audio output unit 136-1 of the information processing device (hard disk recorder) 1.

  The information processing apparatus 200 including the information processing controller 112 in the example of FIG. 19 is also a hard disk recorder, and is configured in the same manner as the information processing apparatus 100 as indicated by reference numerals in parentheses in FIG. is there. However, for example, as shown in FIG. 19, a monitor display device and a speaker device are not connected to the information processing device (hard disk recorder) 200.

  As shown in FIG. 21, the software configuration of the information processing apparatuses (hard disk recorders) 100 and 200, that is, the information processing controllers 111 and 112 includes an MS manager and a capability exchange program as control programs, and video and audio as function programs. Programs for recording, video / audio playback, material search and program recording reservation are provided, and programs for broadcast reception, video output, audio output, external recording unit input / output and network input / output are provided as device drivers.

  The information processing apparatus 300 including the information processing controller 113 in the example of FIG. 19 is a PDA (Personal Digital Assistant). As illustrated in FIG. 22, the hardware configuration is the external recording unit 128-5 illustrated in FIG. The memory card disk is configured so that a memory card disk can be mounted, and a liquid crystal display unit 152, an audio output unit 153, a camera unit 154, and an audio input unit 155 are connected to the bus 151 connected to the bus 129-3 of the information processing controller 113. The keyboard unit 156 and the network connection unit 157 are connected.

  Note that the information processing controller 113 whose interior is omitted in FIG. 11 includes a main processor 121-3, sub processors 123-7, 123-8, 123-9, a DMAC (direct memory access controller) 125-3, and a DC (disk controller). ) 127-3 and the bus 129-3, the main processor 121-3 has an LS (local storage) 122-3, and each of the sub processors 123-7, 123-8, 123-9 has an LS ( Local storage) 124-7, 124-8, 124-9.

  As shown in FIG. 23, the software configuration of the information processing apparatus (PDA) 300, that is, the information processing controller 113 includes an MS manager and a capability exchange program as control programs, and video and audio recording and video and audio reproduction as function programs. , A phone book, a word processor and a spreadsheet program, and a Web browser, and device drivers include video output, audio output, camera video input, microphone audio input, and network input / output programs.

  The information processing apparatus 400 including the information processing controller 114 in the example of FIG. 19 is a portable CD player. As shown in FIG. 24, the hardware configuration is a CD as the external recording unit 128-6 shown in FIG. (Compact Disc) can be mounted, and a liquid crystal display unit 162, an audio output unit 163, an operation button unit 164, and a network connection unit 165 are connected to a bus 161 connected to the bus 129-4 of the information processing controller 114. Are connected.

  Note that the information processing controller 114 that is omitted in FIG. 11 includes a main processor 121-4, sub-processors 123-10, 123-11, 123-12, a DMAC (direct memory access controller) 125-4, and a DC (disk controller). ) 127-4 and bus 129-4, and the main processor 121-4 has an LS (local storage) 122-4, and each of the sub processors 123-10, 123-11, 123-12 has an LS (local storage). Local storage) 124-10, 124-11, 124-12.

  As shown in FIG. 25, the software configuration of the information processing apparatus (portable CD player) 400, that is, the information processing controller 114 includes an MS manager and a capability exchange program as a control program, and a function program for playing music. A program for audio output, CD control, and network input / output is provided as a device driver.

  In the network system in the example of FIG. 19 as described above, the information processing apparatuses 100, 300, and 400 are connected to the network 90, and the information processing apparatus 100 is the master apparatus (MS status = 0). And 400 are set as slave devices (MS status = 1).

  In this state, when the information processing apparatus 200 is newly connected to the network 90, the MS manager executed by the main processor 121-2 included in the information processing controller 112 in the information processing apparatus 200 is The other information processing devices 100, 300, and 400 are inquired of the MS status, recognizes that the information processing device 100 already exists as a master device, and sets its own device (information processing device 200) as a slave device (MS status = Set to 1). Further, the information processing apparatus 100 set as the master apparatus collects apparatus information of each apparatus including the newly added information processing apparatus 200 and updates the apparatus information table in the main memory 126-1.

  In this state, a case where a user performs an operation for recording reservation of a broadcast program for 2 hours in the information processing apparatus (PDA) 300 as a slave apparatus is shown.

  In this case, the information processing apparatus (PDA) 300 as a slave apparatus receives input of recording reservation information including information such as a recording start time, a recording end time, a recording target broadcast channel, and recording quality from the user, and the recording reservation information And a software cell including a recording reservation command as a DMA command is generated and transmitted to the information processing apparatus 100 which is a master apparatus.

  The main processor 121-1 included in the information processing controller 111 in the information processing apparatus 100 that has received the software cell whose DMA command is the recording reservation command reads out the recording reservation command and also stores the apparatus information table in the main memory 126-1. The information processing apparatus that can execute the recording reservation command is specified.

  First, the main processor 121-1 reads the information processing device type ID of each information processing device 100, 200, 300, 400 included in the device information table and can execute a function program corresponding to the recording reservation command. Extract the device. Here, the information processing devices 100 and 200 having the information processing device type ID indicating the recording function are specified as candidate devices, and the information processing devices 300 and 400 are excluded from the candidate devices.

  Next, the main processor 121-1 included in the information processing controller 111 in the information processing apparatus 100 that is the master apparatus refers to the apparatus information table, and the processing capability of the main processor or sub-processor of the information processing apparatuses 100 and 200. Then, information related to the device such as information related to the main memory is read, and it is determined whether or not the information processing devices 100 and 200 satisfy the required specifications necessary for executing the function program corresponding to the recording reservation command. Here, it is assumed that the information processing apparatuses 100 and 200 satisfy the required specifications necessary for executing the function program corresponding to the recording reservation command.

  Further, the main processor 121-1 reads the information related to the external recording unit of the information processing apparatuses 100 and 200 with reference to the device information table, and the free capacity of the external recording unit determines the capacity necessary for executing the recording reservation command. Determine if you are satisfied. Since the information processing apparatuses 100 and 200 are hard disk recorders, the difference between the total capacity and the used capacity of the hard disks 128-1 and 128-3 corresponds to the respective free capacity.

  In this case, the free capacity of the hard disk 128-1 of the information processing apparatus 100 is 10 minutes in terms of recording time, and the free capacity of the hard disk 128-3 of the information processing apparatus 200 is 20 hours in terms of recording time. Suppose that

  At this time, the main processor 121-1 included in the information processing controller 111 in the information processing apparatus 100 that is the master apparatus installs an information processing apparatus that can secure a free space for two hours necessary for executing the recording reservation command. Identifies as an execution request destination slave device.

  As a result, only the information processing apparatus 200 is selected as an execution request destination slave apparatus, and the main processor 121-1 included in the information processing controller 111 in the information processing apparatus 100 that is the master apparatus is operated by the user. The recording reservation command including the recording reservation information transmitted from 300 is transmitted to the information processing apparatus 200 to request execution of recording reservation of the broadcast program for 2 hours.

  Then, the main processor 121-2 included in the information processing controller 112 in the information processing apparatus 200 analyzes the recording reservation command and sends a function program necessary for recording from the hard disk 128-3 as an external recording unit to the main memory. The program is loaded to 126-2, and recording is performed according to the recording reservation information. As a result, the video / audio data of the 2-hour broadcast program reserved for recording is recorded on the hard disk 128-3 of the information processing apparatus 200.

  As described above, also in the network system in the example of FIG. 19, the user operates only one information processing apparatus, and does not operate other information processing apparatuses, thereby operating a plurality of information processing apparatuses 100, 200, 300 and 400 can be operated as one virtual information processing apparatus 700.

  As described above, in the information processing apparatus shown in FIG. 11, the multi-viewpoint video playback apparatus in which the main processor 121 is the processing control device unit 3 shown in FIG. 4 and the sub processor 123 is the processing device unit is configured. Can do.

  Further, a network system including a plurality of information processing apparatuses 100, 200, 300, 400,... Is regarded as one information processing apparatus, and the decoding processing of the video / audio stream is distributed among the information processing apparatuses of the network system. You can also In this case, one of the information processing devices constituting the network system may have a function as a processing control device unit.

  Further, in a network system composed of a plurality of information processing devices of multiprocessors connected through a network, processing target data is distributed to each information processing device, and each information processing device also distributes processing target data to each sub processor. The present invention can also be applied to such control.

  The network may be one that connects each device unit inside a device such as a ring bus, or may be a local area network or a wide area network. In other words, the information processing apparatus can be applied to an apparatus (system) configured by connecting a plurality of processing apparatus units to a bus or a network, although the scale is different.

It is a figure for demonstrating an example of a multi camera system. It is a figure for demonstrating an example of a multi camera system. It is a figure for demonstrating an arbitrary viewpoint image | video reproduction | regeneration systems. It is a figure for demonstrating the arbitrary viewpoint video reproduction apparatus with which one Embodiment of the apparatus by this invention and the method was applied. It is a figure for demonstrating the structural example of the control information formed and managed in the control processing part 3 in order to perform a synchronous process and an asynchronous process. 4 is a block diagram for explaining a processing control device unit 3. FIG. It is a figure for demonstrating the allocation of the process of each layer, such as IP / UDP / RTP. It is a figure for demonstrating the synchronous process in processing apparatus part 1 (1), 1 (2), 1 (3). 3 is a block diagram for explaining a configuration example of a processing device unit 1. FIG. 3 is a flowchart for explaining the operation of a processing control device unit 3; It is a figure for demonstrating the example of the information processing apparatus with which the apparatus by this invention and the method are applied. It is a figure where it uses for description of the information processing controller with which the information processing apparatus used as a processing apparatus part is provided. It is a figure which shows an example of a software cell. It is a figure which shows the data area of a software cell when a DMA command is a status reply command. It is a figure which shows a mode that a some information processing apparatus operate | moves as one virtual information processing apparatus. It is a figure which shows an example of the software configuration of an information processing controller. It is a figure which shows a mode that four information processing apparatuses operate | move as one virtual information processing apparatus. It is a figure which shows the example of the distributed process in the system of FIG. It is a figure which shows the specific example of each information processing apparatus and a system. It is a figure which shows the hardware constitutions of the hard-disk recorder in FIG. FIG. 20 is a diagram showing a software configuration of the hard disk recorder in FIG. 19. It is a figure which shows the hardware constitutions of PDA in FIG. It is a figure which shows the software structure of PDA in FIG. FIG. 20 is a diagram illustrating a hardware configuration of the portable CD player in FIG. 19. FIG. 20 is a diagram showing a software configuration of the portable CD player in FIG. 19. It is a figure for demonstrating the example when a synchronous process is performed in each processing apparatus. It is a figure shown about the synchronous process and idle time which are performed in each processing apparatus. It is a figure for demonstrating a problem when synchronous processing and asynchronous processing are mixed.

Explanation of symbols

  1 (1) to 1 (n): processing unit, 2 ... data bus, 3 ... processing control unit, 30 ... control unit, 31 ... CPU, 32 ... ROM, 33 ... RAM, 34 ... EEPROM, 35 ... CPU Bus, 36 ... interface, 37 ... scheduling unit, 371 ... processing time estimation unit, 372 ... free time calculation unit, 373 ... search selection unit, 4 ... network interface

Claims (10)

A plurality of processing device units capable of executing synchronous processing and asynchronous processing, and a processing control device unit for controlling each of the plurality of processing device units,
The processing control unit is
Estimating means for estimating the time required for the asynchronous processing when the asynchronous processing is interrupted to the processing unit;
Free time calculating means for calculating the free time of each processing unit,
Search means for searching for a processing device unit having the maximum free time based on free times of the plurality of processing device units from the free time calculating means;
An information processing apparatus comprising: a determination unit configured to determine a processing device unit that interrupts the asynchronous process from a search result of the search unit and a required time estimated from the estimation unit. The information processing apparatus according to claim 1,
The synchronization processing executed in the processing device section is one or both of compression encoding processing and compression decoding processing. The information processing apparatus according to claim 1 or 2,
The information processing apparatus, wherein the plurality of processing device units are connected by a ring-shaped bus. An information processing apparatus according to claim 1, claim 2, or claim 3,
The information processing apparatus according to claim 1, wherein the asynchronous processing uses one or both of video data and audio data as processing target data. The information processing apparatus according to claim 4,
The processing target data is provided using RTP (real-time transport protocol),
The processing control unit is
Dividing means for dividing the data to be processed into layers of RTP / UDP (user datagram protocol) / IP (internet protocol);
Distributing means for supplying the target data of each layer divided by the dividing means to different processing device units. An information processing apparatus comprising: An information processing method in an information processing apparatus comprising a plurality of processing device units capable of executing synchronous processing and asynchronous processing, and a processing control device unit for controlling each of the plurality of processing device units,
In the processing control unit,
An estimation step for estimating the time required for the asynchronous processing when the asynchronous processing is interrupted to the processing unit;
A free time calculating step for calculating free time of each processing unit;
A search step of searching for a processing unit having the maximum free time based on the free times of the plurality of processing units calculated in the free time calculating step;
An information processing method comprising: a determination step of determining a processing device unit that interrupts the asynchronous processing from a search result in the search step and a required time estimated in the estimation step. An information processing method according to claim 6,
The information processing method characterized in that the synchronization process executed in the processing unit is one or both of a compression encoding process and a compression decoding process. The information processing method according to claim 6 or 7,
The information processing method, wherein the plurality of processing units are connected by a ring-shaped bus and can perform pipeline processing. An information processing method according to claim 6, 7 or 8,
The information processing method, wherein the asynchronous processing uses one or both of video data and audio data as processing target data. An information processing method according to claim 9,
The processing target data is provided using RTP (real-time transport protocol),
In the processing control device,
A division step of dividing the processing target data into each layer of RTP / UDP (user datagram protocol) / IP (internet protocol);
A distribution step of supplying the processing target data of each layer divided in the division step to different processing device units.

Images