JP2006014180A - Data processor, data processing method and program therefor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a data processor, a data processing method and a program therefor by which data can surely be composited and reproduced so that media object data such as motion picture and audio is synchronized with scenes when reproducing encoded multimedia data composed of a plurality of objects. <P>SOLUTION: A media decoder circuit 104 decodes a plurality of object data from which multimedia data are demultiplexed by a demultiplexer circuit 102. A scene description data decoder circuit 103 decodes scene description data contained in received multimedia data. The demultiplexer circuit 102 outputs an event issuing instruction in a timing of starting reading object data. An event generating circuit 105 generates an event in response to the event issuing instruction. A scene composing circuit 106 composes a plurality of decoded object data based on the scene description data in accordance with the timing when the event is received. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a data processing apparatus, a data processing method, and a data processing apparatus for separating and decoding multimedia data including, for example, encoded moving image object data, audio object data, and scene description data, and synthesizing and outputting the decoded data It is about the program.

  MPEG (Motion Picture Experts Group) -1, MPEG-2, and the like are known as international standards for encoding standards that compress, encode, multiplex, transmit or store moving images and audio, and demultiplex and decode them. ing.

  On the other hand, ISO / IEC 14496 part 1 (MPEG-4 Systems) multiplexes and synchronizes an encoded bit stream of multimedia data including a plurality of objects such as still images, moving images, audio, text, and CG (Computer Graphics). The technique to do is standardized.

  In the MPEG-4 data stream as described above, unlike conventional general multimedia data, each object such as text and CG is arranged in space and time in addition to still images, moving images and audio. As information for this purpose, there is included BIFS (Binary Format for Scenes) in which VRML (Virtual Reality Modeling Language) is extended to handle natural moving images and sounds. Here, BIFS is information describing an MPEG-4 scene in binary.

  Individual objects such as still images, moving images, and audio that make up such multimedia data are individually encoded and transmitted, so that the decoding side individually decodes the above-described objects. It is arranged temporally and spatially according to the scene description information, and the time axis of each data is synchronized with the time axis inside the player, and the scene is synthesized and reproduced.

  In addition to the VRML and BIFS described above, a method for describing a scene configuration is generally SMIL (Synchronized Language Integrated) described using HTML (Hypertext Markup Language) and XML (extensible Markup Language). ) And XMT (extensible MPEG-4 Textual Format).

  When reproducing such a bit stream of multimedia data, it is required to synthesize and reproduce the scene and each object constituting the scene in addition to the moving image and sound. Therefore, a method of synthesizing and reproducing audio, video, and CG in synchronization has been proposed (see, for example, Patent Document 1).

  In addition, by performing prior acquisition of subsequent content data during the playback of content data, it is possible to perform playback while keeping the time specified by the scene description information, and to reduce the delay until the start of playback or the start of the next playback. A technique has been proposed (for example, see Patent Document 2).

JP-A-10-136259 JP 2002-268999 A

  However, the method proposed in Patent Document 1 does not mention anything about the delay required to read a moving image or sound and start playback. For example, the method of moving image or sound distributed via a network is not mentioned. When reading takes time, there is a problem that moving images and sounds cannot be synthesized and reproduced according to the temporal arrangement of scene description information.

  In addition, the method proposed in Patent Document 2 is limited to the case where multimedia data distributed via a network is received and played back, and thus cannot be applied to playback from a storage medium. Depending on the communication speed of the network and the line status, there is a problem that the synchronization of the moving image or sound and the scene is not always ensured.

  In the scene description methods such as VRML and BIFS described above, an operating environment where processing is performed at an infinitely high speed is ideal. However, since reading of moving images and sounds is a heavy work, in an actual operating environment, It takes time to start playback. Thus, the time required to start reproduction depends on the communication speed of the network, the line status, and the processing capability of the receiving terminal. For this reason, when the communication speed is low, the line is congested, or when a receiving terminal with low processing capacity is used, the scene is composed at the start of playback or when the scene description data is updated. There is a problem that synchronization with media object data such as moving images and sounds cannot be achieved.

  The present invention has been made in consideration of the above-described circumstances. When playing back encoded multimedia data composed of a plurality of objects such as moving images, sounds, still images, texts, CGs, An object of the present invention is to provide a data processing apparatus, a data processing method, and a program thereof that can reliably perform synthesis and reproduction in which media object data such as audio and a scene are synchronized.

  The present invention has been made to solve the above-described problems. In the data processing apparatus according to the present invention, multimedia data including one or more encoded moving image and / or audio object data is transmitted via a network. Is a data processing apparatus for receiving and reproducing according to scene description data, and separating means for separating received multimedia data in units of object data, and decoding a plurality of object data separated by the separating means Scene description data when receiving one or more first decoding means and scene description data encoded as part of the data included in the multimedia data or as independent data on a communication path different from the network A second decoding means for decoding the object data and a separating means for reading the object data. The event generation means for generating an event in accordance with the timing at which the event generation means decodes the object data, and the timing at which the event generation means receives the event. And scene synthesizing means for synthesizing a plurality of object data decoded by the first decoding means based on the scene description data decoded by the decoding means.

  In the data processing method according to the present invention, the multimedia data including one or a plurality of encoded moving image and / or audio object data is received via the network and is reproduced according to the scene description data. A data processing method using a processing device, comprising: a separation step of separating received multimedia data in units of object data; and one or a plurality of first decryptions for decoding a plurality of object data separated in the separation step And a second decoding for decoding the scene description data when receiving the scene description data encoded as part of the data included in the multimedia data or as independent data through a communication path different from the network Of object data in the conversion step and separation step An event issuing step for issuing an event according to the start timing or the timing for decoding the object data in the first decoding step, and the second decoding according to the timing for receiving the event generated in the event generation step And a scene synthesis step for synthesizing a plurality of object data decoded in the first decoding step based on the scene description data decoded in the encoding step.

  Also, the program according to the present invention is for a data processing apparatus that receives multimedia data including one or a plurality of encoded moving image and / or audio object data via a network and reproduces the data according to scene description data. A separation step of separating received multimedia data in units of object data, one or more first decoding steps for decoding a plurality of object data separated in the separation step, and multimedia A second decoding step for decoding the scene description data when receiving the scene description data encoded as part of the data included in the data or as independent data through a communication path different from the network; and a separation step When to start loading object data with The event issuing step for issuing an event according to the timing for decoding the object data in the first decoding step, and the second decoding step for receiving the event generated at the event generation step. A program for causing a computer to execute a scene synthesis step for synthesizing a plurality of object data decoded in the first decoding step based on the decoded scene description data.

  Also, the recording medium according to the present invention is a data processing device for receiving multimedia data including one or more encoded moving image and / or audio object data via a network, and reproducing the data according to scene description data. A recording medium on which a program for recording is recorded, a separation step of separating received multimedia data in units of object data, and one or a plurality of first decryptions for decoding a plurality of object data separated in the separation step And a second decoding for decoding the scene description data when receiving the scene description data encoded as part of the data included in the multimedia data or as independent data through a communication path different from the network Open the object data reading in the conversion step and separation step Or an event issuing step for issuing an event according to the timing at which the object data is decoded at the first decoding step, and a second decoding according to the timing at which the event generated at the event generation step is received. A computer-readable recording of a program for causing a computer to execute a scene synthesis step for synthesizing a plurality of object data decoded in the first decoding step based on the scene description data decoded in the step It is a recording medium.

  A data processing apparatus, a data processing method, and a program therefor according to the present invention, when separating and reproducing each object from multimedia data composed of a plurality of objects, media object data such as moving images and sounds and a scene are There is an effect that synchronized composition and reproduction can be performed reliably regardless of the type of communication line, the line status, and the processing capability of the terminal.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[First Embodiment]
FIG. 1 is a diagram showing a basic configuration of a multimedia data receiving apparatus (hereinafter simply referred to as a receiving apparatus) 101 as a first embodiment of the present invention. In FIG. 1, the circuit configuration is shown, and the data flow between the circuits is also shown.

  As shown in FIG. 1, a receiving apparatus 101 includes a demultiplexing circuit (separating unit) 102, a scene description data decoding circuit (second decoding unit) 103, a media decoding circuit (first decoding unit) 104, The event generation circuit 105, the scene synthesis circuit 106, and the output device 107 are configured. In FIG. 1, reference numeral 100 denotes a transmission path typified by various networks, and in this embodiment, a network through which processed and encoded multimedia data is distributed. Here, in the present embodiment, multimedia data is acquired from the transmission line 100, but the present invention is not limited to such a method of acquiring via a communication path such as a broadcast network or a communication network. A method of acquiring multimedia data from a recording medium such as a RAM may be used.

  When receiving the multimedia data distributed via the network via the transmission line 100, the receiving apparatus 101 inputs the multimedia data to the demultiplexing circuit 102. The demultiplexing circuit 102 separates the received multimedia data into scene description data, media object data such as still images, moving images, and audio, and outputs them to the scene description data decoding circuit 103 and the media decoding circuit 104, respectively. To do. Note that, as described above, the receiving apparatus 101 may be configured to read data from a recording medium so that multimedia data read from the recording medium may be input to the demultiplexing circuit 102.

  In FIG. 1, there is only one media decoding circuit 104. However, in this embodiment, a plurality of objects exist in the multimedia data for still image object data, moving image object data, and audio object data. For example, it is assumed that the media decoding circuit 104 includes a plurality of decoding circuits for still images, moving images, and audio.

  The still image object data is data that has been highly efficient (compressed) encoded by, for example, the well-known JPEG method. The moving image object data is data that has been highly efficient encoded by, for example, the well-known MPEG-2, MPEG-4, or H263 system. The audio object data is data that has been subjected to high-efficiency encoding such as well-known CELP (Code Excited Linear Prediction), AAC (Advanced Audio Coding), transform domain weighted interleaved vector quantization (TWINVQ) encoding, and the like. is there.

  The encoded scene description data and media object data are decoded by the scene description data decoding circuit 103 and the media decoding circuit 104, respectively, and supplied to the scene synthesis circuit 106. The scene synthesis circuit 106 synthesizes the scene and the decoded media object data based on the decoded scene description data. The final multimedia data string obtained in this way is supplied to the output device 107 typified by a display, a speaker, a printer, and the like and reproduced.

  In addition, the demultiplexing circuit 102 has a function (issue command means) that transmits an event issue command to the event generation circuit 105 when reading of a moving image or sound related to each node is started. The event generation circuit 105 that has received the event issuance command transmits the event to the scene synthesis circuit 106.

Next, the data structure of multimedia data received by the receiving apparatus 101 in the present embodiment will be described with a specific example.
FIG. 2 is a diagram showing an example of the data structure of the entire multimedia data 200 in the present embodiment. As shown in FIG. 2, the multimedia data 200 is composed of packets such as scene description data 201 and media data 202 to 205. The scene description data 201 is transmitted before the media data 202 to 205 constituting the scene, and the receiving apparatus 101 also acquires the data in the order of the scene description data 201 and the media data 202 to 205.

  For each packet of the scene description data 201 and the media data 202 to 205, time information (DTS, CTS, etc.) for synchronization management is added to the packet header portion. The DTS (Decoding Time Stamp) is information indicating the time at which the packet must arrive at a decoding buffer (not shown) in front of the scene description data decoding circuit 103 and the media decoding circuit 104. Further, CTS (Composition time stamp) is information indicating the time at which the packet must exist in a composition memory (not shown) downstream of the scene description data decoding circuit 103 and the media decoding circuit 104. Each packet is decoded at the DTS time of the packet header part added for each packet, and becomes valid at the time after the CTS. In an ideal operating environment where processing can be performed at an infinitely high speed as described above, media data can be decoded and reproduced according to the time information DTS and CTS.

  Here, the description of the scene description data 201 shown in FIG. 2 will be described with a specific example. FIG. 3 is a diagram showing a description example of the scene description data 201 shown in FIG. As shown in FIG. 3, the scene description data 201 has a description configuration like BIFS data 300 described using, for example, BIFS (Binary Format for Scenes) as a scene description language. Here, the scene is a screen or time structure presented to the viewer, and the above-described BIFS is standardized as a scene description language in the MPEG-4 system part. Here, for simplicity, the BIFS data is represented as text as shown in FIG.

  BIFS data 300 begins with a Group node 301. All BIFSs begin with a type of node called SFTopNode, but Group node 301 is one of the SFTopNodes. In the child node (children field) of the Group node 301, information regarding a moving image is described as a Transform2D node 302. The Transform2D node 302 is defined with the name MOVIE using the keyword DEF. Hereinafter, the Transform2D node 302 is simply referred to as MOVIE 302.

  The actually displayed moving image data is defined by the Movie Texture node 303, and “test.mpeg” described in the field url indicates the location of the moving image object data. Here, “test.mpeg” is, for example, an MPEG1-Video moving image file format.

  In addition, the TimeSensor node 304 generates an event as time passes, and outputs a fraction_changed event in the range of 0.0 to 1.0 from time 0 seconds (field startTime 0) to 1 second (field cycleInterval 1). The TimeSensor node 304 is defined with the name TIMER using the keyword DEF. Hereinafter, the TimeSensor node 304 is simply referred to as TIMER304.

  The PositionInterpolator2D node 305 is one of complement nodes. Linear interpolation is performed on 0 to 1 input values (field key [0 1]), and values (1, 1) to (2, 2) (field keyValue [1, 1 2, 2]) are value_changed. For example, (1.5, 1.5) is output for an input value of 0.5. The PositionInterpolator2D node 305 is defined with the name SCALE using the keyword DEF. Hereinafter, the PositionInterpolator2D node 305 is simply referred to as SCALE305.

  The TIMER 304 fraction_changed output event is routed to the SCALE 305 set_fraction input event by the ROUTE statement 306. By this ROUTE connection, linear interpolation is executed, and the result is sent as a value_changed output event of SCALE 305.

  Further, the value_changed output event of SCALE 305 is route-connected to the field scale of MOVIE 302 by the ROUTE statement 307. By this ROUTE connection, the display scale of the moving image MOVIE 302 is changed.

  Thus, according to the BIFS data 300, the scale of the moving image object data “test.mpeg” is expanded from (1,1) to (2,2) from the time 0 second to 1 second. For example, at 0.5 seconds, both the width and height are displayed on a scale of 1.5 times. It should be noted that detailed description of other nodes and fields is omitted here.

Next, an example of multimedia data including the BIFS data 300 shown in FIG. 3 and differences in decoding and reproduction processing depending on the operating environment for the multimedia data will be described.
FIG. 4A shows an example of multimedia data including the BIFS data 300 shown in FIG. As shown in FIG. 4A, the multimedia data 400 includes the above-described BIFS data 300 and each frame FrameN (N = 0, 1, 2,...) Of the moving image object data “test.mpeg”. The FIG. 4B is a diagram showing a difference in decoding and reproduction processing depending on the operating environment for the multimedia data 400 shown in FIG.

  In the present embodiment, 0 (msec (milliseconds)) is added to the packet of the BIFS data 300 constituting the multimedia data 400 as DTS and CTS, and each frame FrameN of the moving image object data “test.mpeg” is added. Suppose that 50 × N (msec) is added as DTS and CTS.

  When the multimedia data 400 to which the above-described DTS and CTS are added is decoded and played back, the BIFS data 300 and the moving image object data Frame 0 are displayed in an ideal operating environment as shown in the upper part of FIG. 4B. (401) is decoded and reproduced at time 0 msec. The moving image object data Frame 10 (402) is decoded and reproduced at time 500 (msec). Similarly, moving image object data Frame 20 (403) is decoded and reproduced at time 1000 (msec), and moving image object data Frame 30 (404) is decoded and reproduced at time 1500 (msec). In this way, in an ideal operating environment where processing is performed at infinitely high speed, the BIFS data 300 and the moving image object data FrameN are decoded and reproduced at ideal timing according to the time information DTS and CTS.

Here, an example of a screen when the multimedia data 400 is decoded and reproduced at an ideal timing as shown in the upper part of FIG. 4B will be described.
FIG. 5 is a diagram showing an example of a screen over time when the multimedia data 400 is decoded and reproduced at an ideal timing as shown in the upper part of FIG. 4B. As shown in FIG. 5, at time 0 msec, the moving image Frame0 (401) is reproduced with the same scale of width and height (screen 500). Next, at time 500 msec, the moving image Frame 10 (402) is reproduced with a scale of 1.5 times the width and height (screen 501). Similarly, at the time of 1000 msec and 1500 msec, the moving images Frame 20 (403) and Frame 30 (404) are reproduced with the width and height scales doubled (screen 502, screen 503). As described above, each frame FrameN of the moving image object data is reproduced at a timing according to the time information DTS and CTS on the scale intended by the content creator.

  However, when the communication speed of the network to which the transmission line 100 is connected is low, when the line is congested, or when the processing capacity of the receiving apparatus 101 is low, the BIFS data 300 in the multimedia data 400 is used. There may be a case where a delay occurs in the receiving process and the decoding process of the moving image object data FrameN that follows. In such a case, there is a case where the moving image object data FrameN cannot be decoded and reproduced at a timing according to the time information DTS and CTS. The lower part of FIG. 4B shows a case where the moving image object data FrameN cannot be decoded and reproduced at the timing according to the time information DTS and CTS.

  Specifically, the delay environment shown in the lower part of FIG. 4B is an example in the case where a delay of 500 msec occurs in receiving or decoding moving image object data. That is, the BIFS data 300 is decoded and reproduced at time 0 msec, but the frames 401 to 404 of the moving image object data are decoded and reproduced with a delay of 500 msec from the respective time information DTS and CTS.

Here, an example of a screen when the multimedia data 400 is decoded and reproduced in a delay environment in which a delay occurs as shown in the lower part of FIG. 4B will be described.
FIG. 6 is a diagram showing an example of a screen when conventional decoding and reproduction processing is performed on the multimedia data 400 in a delay environment in which a delay occurs as shown in the lower part of FIG. 4B. As shown in FIG. 6, each frame of a moving image is reproduced at a scale different from that intended by the content creator due to a shift in the timing of decoding and reproduction processing.

  That is, at time 0 msec, the BIFS data 300 is decoded, but nothing is displayed because the moving picture Frame 0 (401) has not been decoded (screen 600). Next, at time 500 msec, the moving image Frame 0 (401) to be displayed at time 0 msec is reproduced with a scale of 1.5 times the width and height (screen 601). Next, at the time of 1000 msec, the moving image Frame 10 (402) to be displayed at the time of 500 msec is reproduced with the scale of the width and the height being doubled (screen 602). Next, at the time of 1500 msec, the moving image Frame 20 (403) to be displayed at the time of 1000 msec is reproduced with the scale of width and height being doubled (screen 603). At the time of 2000 msec, the moving image Frame 30 (404) to be displayed at the time of 1500 msec is reproduced with the scale of the width and the height being doubled (screen 604).

  When conventional decoding and playback processing is performed in an environment where delay occurs in this way, scene description data is decoded and played back according to time information, so the scene (field value, etc.) changes over time. If there is a delay in receiving and decoding object data, there is a problem that the scene description data and the media object data cannot be synchronized.

  The reception apparatus 101 according to the present embodiment includes the event generation circuit 105, so that the scene description data and the media object data can be reproduced in synchronism even in the environment where the delay occurs. Hereinafter, a process of combining a plurality of media object data based on scene description data using an event generated by the event generation circuit 105 will be described with reference to flowcharts of FIGS.

  FIG. 7 is a flowchart showing a process in which the scene synthesis circuit 106 synthesizes the scene description data and the media object data when the receiving apparatus 101 receives the scene description data. FIG. 7 particularly describes the processing of the scene synthesis circuit 106, but before step S701 in FIG. 7, the demultiplexing circuit 102 performs demultiplexing processing on the received multimedia data to The description data decoding circuit 103 and the media decoding circuit 104 perform a decoding process on each demultiplexed data. The scene description data included in the multimedia data is transmitted before the media object data as described above, or retransmitted in order to intentionally update the scene from the transmission side.

  First, the scene synthesis circuit 106 reads the decoded scene description data from the scene description data decoding circuit 103 (step S701). Next, the scene synthesis circuit 106 analyzes the scene from the read scene description data (step S702). Next, as a result of the analysis, the scene synthesis circuit 106 determines whether or not the scene description data refers to media data such as a moving image, audio, or still image (step S703). For example, when BIFS is used as the scene description data, this determination is made based on whether or not a node that refers to media data, such as a MovieText / AudioClip / ImageTexture node, is included.

  For example, in the example of the BIFS data 300 shown in FIG. 3, since the Movie Texture node 303 is included, the scene composition circuit 106 determines to refer to the media data (“Yes” in step S703).

  If it is determined in step S703 that there is no reference to media data in the scene (“No” in step S703), the scene synthesis circuit 106 does not need to synthesize media data, and the scene read in step S701. The description data is synthesized and output as it is (step S707).

  If it is determined in step S703 that there is a reference to media data in the scene ("Yes" in step S703), the scene composition circuit 106 sets the current time as the playback start time of the media data. It is determined whether or not (step S704). For this determination, for example, in the case of the aforementioned AudioClip node, it can be determined from the startTime field, and it is determined that the start time of the media data has come when the current time exceeds the startTime. In the case of the above-mentioned Movie Texture node, it is determined that the start time has been reached at the current time if playback in the reverse direction is not specified even if it is not the start time. The reason is that the Movie Texture node needs to display the first frame of the moving image (Frame 0 (401) in the example of FIG. 4) if the value of the speed field is not negative even if the current time exceeds startTime. Because there is. Further, since the Image Texture node does not have a field indicating the start time, it is similarly determined that the current time is the start time.

  If it is determined in step S704 that the current time is not the start time of the media data (“No” in step S704), the scene synthesis circuit 106 does not need to synthesize the media data at the current time. In step S708, it is determined whether the scene description data refers to other media data. On the other hand, when it is determined that the current time is the start time of the media data (“Yes” in step S704), the scene synthesis circuit 106 determines whether an event transmitted from the event generation circuit 105 has been received. Judgment is made (step S705).

  The content of this determination processing uses an event transmitted from the event generation circuit 105 to the scene synthesis circuit 106 when the demultiplexing circuit 102 starts reading media data in the scene. For example, when BIFS is used as the scene description data, a duration_changed event of a Movie Texture node or an Audio Clip node is transmitted from the event generation circuit 105 to the scene synthesis circuit 106 and used for determination. The demultiplexing circuit 102 transmits an event issuing command to the event generation circuit 105 when it starts reading moving images and sounds related to each node. The event generation circuit 105 that has received the event issuing command transmits a duration_changed event to the scene synthesis circuit 106. The duration_changed event indicates the duration of a moving image or sound in seconds. When the value is −1, it means that the moving image or sound has not been read yet or cannot be used for some reason.

  For example, in the example of the BIFS data 300 shown in FIG. 3, when the demultiplexing circuit 102 starts reading the moving image object data “test.mpeg”, it transmits an event issuing command to the event generating circuit 105. The event generation circuit 105 that has received the event issuing command transmits a duration_changed event of the Movie Texture node to the scene composition circuit 106.

  If it is determined that an event has been received from the event generation circuit 105 (“Yes” in step S705), the scene synthesis circuit 106 reads the media object data decoded from the media decoding circuit 104 (step S707). In step S708, it is determined whether the scene refers to other media data. If it is determined that there is no reference (“NO” in step S708), the scene synthesis circuit 106 synthesizes the media data read in step S707 together with the scene description data and outputs it to the output device 107 ( Step S709).

  When it is determined that no event has been received from the event generation circuit 105 (“No” in step S705), reading of media data has not started because the line status is congested and the processing capability of the receiving apparatus 101 is low. Or if it cannot be used for some other reason. In this case, the scene synthesis circuit 106 cannot read the media object data decoded from the media decoding circuit 104, waits for an event to be transmitted from the event generation circuit 105 (step S706), and then repeats step S705. Perform the process. Until the event is sent, the scene does not change over time and the field values are not updated. Therefore, in the example of the BIFS data 300 in FIG. 3, the TIMER 304 does not output the fraction_changed event until the time of 500 msec elapses.

  In step S708, if the scene refers to other media data (“Yes” in step S708), the scene composition circuit 106 determines whether or not the start time of the media object data has come. The process proceeds to S704.

  FIG. 8 is a flowchart showing a process in which the scene synthesis circuit 106 of the receiving apparatus 101 synthesizes the scene description data and the media object data when the scene description data has already been read. That is, the process of FIG. 8 is a process following the process shown in FIG.

  First, the scene synthesizing circuit 106 changes the field value of the already-read scene description data over time (step S801). In the example of the BIFS data 300 in FIG. 3, a fraction_changed event is output according to the time when the TIMER 304 has elapsed.

  Next, the scene synthesis circuit 106 determines whether there is unstarted media data in the media data in the scene (step S802). The BIFS data 300 in FIG. 3 includes the Movie Texture node 303, but it is determined in step S704 shown in FIG. 7 that playback has already started, and it is determined that there is no unstarted media data.

  As described above, when it is determined that there is no unstarted media data (“No” in step S802), there is no reference to the media data in the scene, or reproduction of all media data has already started. Therefore, the scene synthesizing circuit 106 synthesizes and outputs the scene description data (step S808).

  If it is determined that there is unstarted media data in the scene (“Yes” in step S802), the scene composition circuit 106 determines whether the current time is the start time of the media data. (Step S803). This determination is the same process as step S704 shown in FIG.

  If it is determined that the current time is not the disclosure time of the media data (“No” in step S803), the scene synthesis circuit 106 does not need to synthesize the media data at the current time, and step S807. Continue the process.

  If it is determined that the current time is the start time of the media data (“Yes” in step S803), the scene composition circuit 106 determines whether the reading of the media data has started. It is determined whether event reception from the event generation circuit 105 has been completed (step S804). This determination is the same processing as step S705 shown in FIG.

  If it is determined that the event reception has been completed, the scene synthesis circuit 106 reads the media object data decoded from the media decoding circuit 104 (step S806), and there is other unstarted media data in the scene. (Step S807), if not, it is combined with the scene description data and output to the output device 107 (step S808).

  On the other hand, when it is determined that the event reception has not been completed, the scene synthesis circuit 106 waits for an event to be transmitted from the event generation circuit 105, similarly to step S706 shown in FIG. 7 (step S805). .

  If there is other unstarted media data in step S807 (“Yes” in step S807), the scene composition circuit 106 determines whether the media data has reached the start time. The process proceeds to S803.

FIG. 9 is a diagram showing an example of a screen on which the multimedia data 400 is reproduced over time in the above-described environment in which delay occurs based on the flowcharts of FIGS. 7 and 8.
As shown in FIG. 9, at the time of 0 msec, the BIFS data 300 is decoded, but nothing is displayed because the decoding of the moving image object data is not completed (screen 600). At the time of 500 msec, the moving image object data Frame0 (401) is displayed with the same scale of the width and height (screen 500). Until the reading of the moving image object data Frame 0 (401) is started, the scene composition circuit 106 does not change the scene even if time elapses (the TIMER 304 does not output a fraction_changed event), so the scale field value of the MOVIE 302 is (1). , 1).

  When the time is 1000 msec, the moving image object data Frame 10 (402) is displayed with a scale of 1.5 times the width and height (screen 501). At the time of 1500 msec, the moving image object data Frame 20 (403) is displayed with the scale of width and height being doubled (screen 502). When the time is 2000 msec, the moving image object data Frame 30 (404) is displayed with the scale of the width and the height being doubled (screen 503).

  As described above, according to the receiving apparatus 101 of the present embodiment, the scene synthesis circuit 106 stops and waits for the scene until the event from the event generation circuit 105 is received even when the start time of the media data is reached. Can do. As a result, as shown in FIG. 9, in order to wait for the decoding process of the moving image frame Frame 0, there is a delay of 500 msec in the reproduction process of the first screen (screen 500), but subsequent screens (screens of screens 501 to 503). ) Can be reproduced in the same manner as in the ideal environment shown in FIG. That is, according to the receiving apparatus 101 of this embodiment, when each object is separated from the multimedia data composed of a plurality of objects and played back, it is managed by media object data such as moving images and audio and scene description data. The composition and reproduction synchronized with the scene to be performed can be reliably performed regardless of the type of communication line, the line condition, and the processing capability of the terminal.

  In the embodiment described above, in the example shown in FIG. 3 and FIG. 4, the scene configuration is only a moving image for easy understanding. However, the media objects constituting the scene are not limited to the moving image. It is possible to use audio, still images, and the like. Therefore, the control target for synchronizing with the scene is not limited to a moving image, and any object that forms a moving image, audio, still image, or the like can be applied.

[Second Embodiment]
In the above-described first embodiment, when the receiving apparatus 101 receives multimedia data, the multimedia data received by the demultiplexing circuit 102 is separated into scene description data and media object data, and the decoding circuits 103 and 104 are respectively separated. The receiving apparatus 1000 according to the second embodiment, which has a configuration different from that configuration, will be described below.

  FIG. 10 is a diagram illustrating a schematic configuration of a receiving apparatus 1000 according to the second embodiment. As shown in FIG. 10, the receiving apparatus 1000 is configured to receive scene description data through a different path (transmission path 100) from the media object data. 10, the same reference numerals are given to the same functions as those in FIG.

  When receiving the scene description data, the receiving apparatus 1000 according to the present embodiment inputs the scene description data to the scene description data decoding circuit 103 via the transmission path 100 (upper side in FIG. 10) different from the media object data. Also, media object data multiplexed via the transmission path 100 (lower side in FIG. 10) different from the scene description data is input to the demultiplexing circuit 102 as in the first embodiment. The input media object data is demultiplexed by the demultiplexing circuit 102 and input to each media decoding circuit 104. The processing flow in the receiving apparatus 1000 of this embodiment is the same as the processing flow of the receiving apparatus 101 in the first embodiment shown in FIGS.

  According to the receiving apparatus 1000 shown in the present embodiment, VRML, SMIL, or the like is used as scene description data, and when reproducing multimedia data having a configuration in which scene description data and encoded media data are not multiplexed into the same stream. Similarly to the object of the first embodiment, it is possible to reliably perform the synthesizing and reproduction of the scene and the media object data regardless of the communication line, the line status, and the processing capability of the terminal.

[Third Embodiment]
In the first embodiment, the event generation circuit 105 is configured to transmit an event to the scene synthesis circuit 106 when the demultiplexing circuit 102 starts reading media data in a scene. A receiving apparatus 1100 according to a third embodiment will be described below.

  FIG. 11 is a diagram illustrating a schematic configuration of a receiving device 1100 according to the third embodiment. As shown in FIG. 11, the event generation circuit 105 of the reception device 1100 is configured to generate an event according to the status of the media data decoding process in the media decoding circuit 114. In FIG. 11, as indicated by reference numerals 100, 102, 103, 105 to 107, the same functions as those in FIG.

  When the processing capability of the receiving apparatus 101 in the first embodiment shown in FIG. 1 is low, the media decoding circuit 104 takes time for the decoding process, so there is a possibility that the scene and the media data cannot be synchronized. Therefore, in this embodiment, the media decryption circuit 114 has a function (issue command means) for transmitting an event issue command to the event generation circuit 105 when the media data decryption processing is completed. The event generation circuit 105 that has received the event issuance command transmits an event to the scene synthesis circuit 106. Further, the processing in the receiving apparatus 1100 of the present embodiment is the same as the processing of the receiving apparatus 101 in the first embodiment shown in FIGS.

  As described above, according to the receiving apparatus 1100 shown in the present embodiment, since the progress of the scene is controlled in consideration of the progress of the decoding process of the media object data, the encoded multimedia data composed of a plurality of objects However, even if it takes time to decode media object data such as video and audio, the synchronized synthesis and playback of scenes and media object data depends on the communication line, line status, and terminal processing capacity. Therefore, it is possible to carry out with certainty.

[Fourth Embodiment]
In the second embodiment described above, the event generation circuit 105 is configured to transmit an event to the scene synthesis circuit 106 when the demultiplexing circuit 102 starts reading the media data in the scene, but is different from the configuration. A receiving apparatus 1200 according to the fourth embodiment having the configuration will be described below.

  FIG. 12 is a diagram illustrating a schematic configuration of a receiving device 1200 according to the fourth embodiment. As shown in FIG. 12, the event generation circuit 105 of the reception device 1200 is configured to generate an event according to the status of the media data decoding process of the media decoding circuit 114. In FIG. 12, as indicated by reference numerals 100, 102, 103, 105 to 107, the same reference numerals are given to the same functions as those in FIG.

  When the processing capability of the receiving apparatus 1000 in the second embodiment shown in FIG. 10 is low, the media decoding circuit 104 takes time for the decoding process, so there is a possibility that the scene and the media data cannot be synchronized. Therefore, in this embodiment, when the media decoding circuit 114 completes the decoding process of the media data, the event generation command is transmitted to the event generation circuit 105, and the event generation circuit 105 that has received the event generation command receives the event to the scene synthesis circuit 106. Is configured to transmit.

  As described above, according to the receiving apparatus 1200 shown in the present embodiment, since the progress of the scene is controlled in consideration of the progress of the decoding process of the media object data, the encoded multimedia data composed of a plurality of objects However, even if it takes time to decode media object data such as video and audio, the synchronized synthesis and playback of scenes and media object data depends on the communication line, line status, and terminal processing capacity. Therefore, it is possible to carry out with certainty.

[Other Embodiments]
In the above-described embodiment, each function in the receiving apparatus is realized by a circuit. However, the present invention is not limited to this. A recording medium in which a program code of software for realizing each function in the receiving apparatus is recorded is provided to the system or apparatus, and the computer (or CPU or MPU) of the system or apparatus stores the program code stored in the recording medium. Each function in the receiving apparatus may be realized by executing reading. In this case, the program code itself read from the recording medium is for realizing the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

  As the recording medium for supplying the program code described above, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like is used. Can do.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an OS (operating system) operating on the computer based on the instruction of the program code. Needless to say, some or all of the actual processing is performed and the functions of the above-described embodiments are realized by the processing.

  Furthermore, after the program code read from the recording medium is written to the memory provided in the function expansion board inserted into the computer or the function expansion unit connected to the computer, the function is determined based on the instruction of the program code. A CPU or the like provided in the expansion board or the function expansion unit may perform part or all of the actual processing, and the functions of the above-described embodiments may be realized by the processing.

A program product such as a computer-readable recording medium in which the above program code is recorded can also be applied as an embodiment of the present invention.
The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

It is a figure which shows the basic composition of the multimedia data receiver 101 as the 1st Embodiment of this invention. It is a figure which shows the example of a data structure of the multimedia data 200 whole in this embodiment. It is a figure which shows the example of description of the scene description data 201 shown in FIG. It is a figure which shows the difference of the decoding and reproduction | regeneration processing by the operation environment with respect to the example of multimedia data containing the BIFS data 300 shown in FIG. 3, and the example of multimedia data. FIG. 5 is a diagram showing an example of a screen over time when multimedia data 400 is decoded and played back at an ideal timing as shown in FIG. 4. FIG. 5 is a diagram illustrating an example of a screen when conventional decoding and reproduction processing is performed on multimedia data 400 in a delay environment in which a delay occurs as illustrated in FIG. 4. 10 is a flowchart showing a process of combining scene description data and media object data by the scene combining circuit when the receiving apparatus 101 receives the scene description data. 10 is a flowchart showing a process of combining scene description data and media object data by the scene composition circuit when the scene description data has already been read. FIG. 9 is a diagram showing an example of a screen on which multimedia data 400 is reproduced with time in the above-described environment in which delay occurs based on the flowcharts of FIGS. 7 and 8. It is a figure which shows schematic structure of the receiver 1000 in 2nd Embodiment. It is a figure which shows schematic structure of the receiver 1100 in 3rd Embodiment. It is a figure which shows schematic structure of the receiver 1200 in 4th Embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Transmission path 101 Multimedia data receiver 102 Demultiplexing circuit 103 Scene description data decoding circuit 104, 114 Media decoding circuit 105 Event generation circuit 106 Scene synthesis circuit 107 Output device 1000, 1100, 1200 Multimedia data receiving apparatus (receiving apparatus) )

Claims (6)

A data processing device that receives multimedia data including one or more encoded moving image and / or audio object data via a network and performs playback processing according to scene description data,
Separating means for separating the received multimedia data in units of the object data;
One or more first decoding means for decoding the plurality of object data separated by the separation means;
Second decoding for decoding the scene description data when the scene description data encoded as a part of the data included in the multimedia data or as independent data through a communication path different from the network is received; And
Event generating means for generating an event in accordance with the timing at which the separation means starts reading the object data or the timing at which the first decoding means decodes the object data;
Based on the scene description data decoded by the second decoding unit according to the timing at which the event generated by the event generating unit is received, the plurality of the decoding units decoded by the first decoding unit A data processing apparatus comprising: scene combining means for performing object data combining processing.   The data processing apparatus according to claim 1, wherein the scene synthesizing unit waits for the synthesizing process until the event generated by the event generating unit is received.   The multimedia data is data conforming to the MPEG (Motion Picture Experts Group) -4 standard, and the scene description data includes BIFS (Binary Format for Scenes), VRML (Virtual Reality Modeling Language, SM, IL When the data is described in either Integration Language (XMT) or XMT (extensible MPEG-4 Textual Format), the event generation means is a video defined in the BIFS, the VRML, the SMIL, or the XMT. Generate predetermined events for nodes related to image or audio The data processing apparatus according to claim 1 or 2, characterized in Rukoto. A data processing method using a data processing apparatus that receives multimedia data including one or more encoded moving image and / or audio object data via a network and reproduces the data according to scene description data. ,
A separation step of separating the received multimedia data in units of the object data;
One or more first decoding steps for decoding the plurality of object data separated in the separation step;
Second decoding for decoding the scene description data when the scene description data encoded as a part of the data included in the multimedia data or as independent data through a communication path different from the network is received; Step,
An event issuing step for issuing an event in accordance with a timing at which reading of the object data is started in the separation step or a timing at which the object data is decoded in the first decoding step;
Based on the timing of receiving the event generated in the event generation step, based on the scene description data decoded in the second decoding step, a plurality of the decoded in the first decoding step A scene synthesis step for performing synthesis processing of object data. A program for a data processing apparatus that receives multimedia data including one or a plurality of encoded moving image and / or audio object data via a network and reproduces the data according to scene description data,
A separation step of separating the received multimedia data in units of the object data;
One or more first decoding steps for decoding the plurality of object data separated in the separation step;
Second decoding for decoding the scene description data when the scene description data encoded as a part of the data included in the multimedia data or as independent data through a communication path different from the network is received; Step,
An event issuing step for issuing an event in accordance with a timing at which reading of the object data is started in the separation step or a timing at which the object data is decoded in the first decoding step;
Based on the timing of receiving the event generated in the event generation step, based on the scene description data decoded in the second decoding step, a plurality of the decoded in the first decoding step A program for causing a computer to execute a scene composition step for performing object data composition processing. A recording medium for recording a program for a data processing device that receives multimedia data including one or more encoded moving image and / or object data related to sound via a network and performs playback processing according to scene description data There,
A separation step of separating the received multimedia data in units of the object data;
One or more first decoding steps for decoding the plurality of object data separated in the separation step;
Second decoding for decoding the scene description data when the scene description data encoded as a part of the data included in the multimedia data or as independent data through a communication path different from the network is received; Step,
An event issuing step for issuing an event in accordance with a timing at which reading of the object data is started in the separation step or a timing at which the object data is decoded in the first decoding step;
Based on the timing of receiving the event generated in the event generation step, based on the scene description data decoded in the second decoding step, a plurality of the decoded in the first decoding step A computer-readable recording medium having recorded thereon a program for causing a computer to execute a scene synthesis step for synthesizing object data.

Images