JP2003199030A - Data processor, data processing server, data processing system, method of controlling data processor, method of controlling data processing server, computer program and computer-readable storage medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a data processor, a data processing server and a data processing system which facilitate data editions effectively utilizing features of the object-based coding or generating multimedia data coded on the object base. <P>SOLUTION: A multimedia editing-reproducing terminal 201 instructs a multimedia edition server 202 to edit multimedia data composed of a plurality of object data. A communication client 214 receives layout information of the multimedia data. A reproducer 212 displays the acquired layout information, an edition designator 211 designates one of the displayed layout information, and the designated layout information is outputted via the communication client 214. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

TECHNICAL FIELD The present invention relates to MPEG-4 (Mo
ving Picture Expert Group Phase 4) Data processing device, data processing server, data processing system, data processing device for editing multimedia data having encoding and object description functions and other multimedia data Control method of a data processing server, a computer program, and a computer-readable storage medium.

[0002]

2. Description of the Related Art In recent years, with the spread of digital data broadcasting and digital video, it has become possible for individuals to easily handle multimedia data. Also,
With the widespread use of mobile communication devices such as mobile phones, it has become easier to carry out wireless communication on the go, and the environment for connecting to a network has been improved. Therefore, in response to the increasing familiarity with multimedia, mobile communication devices capable of handling multimedia data, such as so-called third-generation mobile phones, have appeared.

However, the conventional portable terminal device is not necessarily sufficient in processor capacity and storage area in order to reduce its size and weight. Therefore, in order to process multimedia data, it is generally performed by, for example, reducing the resolution of an image or reducing the frame rate (number of frames per unit time) of a moving image. .

[0004]

However, the conventional mobile terminal device has a problem in that it does not perform processing that makes full use of the characteristics of the multimedia data handled. Therefore, in order to take advantage of the advantages of portable terminal devices, which are typified by small and lightweight mobile communication devices, they have been made smaller and lighter and have limited data processing capacity, memory, or communication. Various techniques have been devised to process multimedia data within the limited bandwidth of. As one example of such a device, MP
An example is the use of a low bit rate, high compression multimedia coding scheme such as EG-4.

Further, in the object-based coding represented by MPEG-4, one multimedia data can be composed of a plurality of objects. Data in various formats such as moving images, still images, computer graphics (or vector graphics), text, and audio can be applied to these objects. Then, these objects can be held together with the layout information of the tree structure.

However, the present specifications regarding multimedia data are in a very complicated format,
There is a problem that it is difficult to edit multimedia data under limited restrictions.

Conventionally, when editing such complicated multimedia data, it is possible to request a third party who specializes in data editing equipment to store and edit the data in a personal computer or the like with few restrictions. It was common. Along with this, services such as performing edit processing on a server on the Internet having a data edit function and returning the edit result to the client, and related technologies have been studied.

At present, a still image is sent to a server on the Internet, and composite data is created by combining the still image with a photo frame, music, a message or the like, and a URL (Uniform Resource L) indicating the processing result is created.
ocators, RFC 1738) has been put into practical use, a service that realizes simple multimedia e-mail. In other words, due to the restriction of the processing capacity of the terminal device, the user has been performing the data editing operation with a device having another processing capacity connected by a communication line or the like.

As described above, according to the current technology, the process of causing the server on the Internet to edit the data and returning the processing result is a still image or a simple rectangular moving image in HTML (Hyper Text Markup Language). It is possible with regard to the degree of synthesizing in. However, it is not a process of editing multimedia data coded by the object-based coding method and acquiring the edited result by the object-based coding method.

The present invention has been made in consideration of such a situation, and even a portable terminal having a limited processing capability can connect to a server capable of editing multimedia data to instruct editing. The object of the present invention is to provide a data processing device, a data processing server, and a data processing system that can easily perform data editing utilizing the characteristics of object-based encoding and generation of object-based encoded multimedia data. And

[0011]

In order to solve the above-mentioned problems, the data processing apparatus according to the present invention can be connected via a network to a data processing server which edits multimedia data composed of predetermined object data. A data processing device, receiving means for receiving a template for editing processing for multimedia data from the data processing server, designating means for designating the template received by the receiving means, and for the multimedia data. In order to perform a desired editing process, it is characterized by including information indicating the template designated by the designating unit and transmitting unit for transmitting the multimedia data to the data processing server.

The data processing apparatus according to the present invention is a data processing apparatus connectable via a network to a data processing server that generates multimedia data composed of predetermined object data. Receiving means for receiving from the data processing server a template of edit processing for generating, designating means for designating the template received by the receiving means, and for designating desired multimedia data by the designating means. It is characterized by comprising information indicating the designated template and transmission means for transmitting predetermined object data to the data processing server.

Further, the data processing server according to the present invention is
A data processing server connectable via a network to a data processing device for generating multimedia data composed of predetermined object data, wherein the data processing device includes multimedia data and editing processing for the multimedia data. Receiving means for receiving the information indicating the template, separating means for respectively separating the object data constituting the multimedia data received by the receiving means, object data separated by the separating means and received by the receiving means. By using the information indicating the template of the edited processing,
And editing means for generating new multimedia data.

The data processing server according to the present invention is
A data processing server connectable to a data processing device for generating predetermined object data via a network, wherein the data processing device generates predetermined object data and desired multimedia data from the predetermined object data. And a receiving unit that receives information indicating a template of an editing process for editing, and an editing unit that generates multimedia data using the object data received by the receiving unit and the information indicating the template of the editing process. It is characterized by

A data processing device control method according to the present invention is a data processing device control method connectable to a data processing server for editing multimedia data composed of predetermined object data via a network. A receiving step of receiving a template of editing processing for multimedia data from the data processing server, a designating step of designating the template received in the receiving step, and a desired editing processing for the multimedia data. To this end, the information processing apparatus further comprises information indicating the template designated in the designation step and a transmission step of transmitting the multimedia data to the data processing server.

A data processing apparatus control method according to the present invention is a data processing apparatus control method connectable via a network to a data processing server for generating multimedia data composed of predetermined object data. A receiving step of receiving a template of an editing process for generating multimedia data from the data processing server, a specifying step of specifying the template received in the receiving step, and generating desired multimedia data. And a transmission step of transmitting information indicating the template designated in the designation step and predetermined object data to the data processing server.

A data processing server control method according to the present invention is a data processing server control method connectable via a network to a data processing device for generating multimedia data composed of predetermined object data. A receiving step of receiving predetermined multimedia data from the data processing device and information indicating a template of an editing process for the multimedia data, and object data constituting the multimedia data received in the receiving step. A separating step of separating, and an editing step of generating new multimedia data using the object data separated in the separating step and the information indicating the template of the editing process received in the receiving step. Characterize.

A data processing server control method according to the present invention is a data processing server control method connectable to a data processing device for generating predetermined object data via a network. A receiving step of receiving predetermined object data and information indicating a template of an editing process for generating desired multimedia data from the predetermined object data, an object data received in the receiving step and the editing process. And an editing step of generating multimedia data using information indicating a template.

The computer program according to the present invention is a computer program for controlling a data processing device connectable via a network to a data processing server for editing multimedia data composed of predetermined object data. Then, the code of the receiving step of receiving the template of the editing process for the multimedia data from the data processing server, the code of the specifying step of designating the template received in the receiving step, and the desired code for the multimedia data. In order to perform the editing process, it includes information indicating the template designated in the designation process and a transmission process code for transmitting the multimedia data to the data processing server.

The computer program according to the present invention is a computer program for controlling a data processing device connectable via a network to a data processing server for generating multimedia data composed of predetermined object data. , A code of a receiving step of receiving a template of an editing process for generating multimedia data from the data processing server, a code of a specifying step of designating the template received in the receiving step, and generating multimedia data. In order to do so, it is characterized by including information indicating the template designated in the designation step and a code of a transmission step of transmitting predetermined object data to the data processing server.

Further, the computer program according to the present invention is a computer program for controlling a data processing server connectable via a network to a data processing device for generating multimedia data composed of predetermined object data. A predetermined multimedia data, a code of a receiving step of receiving information indicating a template of an editing process for the multimedia data from the data processing device, and object data constituting the multimedia data received in the receiving step. Using the code of the separation process for separating each of the objects, the object data separated in the separation process, and the information indicating the template of the editing process received in the reception process,
And a code of an editing process for generating new multimedia data.

A computer program according to the present invention is a computer program for controlling a data processing server connectable to a data processing device for generating predetermined object data via a network,
A code of a receiving process for receiving from the data processing device predetermined object data and information indicating a template of an editing process for generating desired multimedia data from the predetermined object data, and the code received in the receiving process. It is characterized in that it includes object data and information of an editing process for generating multimedia data using the information indicating the template of the editing process.

[0023]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings.

First, prior to the description of the embodiments, the object-based coding will be described. MPEG-4
In object-based coding represented by, one multimedia data can be composed of a plurality of objects. Data in various formats such as moving images, still images, computer graphics (or vector graphics), text, and audio can be applied to these objects. Then, in the multimedia data, information about a combination between these objects is held as layout information of a tree structure (hereinafter, referred to as “scene”).

FIG. 1 is a schematic diagram for explaining multimedia data described by object-based coding. As shown in FIG. 1, the multimedia data 101 includes a moving image 102, a background still image 103, computer graphics 104 described by vector graphics, text 105 that is a character string that conveys a message, and voice 106. It is composed of a plurality of objects and a scene 107 that gives information for synthesizing these objects. The structure of the multimedia data does not necessarily have to be the structure shown in FIG. For example, the text 105 may not exist, or different types of objects not illustrated in FIG. 1 may be added.

As described above, in the object-based coding, each object exists independently, and these objects are stored in one or more files. For example, a system in which all objects are stored as a single file, or objects are stored in a certain unit in a different format, and are grouped together as if they were one file during playback. Some systems are regenerated.

For example, in the case of MPEG-4, a file format called MP4 is standardized as one of the storage methods (ISO / IEC 14496-1 / PDAM1 "MPEG-4 ver".
sion2 Intermedia Format -MP4 "). In the following, regarding the object storage method, it is assumed that all objects are stored in one file for the sake of simplicity.

When editing multimedia data by object-based coding as shown in FIG. 1, not only the function of editing each object itself but also the editing of the scene 107 that gives the layout of each object is taken into consideration. There is a need.

The handling of multimedia data by object-based coding is obviously more complicated than the conventional handling of data in which moving images and still images are given alone. Therefore, the editing of the scene 107 involves editing the positions of the individual objects and the synchronization between them, which may be difficult depending on the processing capability of the terminal. Also, generating multimedia data requires more processing power than reproducing multimedia data.

<First Embodiment> First, a simple editing is carried out by utilizing a multimedia editing server device different from a terminal device for reproducing multimedia data, which takes advantage of the feature of object-based coding. The form will be described.

FIG. 2 is a block diagram for explaining the configuration of a multimedia editing system for editing multimedia data according to the first embodiment of the present invention.
As shown in FIG. 2, the multimedia editing system according to the present embodiment has a multimedia editing / playback terminal device 201 corresponding to a mobile terminal having a certain limitation in capability.
And a multimedia editing server device 202 which is a main body for performing complicated editing, and a network 203 such as the Internet for connecting both devices.

Multimedia editing / playback terminal device 201
Is an editing instruction section 211 for giving instructions such as editing,
A reproduction unit 212 for reproducing / displaying the edit progress, and a client edit processing unit 213 for performing a simple edit process.
And a communication client unit 214 for various data communication with the multimedia editing server device 202. On the other hand, the multimedia editing server device 202
Is a communication server unit 2 for receiving communication such as editing instructions.
21 and a server edit processing unit 222 for performing edit processing of multimedia data.

Multimedia editing / playback terminal device 201
In the above, the editing instruction unit 211 displays the GUI (Graphi
cal user interface), etc., which is responsible for accepting user operations and instructing editing operations. In the present embodiment, it is a device for performing general processing except functions related to multimedia data reproduction. is there.

The reproducing unit 212 is a device for reproducing multimedia data, and is a device for decoding and displaying multimedia data. The client edit processing unit 213 also includes an edit instruction unit 211 and a reproduction unit 21.
The multimedia editing / playback terminal 201 has various data editing functions connected to the multimedia editing / playback terminal apparatus 201.

Further, the communication client section 214 is connected to the client edit processing section and performs various operations related to the editing operation instructed by the edit instructing section 211 on the network 2.
It has a function of sending to the multimedia editing server device 202 via 03. Furthermore, the communication client unit 21
4 is a function for receiving multimedia data from the multimedia editing server device 202 to a reproducing unit 212 for presenting the editing result and editing process performed by the multimedia editing server device 202 in a recognizable form to the user. Is a device for performing communication processing.

On the other hand, the multimedia editing server device 20
2, the communication server unit 221 uses the network 20
3, which is connected to the multimedia editing / playback terminal device 20.
1 is a device for performing a communication processing function on the multimedia editing server device 202 corresponding to the communication client unit 214 in FIG. In addition, the editing unit 222 performs multimedia editing / editing processing on multimedia data.
This is a device for performing based on an instruction from the editing instruction unit 211 of the reproduction terminal device 201.

The network 203 is, specifically,
A wireless communication system represented by a communication environment such as a PDC (Personal Digital Cellular) system or a wireless LAN (Lo
cal area network), or various communication environments such as a wired LAN typified by Ethernet (registered trademark) (Ethernet (registered trademark)).

Next, the electrical configurations of the multimedia editing / playback terminal device 201 and the multimedia editing server device 202 will be described. FIG. 3 is a block diagram simply showing the electrical configuration of the multimedia editing / playback terminal device 201. As shown in FIG. 3, the multimedia editing / playback terminal device 201 includes a transmitter 301 and a receiver 30.
2, or a communication device 303 in which these are integrated, an input / output operation unit 304, a display unit 305, and a calculation / control unit 3
06, a memory unit 307, a code processing unit 308, and an external data input / output unit 309.

When the multimedia editing / playback terminal device 201 is, for example, a mobile communication device, the transmitter 301 and the receiver 302, or the communication device 303 in which these are integrated, communicates with the mobile communication station. It provides a communication means for performing. Further, when the multimedia editing / playback terminal device 201 is connected by, for example, a LAN or the like, it provides a communication means such as an IP protocol.

Information to be communicated by the transmitter 301 and the receiver 302, or the communication device 303 in which these are integrated includes, for example, multimedia data itself and information processing requests such as sending instructions and editing instructions thereof. is there.

The input / output operation unit 304 is a pointing device, a keypad or the like for the user to operate.
It may also include a lamp or a speaker for notifying the operation status. That is, the input / output operation unit 304 provides the user with specific operation means for multimedia editing and other device operation functions.

The display unit 305 is an LCD (Liquid Crystal).
The display function represented by Display) is provided, and the user can see the operation content and the reproduction result of the multimedia data through the display function. Incidentally, in the reproduction of multimedia data, sound data such as music and voice may be expressed at the same time, but here, in order to simplify the description, those expression functions are also represented on the display unit 305. There is.

The arithmetic / control unit 306 is a microprocessor, a clock, a bus controller, etc., and controls the entire multimedia editing / playback terminal device 201. The control of the entire multimedia editing / playback terminal device 201 performed by the calculation / control unit 306 includes, for example, control of transfer of display data to the display unit 305 and refresh instruction.

The memory unit 307 is a storage medium for storing multimedia data, a basic program for controlling the multimedia editing / playback terminal device 201, an application program for providing an editing function, and the like. As the memory unit 307, an internal storage area of a semiconductor memory, a storage medium represented by a magnetic disk, or the like can be considered. In general, a nonvolatile semiconductor memory or the like is often used in a mobile terminal device or the like, but a combination of various types of storage media described above may be used.

The memory unit 307 is the memory unit 307.
May be composed of a memory built in the device and a semiconductor disk such as a removable IC card. If it is a semiconductor disk such as a removable IC card,
Input / output of data is performed by an external data input / output unit 309 described below.

The coding processing unit 308 performs coding processing of multimedia data, and performs compression / expansion processing for coding / decoding multimedia data, for example. By having a function related to this compression / expansion processing, for example, the multimedia editing / playback terminal device 201
When the camera has a camera function, the captured information can be stored in the memory unit 307 through the code processing unit 308. The external data input / output unit 309 is a device for inputting / outputting an external storage medium such as a semiconductor disk, a magnetic disk, or a magneto-optical disk.

FIG. 4 shows the multimedia editing server device 2
It is a block diagram which shows the electric constitution of 02 simply. As shown in FIG. 4, the multimedia editing server device 202
Is a transmitter 401 and a receiver 402, or a communication device 403 in which these are integrated, a calculation / control unit 404, a memory unit 405, a code processing unit 406, and an external storage device 4.
07 and. This configuration is basically similar to the multimedia editing / playback terminal device 201, but is different in that the input / output operation unit and the display unit are not necessarily provided. However, the input / output operation unit and the display unit may be provided for maintenance of the device or other reasons.

A transmitter 401 and a receiver 402, or a communication device 403 in which these are integrated, provides a communication means with the multimedia editing / playback terminal device 201. For example, the network 203 is the Internet, and the multimedia editing / playback terminal device 201 is an IS
When connected via P (Internet Service Provider), it is an Ethernet connection device connected to the backbone of the Internet.

The calculation / control unit 404, the memory unit 405, and the external storage device 407 are respectively provided with multimedia editing and
This corresponds to the arithmetic / control unit 306, the memory unit 307, and the external data input / output unit 309 in the reproduction terminal device 201, and is essentially equivalent to an information processing device such as a general computer.

The code processing unit 406 performs a coding process of multimedia data, and may have a compression / expansion function for coding / decoding the multimedia data, for example. It may also have an arithmetic processing function necessary for editing multimedia data. The code processing unit 308 shown in FIG. 3 and the code processing unit 406 shown in FIG. 4 both execute the code processing of multimedia data.

In many cases, processing of multimedia data requires a large amount of resources of the information processing apparatus, and conventionally, a dedicated code processing section has been used for such a purpose. However, due to recent improvements in the processing capacity of the arithmetic / control unit, it has become possible to cause the arithmetic / control unit to execute processing of multimedia data. Therefore, in the present invention, the code processing unit 308 and the code processing unit 406.
Does not have to be an essential component.

In addition, the multimedia editing server device 20
2 is not a single device, but for the purpose of load balancing,
It may be composed of a plurality of devices. A parallel processor and the like are also included in this range.

As described above, the multimedia editing / playback terminal device 201 and the multimedia editing server device 20.
2 has been described, the edit instruction unit 211, the reproduction unit 212, the client edit processing unit 213, the communication client unit 214, and the communication server unit 221 in the block diagram of the logical configuration shown in FIG. The server edit processing unit 222 corresponds to the logical operation configuration when the actual processing operation according to the present invention is performed.

For example, when the editing instruction unit 211 performs the processing of the editing instruction, the calculation / control unit 306 takes in data from the external data input / output unit 309 and responds to the input from the input / output operation unit 304. The processing is executed and the display unit 305
The operation content is displayed on.

Next, processing for editing multimedia data using the multimedia editing system including the multimedia editing / playback terminal device 201 and the multimedia editing server device 202 according to the present embodiment as described above. The procedure will be described.

FIG. 5 is a flow chart for explaining the processing procedure of the entire multimedia editing system according to this embodiment. A user of this system for editing multimedia data is required to use the multimedia editing / playback terminal device 2
First, the multimedia data to be edited is selected using 01 (step S501). At this time, it is assumed that, for example, data such as the multimedia data 101 shown in FIG. 1 is selected in the multimedia editing / playback terminal device 201.

Next, the multimedia editing server device 202 connected via the network 203 confirms the access from the multimedia editing / playback terminal device 201, and then edits the data from the database such as the external storage device 407. A list of templates capable of recalling the editing result serving as the template at the time of is read and sent to the multimedia editing / playback terminal device 201 (step S502). The multimedia editing / playback terminal device 201 obtains information regarding the list of transmitted templates (step S503).

The template means multimedia data configuration information including scenes indicating the positions of individual objects constituting the multimedia data and preset objects such as computer graphics.

Next, the user selects a desired template such as a template close to the layout of the multimedia data that the user wants to edit from the list of acquired templates (step S504). When the template is selected, the multimedia editing / playback terminal device 201 uniquely identifies the target multimedia data selected in step S501 and the selected template to the multimedia editing server device 202. Information is sent (step S505).

The information for uniquely identifying the template may be something like a template ID made up of an arbitrary character string or a URL (Uniform Resource Loc).
ators, RFC 1738).

The multimedia editing server device 202
Information for uniquely identifying the multimedia data and the selected template is received (step S506). Then, in the multimedia editing server device 202, the editing process of the multimedia data by the template is started.

In the editing process, first, the sent multimedia data is separated for each object (step S507). This process includes so-called objects
A splitter, which is a separation function, is used to separate various objects such as moving images, still images, computer graphics (or vector graphics), text, audio, and scenes.

In the object-based coding system,
Objects are not separated by type, but by individual object units. That is, when one multimedia data includes three moving image data, it is divided into three moving image objects.

This separated object group can be applied to the object structure described in the template selected in step S504. Therefore, an application process for synthesizing the object configuration of the selected template and the separated object is performed (step S508).

At this time, it should be noted that the structure of the object set as the template does not always match the structure of the object of the multimedia data to be edited. Therefore, in the template application processing in step S508, the operation processing of the template application status may be performed from the multimedia editing / playback terminal device 201 with the operation of the user (step S509).

Then, a new multimedia data is edited by synthesizing the object group and the template (step S510). The new multimedia data, which is the result of this editing, is sent back to the multimedia editing / playback terminal device 201 (step S511).

Multimedia editing / playback terminal device 201
Then, the new multimedia data transmitted from the multimedia editing server device 202 is received (step S
512), it becomes possible to perform processing such as reproduction and storage on the data (step S513).

FIG. 6 is a diagram schematically showing an editing process of multimedia data edited by the processing procedure of FIG. As shown in FIG. 6, in step S501, the multimedia data 601 to be the editing source is selected in advance. Next, a list 602 of templates indicating the contents to be edited is selected and acquired by the multimedia editing / playback terminal device 201 in step S503.
In the example shown in FIG. 6, two templates of ID = 1 and ID = 2 are displayed as a list.

In the example shown in FIG. 6, the structure of the data shown in the multimedia data 601 and the template list 602 is simple. When the multimedia data is separated into objects, the object group 603 includes a moving image A, a computer graphics A, and a background still image A.
And a scene A that constitutes them. Also,
An object group 604 forming the selected template (here, ID = 2) includes an object node including a moving image, a computer graphics B, a background still image B, and a scene B forming them. Composed of. The object node means a place where an object originally exists, and the object group 604 does not actually exist at that place.

In the editing result 605 obtained by editing these objects, the moving image A included in the editing-source multimedia data 601 is displayed for this object node.
The objects in are replaced. Then, in the synthesis result 606 obtained by performing the object synthesis, the template other than the object of the moving image A included in the multimedia data 601 as the editing source is the selected template.

Here, an application process (step S508) for synthesizing the object configuration of the selected template and the separated objects in the flowchart shown in FIG. 5 and a synthesizing process of the object group and the template ( The step S510) will be further described.

In general, object-based coding is M
In the case of PEG-4, the scene is VRML (Virtual
A language called Reality Modeling Language) is used. In addition, as an object-based encoding scene, XML (eXtensible Markup Language) or HTML is used.
(HyperText Markup Language), SMIL (Synchroni
zed Multimedia Integration Language) or MHE
G (Multimedia and Hypermedia information coding E
xperts Group) or the like may be used. Furthermore, in digital broadcasting, BML (Broadcast Markup Language) is used.
Are used, and these various languages may be used to describe the scene.

Since it is difficult to explain all the above-mentioned languages using this embodiment, here,
A case will be described where object-based encoding is performed in MPEG-4, VRML is used as scene information, and object information is internally defined by a structured description language such as XML.

The multimedia data which is the editing source is M
When described by PEG-4, the data of each object is associated with a scene by a value called ES_ID. For example, ES_ID 1 is a moving image, 2 is a still image, ES_ID 3 is a second still image, and ES_ID 4 is a scene information itself.

In addition, OD (Object
Descriptor) is associated with each other, and the type of each object is StreamType.
It is standardized to be set with a value called e (ISO / IEC 14496-1).

As described above, when the multimedia data to be edited is described in MPEG-4, the information of each object is referred to by referring to this information when object division of the multimedia data is performed. You can learn about the type and which part of the scene the object is bound to. For example, the moving image A in the example shown in FIG.
The reamType is described as 4.

On the other hand, in the template, a scene is described by VRML and information related to the scene is described by a structured description language. These may be described as one file, or may exist as data on different memories. FIG. 7 is a diagram showing an example in which a part of the description of the scene expressed in VRML is extracted.

A scene 701 describes a part of the scene expressed in VRML, and% PLACE_HOL
In DER%, a reference to the object to be actually embedded is described. The information 702 related to the scene includes an X
Information of the scene expressed in ML is described, and similarly, in% PLACE_HOLDER%, a reference to an object to be actually embedded is described. Also, Dec
As in the odeConfig tag, Stream
Since 4 is set in Type, it can be seen that the type of this object is 4.

Therefore, by referencing each other with this StreamType, it is determined how the object included in the multimedia data to be edited should be set to the object node existing in the template. be able to.

It is necessary to give sufficient consideration to the matching of the type and number of objects between the individual objects of the multimedia data to be edited and the object nodes included in the template. For example, if the number of object nodes applicable to the object included in the template is larger than the number of object nodes of the editing source, there is a possibility that objects may be missing in the process of performing object composition. Also, in the opposite case, not all objects can be applied to the template. For this reason, FIG.
In the procedure of the editing process shown by, the operation process of the template application status accompanied by the user's operation (step S509)
In the part (1), conversation processing or the like for solving the above-mentioned problem may be performed.

Next, the order of processing performed between the multimedia editing / playback terminal device 201 and the multimedia editing server device 202 operated by the user will be described. Of the processing procedure shown in FIG. 5, step S501.
The order of the selection of the editing-source multimedia data performed in step S503 and the acquisition of the template list information performed in step S503 are arbitrary, and may be the reverse of the above-described embodiment. Even if the source multimedia data is selected after selecting a template from the template list information, it is clear that those processes are independent of each other and do not affect each other's processes. Is.

Alternatively, the template may be selected after the multimedia data to be edited is first sent to the multimedia editing server device 202 to separate the objects. In the case of such a processing order, since the object division has already been performed before selecting the template, only present the template to the user that matches the number of objects of the multimedia data to be edited. Will also be possible.

As described above, the multimedia editing server device 202 side stores the template of the multimedia data by the object base coding, and the multimedia data to be edited is sent to the multimedia editing / playback terminal device 201. By performing the editing process in this manner, it is possible to obtain the effect of easily editing the object-based encoded data. In addition, by performing the editing process based on the scene instead of performing the encoding process of individual objects, it is possible to reduce the processing content with a large processing load such as the encoding of a moving image as much as possible. Effect is also obtained.

<Second Embodiment> Next, in the case of easily editing multimedia data utilizing a feature of object-based coding, a multimedia editing server device different from a reproducing / editing terminal device is used. An embodiment will be described.

In the first embodiment, the case of selecting and editing the multimedia data by the already-created object-based coding has been described, but the case of generating the multimedia data from individual objects of the present invention is also described. Application is possible.

FIG. 8 is a schematic diagram showing the structure of the multimedia editing system according to the second embodiment. In FIG. 8, a photographing terminal 801 is a photographing device such as a digital camera or a video camera, and can output a photographed scene or the like as digital data as moving image data 802 or still image data 803. Shooting terminal 80
The moving image data 802 and the still image data 803 output from No. 1 are easily input to the multimedia editing / playback terminal device 201.

However, when the input digital data is an individual object that does not include a scene,
The editing of multimedia data by the created object-based coding as shown in the first embodiment cannot be applied. Instead, the multimedia editing / playback terminal device 201 can send the moving image data 802 and the still image data 803 to the multimedia editing server device 202 as individual objects.

The digital data sent to the multimedia editing server apparatus 202 in this way is the same as the state in which the processing for separating the multimedia data for each object in the flowchart shown in FIG. 5 (step S507) is completed. It is in a state.

Therefore, if the data sent 805 from the multimedia editing / playback terminal device 201 can be used as an object that matches the template 806, object synthesis is performed to edit the edited data by multimedia editing / playback. Sending 807 to the terminal device 201 makes it possible to generate multimedia data by object-based coding. The multimedia data created in this way can be reproduced and displayed 808, and so-called rich multimedia data can be generated by a simple method by template selection.

Next, a description will be added about the types of objects in the process of generating multimedia data from these individual objects. The multimedia editing server device 20 of the multimedia editing system shown in FIG.
An object encoder 809 is connected to 2. The object encoder 809 may be included as one component of the multimedia editing server device 202.

The object encoder 809 is a module for adapting each object sent from the multimedia editing / playback terminal device 201 to the multimedia editing server device 202 to the object base coding system.

For example, assume that the object-based coding system to be generated is MPEG-4. The type of the object transmitted from the multimedia editing / playback terminal device 201 is not handled by MPEG-4 Motion-JP
When the object is in the EG format, the object in the Motion-JPEG format needs to be encoded in a format handled by MPEG-4. In such a case, the object encoder 809 is used. Object encoder 809
Then, for example, it is possible to operate by decoding the data in the Motion-JPEG format into a so-called bitmap format and encoding the data as an input into the MPEG-4 moving image object format.

Further, in order to perform the above-mentioned processing, it is necessary to add a function of judging the format of each input object and deciding what format the object should be converted to. Is. FIG. 9 is a diagram showing an example of an object encoder judgment table used for judging the format of an object. In FIG. 9, an input format 901 is a type of a corresponding input object, and is a MIME (Multipurpose Internet Mail Ext.
ension, RFC 2045 to 2049) Type and file format can be specified.

The output format 902 is the type that the object encoder 809 should output, and similarly, the MIME
It may be specified by type or any proprietary format. The object encoder 903 specifies a function entry of the object encoder, a value to be given to a judgment processing routine for calling a function, a library name, and the like. By using the object encoder judgment table as shown in FIG. 9, each object sent from the multimedia editing / playback terminal device 201 to the multimedia editing server device 202 can be converted in an optimum format.

The object encoder determination table shown in FIG. 9 includes additional information such as processing parameters and a format in object encoding, that is, MPEG.
-4, the above-mentioned StreamType or the like may be included. If the output format 902 is uniquely determined in advance by the object encoder, the output format 9
02 may be omitted as an implicit understanding.

Further, in the determination of the input format 901, it is possible to interpret and judge individual objects sent from the multimedia editing / playback terminal device 201 to the multimedia editing server device 202. Values may be sent from the media editing / playback terminal device 201 to the multimedia editing server device 202.

The file format and bit stream for storing objects include a data format unique to each object type, and the method of interpreting the object data and determining the object type is generally It is a method that is performed in. Further, it is generally considered that the multimedia editing / playback terminal device 201 which sends the object knows the type of the object to be sent, and it is not difficult to notify the type when the object is sent.

As described above, according to the present invention, not only the multimedia data that has been created in advance but also individual objects are copied from the multimedia editing / playback terminal device 201 to the multimedia editing server device 20.
By sending the data to the second terminal and synthesizing the objects using the template, even the multimedia editing / playback terminal device whose multimedia data generation capability is not sufficient can easily generate the multimedia data. In addition, the multimedia editing server device 2 is previously
The present invention can be applied even when the multimedia data has been transmitted to 02.

<Third Embodiment> The above-mentioned first and second embodiments
In the above embodiment, the editing is completed by sending the editing result of editing the multimedia data by the multimedia editing server device 202 back to the multimedia editing / playback terminal device 201. Here, when the edited result is not required on the side of the multimedia editing / playback terminal device 201, the load on the multimedia editing / playback terminal device 201 may be further reduced in some cases.

For example, assuming that the final acquisition of the editing result is performed by another terminal device having a faster communication line, only the editing operation and the confirmation of the editing result are performed by the multimedia editing / playback terminal device 201. There are cases.

FIG. 10 is a block diagram showing the structure of the multimedia editing system according to the third embodiment. As shown in FIG. 10, in the third embodiment, as the transmitter 401 in the multimedia editing server device 202,
This can be realized by providing a transmitter equipped with a stream device 1001 such as RTP (Realtime Streaming Protocol, RFC 1889). The stream client device 1002 provided in the terminal device 201 provides a reception function corresponding to the stream device 1001.

In the edit result transmission process (step S511) in the flowchart shown in FIG. 5, it is not difficult at all to use multimedia data as the edit result with a stream device such as RTP. Such a stream device itself is generally widely used, and the feature of the present invention is that these are applied to a multimedia editing / playback terminal device, a multimedia editing server device, and a multimedia editing system using them. Especially.

Here, an example of the effect obtained by implementing the multimedia editing system according to the present invention will be specifically described. When the editing result is received as a file, the longer it is, the longer the communication time is required, and the multimedia editing / playback terminal device 201 is loaded. Therefore, by performing streaming, the editing result cannot be immediately received, but the editing result can be viewed without waiting for the file transfer time.

The effect of such a mechanism will be specifically described. Therefore, as the multimedia data that becomes the template, a preset object different from the object set by the user is embedded,
Assume that the size of the object was 10 Mbytes. For example, when an object is a moving image, the size of such an object is common sense.

On the other hand, the object to be edited by the user is a 300 kbyte still image, and the user may edit the still image captured by the user as a multimedia object combined with a moving image embedded in advance. Let's expect.

For example, when the communication capability of the multimedia editing / playback terminal device 201 possessed by the user is about 64 kbps, the expected time for transmitting the above-mentioned object is about 40 seconds. The time required to acquire it will exceed 20 minutes.
Although the communication speed has been increasing year by year with the progress of technology, such a time difference is not preferable.

Therefore, when a streaming device is used to acquire the editing result, such acquisition time is at least greatly reduced, and the restriction of the storage device required for the multimedia editing / playback terminal device 201 is greatly relaxed. To be done. That is, it is possible to obtain the effect that the user is expected to perform a light editing process, and the editing process can be performed more smoothly on a mobile terminal having a limited processing capability.

Further, the judgment as to how to reproduce the edited result may be made by confirming whether the edited multimedia itself is required by the multimedia editing / playback terminal device 201 or only the edited result. It should be done in the interest of the user. Such a determination can be realized, for example, by providing options on an operation screen for editing instructions.

<Other Embodiments> The network and device configuration for connecting communication devices in the present invention are not necessarily limited to the fields of the Internet and mobile communication. The present invention may be applied to various networks such as a so-called home LAN and a wireless communication network such as Bluetooth which has a limited communication distance.

Further, a software program for realizing the functions of the above-described embodiments is provided to a computer in an apparatus or system connected to the various devices so as to operate the various devices so as to realize the functions of the above-described embodiments. It is also within the scope of the present invention that a system (apparatus or MPU) of the system or apparatus is operated by operating the various devices according to a stored program by supplying the code.

Further, in this case, the program code itself of the software realizes the function of the above-described embodiment, and the program code itself and means for supplying the program code to the computer, for example, such program code. The storage medium storing the above constitutes the present invention.

As a storage medium for storing the program code, for example, a floppy (registered trademark) disk,
Hard disk, optical disk, magneto-optical disk, CD-
ROM, magnetic tape, non-volatile memory card, RO
M or the like can be used.

Further, not only the functions of the above-described embodiments are realized by executing the supplied program code by the computer, but also the OS (operating system) in which the program code is operating in the computer, or other Needless to say, the program code is also included in the embodiment of the present invention even when the above-described embodiment is realized in cooperation with the application software or the like. Further, the supplied program code is stored in the memory provided in the function expansion board of the computer or the function expansion unit connected to the computer, and then the CPU provided in the function expansion board or the possible expansion unit based on the instruction of the program code. Needless to say, the present invention also includes a case where a part or all of the actual processing is performed and the functions of the above-described embodiments are realized by the processing.

As described above, according to the present invention, even a multimedia editing / playback terminal device having a limited processing capacity can be used as an object-based coding system by interlocking with the multimedia editing server device. It is possible to easily perform editing that takes advantage of the features and generation of multimedia data by an object-based coding method. In addition, it is easy to check the result of the editing process even for multimedia data having a relatively large size.

Further, according to the present invention, communication / conversation is made with a multimedia editing server device capable of performing editing processing of multimedia data to be edited, and the multimedia editing server device is requested to edit the multimedia data. To do so, it is possible to send multimedia data. In addition, by receiving the edited multimedia data in the multimedia editing server device and reproducing / storing the edited multimedia data received from the multimedia editing server device, the object can be processed even if the processing capacity is restricted. Editing that makes use of the characteristics of the base encoding method can be performed.

Further, according to the present invention, the MPEG-4 (Moving Picture Expert Group) which defines the contents of the editing process.
It is possible to store editing layout information of multimedia data by a similar multimedia coding method having a similar object description function or a method called simply object-based coding method. Then, the editing layout information of these multimedia data is sent to the multimedia editing / playback terminal device as a list, and the multimedia data and editing request sent from the multimedia editing / playback terminal device are received, Data can be edited / generated and the result can be sent to the multimedia editing / playback terminal device.

Further, according to the present invention, the result edited in the multimedia editing server device is streamed, so that the result can be easily confirmed in the multimedia editing / playback terminal device having a limited communication capacity or processing capacity. be able to.

Furthermore, according to the present invention, in the multimedia editing / playback terminal device, the editing layout information of the multimedia data provided by the multimedia editing server device is selected, and the selected multimedia data is edited. The layout information and the multimedia data to be edited, or the individual multimedia data that can be converted into individual objects in the object-based encoding method are sent to the multimedia editing server device, and further, in the multimedia editing server device. The multimedia data can be edited by dividing the sent multimedia data into objects, determining the type of the object, applying the object according to the corresponding type, and synthesizing the objects.

Further, according to the present invention, there is provided a mechanism for effectively operating the present invention in a multimedia editing system in which a multimedia editing / playback terminal device and a multimedia editing server device are connected via a network. However, the user will be able to perform more flexible and highly expandable multimedia editing.

[0120]

As described above, according to the present invention, even in a mobile terminal having a limited processing capability, by connecting to a server capable of editing multimedia data and instructing to edit the object-based code. It is possible to easily perform data editing that takes advantage of the characteristics of encoding, and generation of object-based encoded multimedia data.

[Brief description of drawings]

FIG. 1 is a schematic diagram for explaining multimedia data described by object-based coding.

FIG. 2 is a block diagram illustrating a configuration of a multimedia editing system that edits multimedia data according to the first embodiment of the present invention.

FIG. 3 is a block diagram simply showing an electrical configuration of a multimedia editing / playback terminal device 201.

FIG. 4 is a block diagram simply showing an electrical configuration of a multimedia editing server device 202.

FIG. 5 is a flowchart for explaining a processing procedure of the entire multimedia editing system according to the present embodiment.

6 is a diagram schematically showing an editing process of multimedia data edited by the processing procedure of FIG.

FIG. 7 is a diagram showing an example in which a part of the description of a scene expressed in VRML is extracted.

FIG. 8 is a schematic diagram showing a configuration of a multimedia editing system according to a second embodiment.

FIG. 9 is a diagram showing an example of an object encoder determination table used for determining the object format.

FIG. 10 is a block diagram showing a configuration of a multimedia editing system according to a third embodiment.

[Explanation of symbols]

201 Multimedia editing / playback terminal device 202 multimedia editing server device 211 Edit instruction section 212 Playback section 213 Client edit processing unit 214 Communication Client Unit 221 Editing server section 222 Server edit processing unit

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁷ Identification code FI theme code (reference) H04N 7/24 H04N 5/92 H (72) Inventor Tomomi Fukuoka 3-30-2 Shimomaruko, Ota-ku, Tokyo Canon Inc. F term (reference) 5C053 FA14 FA29 GA11 GB37 JA16 KA03 LA06 LA15 5C059 KK37 KK39 MA00 MB21 PP01 PP04 PP19 PP20 PP28 SS08 SS10 UA02 5C064 BB10 BC16 BC23 BC25 BD02 BD07 BD08

Claims (25)

[Claims] 1. A data processing device connectable via a network to a data processing server for editing multimedia data composed of predetermined object data, wherein the template of the editing process for multimedia data is used as the data processing server. Receiving means for receiving from the receiving means, designating means for designating the template received by the receiving means, and information indicating the template designated by the designating means for performing desired editing processing on the multimedia data, A data processing device, comprising: a transmitting unit that transmits the multimedia data to the data processing server. 2. A data processing apparatus connectable via a network to a data processing server for generating multimedia data composed of predetermined object data, wherein a template for editing processing for generating multimedia data is provided. Receiving means for receiving from the data processing server, designating means for designating the template received by the receiving means, and information for indicating the template designated by the designating means and predetermined information for generating desired multimedia data. And a transmitting means for transmitting the object data of 1. to the data processing server. 3. The data processing server according to claim 1, further comprising receiving means for receiving new multimedia data edited in the data processing server, and storage means for storing the multimedia data. Data processing equipment. 4. The data processing apparatus according to claim 3, further comprising reproducing means for reproducing the multimedia data. 5. The data processing apparatus according to claim 4, further comprising streaming means for reproducing the received multimedia data by streaming. 6. The multimedia data according to claim 1, wherein the multimedia data is data described by MPEG-4 encoding, encoding having an object description function, or other object-based encoding. The data processing device according to item 1. 7. The plurality of object data, wherein the multimedia data includes at least one object of moving image, still image, audio, text, computer graphics or vector graphics, and layout information of a tree structure related to the combination of the objects. 7. The data processing device according to claim 1, wherein the data processing device is data composed of 8. The object data, wherein the multimedia data is object data composed of objects of the same type among object data of any one of moving image, still image, audio, text, computer graphics and vector graphics, and the object data. 7. The data processing device according to any one of claims 1 to 6, wherein the data processing device is composed of layout information of a tree structure related to the combination. 9. The data processing according to claim 1, wherein the multimedia data is audiovisual data obtained by combining a moving image and audio, or a still image and audio. apparatus. 10. A data processing server connectable via a network to a data processing device for generating multimedia data composed of predetermined object data, the multimedia data from the data processing device, and the multimedia. Receiving means for receiving information indicating a template for editing processing of data, separating means for separating object data constituting the multimedia data received by the receiving means, object data separated by the separating means, and the A data processing server, comprising: an editing unit that generates new multimedia data using the information indicating the template of the editing process received by the receiving unit. 11. A data processing server connectable to a data processing device for generating predetermined object data via a network, the predetermined object data from the data processing device,
Receiving means for receiving information indicating a template for editing processing for generating desired multimedia data from the predetermined object data; and object data received by the receiving means and information indicating a template for the editing processing. A data processing server, comprising: an editing unit configured to generate multimedia data by using the data processing server. 12. A data storage means for storing predetermined object data is further provided, wherein the editing means uses the object data stored in the data storage means and the object data received by the receiving means to newly add data. 12. The data processing server according to claim 10 or 11, wherein the multimedia processing device generates various multimedia data. 13. The apparatus according to claim 10, further comprising a layout storage unit that stores information about the editing process template, and a sending unit that sends information about the editing process template to the data processing device. The data processing server according to any one of 12 to 12. 14. A first judging means for judging an encoding method of the object data received by the receiving means, and a second judging means for judging an encoding method of the object data corresponding to the information indicating the template of the editing process. 12. The data processing server according to claim 11, further comprising: determining means, wherein the editing means generates new multimedia data according to a coding method of the corresponding object. 15. A data processing system, comprising: the data processing device according to claim 1; and the data processing server according to claim 10. 16. A data processing device according to claim 2,
A data processing system comprising: the data processing server according to claim 11. 17. A method of controlling a data processing device, which is connectable via a network to a data processing server for editing multimedia data composed of predetermined object data, wherein a template of editing process for multimedia data is provided. A receiving step of receiving from the data processing server, a designating step of designating the template received in the receiving step, and a template designated in the designating step for performing desired editing processing on the multimedia data. A method of controlling a data processing device, comprising: the information shown and a transmitting step of transmitting the multimedia data to the data processing server. 18. A method of controlling a data processing device connectable via a network to a data processing server for generating multimedia data composed of predetermined object data, the editing process for generating multimedia data. A receiving step of receiving the template from the data processing server, a designating step of designating the template received in the receiving step, and a template designated in the designating step in order to generate desired multimedia data. A method of controlling a data processing device, comprising: a transmitting step of transmitting information and predetermined object data to the data processing server. 19. A method of controlling a data processing server connectable via a network to a data processing device for generating multimedia data composed of predetermined object data, the predetermined multimedia data being transmitted from said data processing device. And a receiving step of receiving information indicating a template of an editing process for the multimedia data, a separating step of separating the object data forming the multimedia data received in the receiving step, and a separating step of separating the object data. A method of controlling a data processing server, comprising: an editing step of generating new multimedia data using the object data and the information indicating the template of the editing processing received in the receiving step. 20. A method of controlling a data processing server, which is connectable to a data processing device for generating predetermined object data via a network, comprising: predetermined object data from the data processing device;
A receiving step of receiving information indicating a template of an editing process for generating desired multimedia data from the predetermined object data, an object data received in the receiving step, and information indicating a template of the editing process. A method of controlling a data processing server, comprising: using the editing process to generate multimedia data. 21. A computer program for controlling a data processing device connectable via a network to a data processing server for editing multimedia data composed of predetermined object data, the editing process for multimedia data. A receiving step code for receiving the template from the data processing server, a specifying step code for specifying the template received in the receiving step, and performing the desired editing process on the multimedia data, A computer program comprising: information indicating a template designated in a designation step; and a code of a transmission step of transmitting the multimedia data to the data processing server. 22. A computer program for controlling a data processing device connectable via a network to a data processing server for generating multimedia data composed of predetermined object data, the multimedia program generating multimedia data. For receiving the editing process template from the data processing server, a specifying process code for specifying the template received in the receiving process, and a process for specifying the template in the specifying process for generating multimedia data. A computer program comprising information indicating a designated template and a transmission process code for transmitting predetermined object data to the data processing server. 23. A computer program for controlling a data processing server connectable via a network to a data processing device for generating multimedia data composed of predetermined object data, the predetermined multimedia data , A code of a receiving step of receiving information indicating a template of an editing process for the multimedia data from the data processing device, and a code of a separating step of separating the object data forming the multimedia data received in the receiving step. And a code of an editing process for generating new multimedia data using the object data separated in the separating process and the information indicating the template of the editing process received in the receiving process. A computer program to do. 24. A computer program for controlling a data processing server connectable to a data processing device for generating predetermined object data via a network, comprising: predetermined object data and a desired one of the predetermined object data. The code of the receiving step of receiving the information indicating the template of the editing process for generating the multimedia data from the data processing device, the object data received in the receiving step, and the information indicating the template of the editing process. A computer program comprising: a code of an editing process for generating multimedia data by using the computer program. 25. A computer-readable storage medium having the computer program according to any one of claims 21 to 24 stored therein.