JP2006019964A - Multimedia execution system and multimedia file executing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a multimedia execution system and a multimedia execution method capable of operating external equipment in synchronization with the reproduction of multimedia information. <P>SOLUTION: An SMAF file 20 incorporates performance sequence information for controlling performance reproduction, and external equipment operation control sequence information for controlling the operation of the external equipment. A sequence 26 reproduces sequence information stored in respective sequence tracks 22-25, based on synchronization information, or the like stored in a master track 25. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a multimedia execution system that handles various types of multimedia information such as sounds, images, and texts.

  One example of the integrated handling of multimedia information from the user side is content browsing software on the Internet (hereinafter referred to as a browser). The user can handle multimedia information such as sound, image, text and the like on one screen by operating the browser on the personal computer screen. Normally, each piece of information is stored in a location defined by a different path in the server, and the browser retrieves each piece of information individually and reproduces it on one screen. All the information on the Internet can be handled by HTML (Hyper Text Markup Language), and the browser outputs the information by interpreting the HTML.

  In addition, software written in a language different from HTML and reproducing sounds and images (especially moving images) synchronously has been put into practical use on the Internet. The file structure handled by this software integrates sound information and image information and can be handled as one file. However, in the conventional system that handles sound information and image information in one file format in this way, each piece of information is only reproduced independently from the beginning. There was an inconvenience that fine synchronization could not be achieved. In view of such a background, a multimedia execution system that enables fine synchronization control between pieces of information has been proposed (see, for example, Patent Document 1).

JP 2002-55782 A

  According to the multimedia execution system described above, fine synchronization control between each piece of information can be performed at a terminal (such as a personal computer or a mobile phone) that reproduces multimedia information. However, among users who use the multimedia execution system, Then, there is a demand for operating an external device in synchronization with reproduction of multimedia information. However, the present situation is that a multimedia system that meets such a demand has not yet been provided.

  The present invention has been made in view of the circumstances described above, and provides a multimedia execution system and a multimedia file execution method capable of operating an external device in synchronization with reproduction of multimedia information. With the goal.

  In order to solve the above-described problem, the multimedia execution system according to the present invention includes performance sequence information for controlling performance playback, external device operation sequence information for controlling operation of an external device, the performance playback, and the external A first storage unit that stores a multimedia file that incorporates synchronization information for synchronizing operation of the device, a sequencer that executes a sequence operation of the multimedia file, and a communication unit that performs communication with the sequencer. A second storage unit that stores an application program that controls execution of a media file, and a program execution unit that executes the application program are provided.

  According to such a configuration, it is possible to control the operation of an external device (for example, a monkey toy; see FIG. 5) in synchronization with performance playback (for example, performance of a ringing melody) by the multimedia execution system (for example, a mobile phone). It becomes. Thereby, for example, it is possible to provide a service full of entertainment such that a monkey toy plays a cymbal in synchronization with the reproduction of an incoming melody by a mobile phone.

  Here, in the above configuration, the multimedia file includes a performance sequence track for storing performance sequence information for controlling performance reproduction and external device operation sequence information for controlling operation of the external device. A mode in which an operation sequence track to be stored and a master track in which synchronization information of each sequence track is stored is preferable.

  In the above configuration, the external device sequence information includes external device identification information indicating a destination of the external device, and the external device operation sequence information read by the sequence operation is included in the external device identification information. It is further preferable to further include a transmission means for transmitting to the external device shown.

  As described above, according to the present invention, it is possible to operate an external device in synchronization with the reproduction of multimedia information.

  Embodiments according to the present invention will be described below with reference to the drawings.

A. First Embodiment FIG. 1 is a hardware configuration diagram of a multimedia execution system 100 according to the present embodiment. In this embodiment, a mobile phone is assumed as the hardware of the multimedia execution system 100. However, if each component as shown in FIG. 1 is provided, a personal computer, a PHS (Personal Handyphone System: registered trademark), a PDA (Personal Digital Assistance).

  The execution control unit 1 includes a CPU, a ROM, a RAM, and the like, and performs execution of sequencer programs and application programs, input / output control, and the like. For the execution control unit 1, a sequencer program storage unit 2 that stores a sequencer program, an application program storage unit (second storage unit) 3 that stores an application program, and a storage unit (first storage unit) that stores a multimedia file. Storage unit) 4 is connected. In the present embodiment, a SMAF (Synthetic music Mobile Application) in which a performance sequence track, a drawing sequence track, an audio sequence track, an external device operation control sequence track, a master track, and a content information storage unit are incorporated as a multimedia file in the same file. Format) file is assumed, but the present invention is not limited to this, and various types of files can be applied (described later).

The storage unit (first storage unit) 4 stores the first to nth SMAF files, and one of the SMAF files is selected according to input information from the application program storage unit 3 or the user.
A sound source device 5, a display device 6, an audio device 7, and an external device operation control device 13 are connected to the execution control unit (program execution unit) 1.

To the sound source device 5, performance sequence information in the SMAF file is input and converted into a performance signal. The performance sequence information is MIDI sequence information in the present embodiment, and a MIDI sound source device is used as the sound source device 5.
The display device 6 receives drawing sequence information in the SMAF file. As will be described later, the drawing sequence information is one of text, binary image, and arbitrary image sequence information, and the display device 6 converts these information into image signals.

Audio sequence information in the SMAF file is input to the audio device 7, and the audio device 7 converts the input audio sequence information into an audio signal. For example, when the audio sequence information is ADPCM information, the audio device 7 converts the ADPCM information into an analog audio signal.
External device operation control sequence information in the SMAF file is input to the external device operation control device (transmission means) 13. The external device operation control sequence information is information for controlling the operation of an external device that communicates with its own terminal using IrDA (Infrared Data Association), Bluetooth (registered trademark), or the like (details will be described later). The external device operation control device 13 wirelessly transmits (or wire-transmits) the external device operation control sequence information to the designated external device.

  The musical sound / sound output unit 8 outputs the synthesized output of the sound source device 5 and the audio device 7 from the speaker 8a. The monitor 9 displays the image output from the display device 6 on the display screen. An input unit 10 and a communication unit 11 are further connected to the execution control unit 1. The input unit 10 inputs various parameter setting instructions supplied from the operation unit 12. The communication unit 11 exchanges various data with various servers and other terminals (not shown) connected to a mobile communication network or the Internet.

FIG. 2 is a diagram illustrating a software configuration of the multimedia execution system 100.
The SMAF file 20 includes a content information storage unit 21, a performance sequence track 22, a drawing sequence track 23, an audio sequence track 24, an external device operation control sequence track 24a, and a master track 25. These are integrated into one file. Built in.
The content information storage unit 21 stores information regarding the contents of the entire SMAF file 20. The performance sequence track 22 stores performance sequence information, the drawing sequence track 23 stores drawing sequence information, the audio sequence track 24 stores audio sequence information, and the external device operation control sequence track 24a stores external device operation control sequence information. Remember. The master track 25 stores synchronization information of the sequence tracks 22 to 24a. This synchronization information is information for synchronizing the reproduction of each sequence information (for example, synchronizing the performance reproduction based on the performance sequence information and the operation of the external device based on the external device operation control sequence information). The master track 25 stores synchronization information of the sequence tracks 22 to 24a. The track 25 itself is also a sequence track.

  The sequencer 26 controls the sequence operations of the performance sequence track 22, the drawing sequence track 23, the audio sequence track 24, the external device operation control sequence track 24a, and the master track 25. Each sequence track is configured by combining an event and a duration, and the duration specifies a time interval between events. As a result, it is possible to know the execution start time of the event by accumulating the time indicated by the duration from the beginning of the sequence track. Further, even if it takes time to process the event itself, it does not affect the passage of time on the sequence data. The passage of time can be expressed by duration regardless of event processing. The master track 25 stores time axis direction control information such as a pause (stop) event, a branch event, and a repeat event as synchronization information. When these events occur, the master track 25 instructs the sequencer 26 to perform sequence control such as pause, branch and repeat. For example, when a pause event occurs, the sequence operation of each of the sequence tracks 22 to 25 is temporarily stopped. On the other hand, when a branch event occurs, the sequence operation point of each sequence track simultaneously branches to a specific position. When a repetitive event occurs, a certain sequence operation is repeatedly executed for a predetermined period.

  The sequence output of the performance sequence track 22 is input to the sound source device 27 and output as a sound. The output of the drawing sequence track 23 is output to the display device 28 and drawn on the display monitor. The output of the audio sequence track 24 is output to the audio device 29 and output as sound. Further, the sequence output of the external device operation control sequence track 24a is input to the external device operation control device 24b and output to the external device 40 as external device operation control sequence information.

  The sequencer 26 is controlled by the application program 30. The application program 30 may be anything as long as it can control the sequencer 26. The application program 30 outputs a start / stop signal and a status read signal to the sequencer 26. The sequencer 26 notifies the application program 30 of the status (state). For example, when a pause event occurs as an event of the master track 25, the sequencer 26 sets the sequence operation to a pause state (pause state) and notifies the application program 30 of the status. Lead. The status content in this case is pause (pause). The application program 30 performs a predetermined display for the user or waits for an input operation from the user via the user interface 31 according to the content of the status. In addition, various data is exchanged between the server and other terminals via the communication interface 32. When an event that exits the pause state occurs (this event is determined by the application program 30. For example, when there is a user input), the application program 30 instructs the sequencer 26 to restart. In this way, data communication between the sequencer and the application program is performed.

FIG. 3 shows the data structure of the sequence track.
As described above, the sequence data is expressed by describing event E and duration D in combination. The data string starts from event E, and EOS which is sequence end data is placed at the end of the data. The lifetime represents the effective length of the event. For example, in the case of performance sequence information, it is a pronunciation time. Duration D specifies the time interval between events. Therefore, the start time of a specific event can be determined by accumulating the duration value from the beginning of the data. For example, the start time of event 3 is the time obtained by adding the added value of durations 1 and 2 to time 0. Further, in order to branch from the start time of event 3 to event 1, a value obtained by adding duration 1 and duration 2 is subtracted from the start time of event 3. By such a method, the sequence operation of each sequence track can be arbitrarily controlled. The master track 25 describes this control content, that is, synchronization information of each sequence track. In this embodiment, the sequence data describes the event E and the duration D alternately. However, the sequence data may not necessarily be described alternately.

  FIG. 4 shows schematic operations of the sequencer 26 and the application program 30 in FIG. When the process is started, the sequencer 26 is initialized (step 100), and then waits until a sequence start command is received from the application program 30 (step 101). When this sequence start command is received (step 200, step 101: Y), the sequence operation in the sequencer 26 starts (step 102), and the occurrence of an event in the master track 25 is monitored (step 103). The event of the master track 25 is referred to herein as a checkpoint event. When this checkpoint event occurs, the sequencer 26 sends a status notification to the application program 30 and transmits the contents of the checkpoint event (step 104). The application program 30 receives this status at step 201 and performs processing according to the content (step 202). For example, if the checkpoint event is a pause event, the application program 30 performs a process of waiting for input from the user in response to the pause event. Alternatively, specific data is downloaded (or uploaded) from the server or the like via the communication interface 32 in response to the pause event. The application program 30 further gives a predetermined instruction to the sequencer 26 in accordance with the processing of step 202. That is, each sequence track is controlled in accordance with the content input from the user or data from the server or the like. The sequencer 26 performs processing corresponding to the instruction from the application program 30 in step 105. In the sequencer 26, after the above processing is performed, if the sequence is not finished, the operation from step 103 is performed again. In the application program 30, if the program is not finished, the process returns to step 201 again. .

<Specific Operation of Multimedia Execution System 100>
FIG. 5 is a diagram for explaining the operation of the multimedia execution system 100, and FIG. 6 is a diagram for explaining the reproduction timing of each sequence information. In the following description, the mobile phone 100 is assumed as the multimedia execution system 100, and the monkey toy 40 is assumed as the external device 40.
The performance sequence track 22 in the SMAF file mounted on the mobile phone 100 stores MIDI sequence information for playing an incoming melody in response to an incoming call from the outside (see FIG. 6).

  On the other hand, the external device operation control sequence track 24a in the SMAF file controls the operation of the monkey toy 40 according to the performance of the ringing melody (such as the monkey toy dancing according to the performance of the ringing melody). The external device operation control sequence information is stored (see FIG. 6). The external device operation control sequence information includes identification information (external device identification information) for identifying the monkey toy 40. This identification information is information indicating the destination (address) of the monkey toy 40, for example. The sequencer 26 can transmit the external device operation control sequence information to the monkey toy 40 by referring to the identification information.

When the mobile phone 100 receives an incoming call from the outside, the sequencer 26 of the mobile phone 100 receives a sequence start command from the application program 30. Upon receiving the sequence start command, the sequencer 26 sequentially reads MIDI sequence information, external device operation control sequence information, and synchronization information from the performance sequence track 22, the external device operation control sequence track 24a, and the master track 25, respectively, and starts the sequence operation. (See FIG. 6).
The MIDI sequence information read by the sequencer 26 is sequentially supplied to the sound source device 27 and output from the speaker as a ringing melody sound (see FIG. 2).

  On the other hand, the external device operation control sequence information read by the sequencer 26 is supplied to the external device operation control device 24b (see FIG. 2). The external device operation control device 24b refers to the identification information included in the external device operation control sequence information, and identifies the external device that is the transmission destination of the sequence information. Then, the external device operation control device (transmission means) 24b transmits the external device operation control sequence information to the external device 40 (here, the monkey toy 40) indicated by the identification information using wireless communication or the like. When the monkey toy 40 receives the external device operation control sequence information from the mobile phone 100, the monkey toy 40 controls the operation of each unit (for example, the operation of hitting a cymbal) based on the received external device operation control sequence information. In the above example, the case where the sequencer 26 sequentially reproduces the MIDI sequence information, the external device operation control sequence information, and the synchronization information based on the sequence start command supplied from the application program 30 has been described. The timing at which information (audio sequence information or the like) is reproduced can be appropriately set or changed in the SMAF file.

  As described above, according to the multimedia execution system 100 according to the present embodiment, the external device operation control sequence information for controlling the operation of the external device 40 in the multimedia file (SMAF file in the present embodiment) is stored. By incorporating it, it becomes possible to control the operation of the external device 40 in synchronization with the reproduction of the multimedia file. Thereby, for example, it is possible to perform control such that a monkey toy (external device) plays a cymbal in synchronization with the reproduction of the incoming melody by a mobile phone (multimedia execution system).

B. Although the embodiment of the present invention has been described above, the above embodiment is merely an example, and various modifications can be made without departing from the spirit of the present invention. As modifications, for example, the following can be considered.

<Modification 1>
In the above-described embodiment, the monkey toy that performs a cymbal performance or the like in synchronization with the reproduction of the incoming melody by the mobile phone is exemplified as the external device. The present invention can be applied to all electronic devices capable of wireless communication (or wired communication) with mobile phones such as radio control cars, MIDI musical instruments such as electronic guitars and keyboards. Here, when applied to a MIDI musical instrument, the same melody (or different melody) as the incoming melody can be played with the tone color of the musical instrument in synchronism with the reproduction of the incoming melody by the mobile phone.

<Modification 2>
In the present embodiment, the case where the operation of the external device is controlled in synchronization with the reproduction of the incoming melody by the mobile phone has been described. However, the operation of the external device is controlled without causing the mobile phone to play the incoming melody. Also good. Specifically, when an incoming call is received from the outside, the mobile phone is stored in the external device operation control sequence track 24a for a predetermined period (for example, until the end of the incoming call) with the incoming call as a trigger. Continue to send external device operation control sequence information to external devices. Since subsequent operations can be described in the same manner as in the present embodiment, description thereof is omitted. As described above, the operation of the external device may be controlled by using the incoming call to the mobile phone as a trigger.

<Modification 3>
In the above-described embodiment, the external device operation control sequence track 24a is provided in the SMAF file (see FIG. 2), and the external device operation control sequence information is stored in the track. Instead of providing a new one, it may be possible to respond by appropriately embedding external device operation control sequence information in sequence information (for example, performance sequence information, option sequence information, etc.) of another sequence track. If the response of the external device is slow, the external device operation control sequence information is forwarded to the external device and transmitted, so that the operation of the external device is controlled in synchronization with the reproduction of the incoming melody by the mobile terminal. You may do it.

<Modification 4>
Further, in the above-described embodiment, the case where the operation of one external device is controlled in synchronization with the reproduction of the ringing melody of the mobile terminal has been described. However, the operation of a plurality of external devices may be controlled.
FIG. 7 is a diagram for explaining the operation of the multimedia execution system 100 according to the modification 4. FIG. 8 is a diagram for explaining the reproduction timing of each sequence information according to the modification 4. In the following description, an alarm clock (hereinafter abbreviated as ringtone clock) 100 corresponding to an incoming melody is assumed as the multimedia execution system 100, an air conditioner is assumed as the first external device 40a, and a television is assumed as the second external device 40b. To do.

  The ringtone clock 100 has a function of playing an incoming melody in accordance with an incoming melody performance instruction transmitted from the outside. The performance sequence track 22 in the SMAF file mounted on the ringtone clock 100 stores MIDI sequence information for playing an incoming melody in response to an incoming call from the outside.

  On the other hand, in the external device operation control sequence track 24a in the SMAF file, the first external device for controlling the operation of the first external device (such as on / off control of the air conditioner switch) according to the performance of the ringing melody. Device operation control sequence information and second external device operation control sequence information for controlling the operation of the second external device 40b (such as on / off control of a TV switch) in accordance with the performance of the ringing melody are stored. ing. The external device operation control sequence information (see FIG. 8) will be described in detail. The first external device operation control sequence information is sequence information for turning on the air conditioner at the same time when the ringing melody is started. The second external device operation control sequence information is sequence information for turning on the television switch after a predetermined time has elapsed since the reproduction of the incoming melody was started (for example, after a time of 5 bars has elapsed since the reproduction was started). It is.

  The first external device operation control sequence information and the second external device operation control sequence information include first identification information indicating the destination of the first external device and second identification information indicating the destination of the second external device, respectively. It is. The sequencer 26 reads the first external device operation control sequence information and the second external device operation control sequence information at the timing as shown in FIG. To the external device) and the television 40b (second external device). As a result, the air conditioner 40a is turned on simultaneously with the start of reproduction of the incoming melody by the ringtone clock 100, and the television 40b is turned on after a predetermined time has elapsed. Thereby, the user can be awakened with good feeling. In the above example, the operation of each external device is controlled after a while. For example, the operation of each external device is controlled at the same time, for example, the air conditioner and the TV switch are turned on simultaneously with the start of the ringtone playback. Also good.

<Modification 5>
Since the SMAF file 20 according to this embodiment and the modification described above is independent of the sequencer 26 and the application program 30, the SMAF file 20 is recorded on an arbitrary recording medium (CD-ROM or the like) and distributed. Or distributed via a communication network such as the Internet.

It is a figure which shows the structure of the multimedia execution system which concerns on this embodiment. It is a figure which shows the software structure of the multimedia execution system which concerns on the same embodiment. It is a figure which shows the data structure of the sequence track | truck concerning the embodiment. It is a figure which shows schematic operation | movement of the sequencer and application program which concern on the embodiment. It is a figure for demonstrating operation | movement of the multimedia execution system which concerns on the embodiment. It is a figure for demonstrating the reproduction timing of each sequence information which concerns on the embodiment. It is a figure for demonstrating operation | movement of the multimedia execution system which concerns on the modification 4. It is a figure for demonstrating the reproduction timing of each sequence information which concerns on the modification.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 ... Multimedia execution system, 40 ... External device, 30 ... Application program, 26 ... Sequencer, 20 ... SMAF file, 22 ... Performance sequence track, 24a ... External device Operation control sequence track, 24b ... external device operation control device, 26 ... sequencer.

Claims (4)

Multimedia incorporating performance sequence information for controlling performance playback, external device operation control sequence information for controlling operation of an external device, and synchronization information for synchronizing the performance playback and operation of the external device A first storage unit for storing files;
A sequencer for executing a sequence operation of the multimedia file;
A second storage unit for storing an application program for performing execution control of the multimedia file by communicating with the sequencer;
A multimedia execution system comprising: a program execution unit that executes the application program. The multimedia file includes a performance sequence track in which performance sequence information for controlling performance playback is stored,
An external device operation control sequence track for storing external device operation control sequence information for controlling the operation of the external device;
2. The multimedia execution system according to claim 1, further comprising a master track in which the synchronization information is stored. The external device operation control sequence information includes external device identification information indicating a destination of the external device,
2. The multimedia execution system according to claim 1, further comprising: a transmission unit configured to transmit the external device operation control sequence information read by the sequence operation to an external device indicated by the external device identification information. Performance sequence information for controlling performance playback, external device operation control sequence information for controlling operation of an external device, and synchronization information for synchronizing the performance playback and operation of the external device are incorporated. The process of entering multimedia files into the sequencer;
A program execution unit executing an application program for controlling execution of the multimedia file by communicating with the sequencer;
The sequencer comprises a step of executing a sequence operation of the multimedia file based on the synchronization information.

Images