JPH05143708A - Multimedia processing system - Google Patents

Abstract

PURPOSE:To absorb asynchronism between the timing of a sound output and that of an image display output in an application software by synchronizing the display time of display data and the output time of sound. CONSTITUTION:This system in provide with a computer 1101 to control multimedia data, internal timer 1102 housed in the computer for informing current time of the computer 1101, and hard disk 1103, etc., storing the control data of a processing object and a program expressing a control instruction. As the lapse of time after starting ringing a sound, table data re provided while making the identification number of the display data correspondent to the display time history of data and by using these table data, the display time of the display data is synchronized to the output time of sound. Therefore, the application software can absorb the asynchronism between the timing of the sound output for prepared CD-ROM data and the display output timing of images, and the media of sound, animation and moving images, etc., can be easily synchronized.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a multimedia processing system, and more particularly to a multimedia processing system having an improved method for timing audio, animation and moving images of multimedia data, particularly data in a read-only storage device.

[0002]

2. Description of the Related Art Conventionally, a read-only optical disk device (hereinafter referred to as "CD-ROM") is generally used as a read-only storage device. The output time of the sound of the CD-ROM is
It is the time until the sound is output from the speaker after the output instruction is issued, and the command issue time and the overhead such as the seek time of the CD-ROM drive are the main elements. In the case of sound output, a PLAY command is used to play a note or song starting from the performance start address by sending the performance start address.
When the PLAY command is issued from the computer, the CD-RO
The M side understands the command and outputs the corresponding song to the amplifier and speaker. Therefore, since the computer CPU is released after the PLAY command is executed, the computer CPU is hardly loaded. On the other hand, media such as text, figures, and still images displayed on the display place a heavy load on the computer CPU. The display time of media such as text, figures, and still images is the time until the screen is displayed after the display instruction is issued. The display time is affected by the number of bytes of data and the addition of the CPU by another process, and varies on the time axis. In addition, even though the data creation method is different for voice, animation, video, etc.,
When handling multimedia with a computer, it is required to use it simultaneously on the basis of a time scale.

[0003]

However, in the conventional system of this type, for example, "The Multi-media CD-R is used.
OM for Educational Use "(IEEE Transactions
on Consumer Electr-onics, Vol.1, CE-33, No.
CD-ROM as described in No. 4, November 1987).
It was difficult to synchronize the sound data and the output of the display screen before the pre-mastering process of. Therefore, conventionally, a very inefficient method has been adopted in which, after making one sheet, the timing is finely adjusted, the CD-ROM is burned again, and the adjustment is performed again. In the CD-ROM creating process, the pre-mastering process is time-consuming and costly, and the timing of the created data cannot be finely adjusted. As another method of finely adjusting the timing of the change of the media by hand, there is also a method of performing burning on a Write-Once / CD-ROM many times and performing trial and error. However, in addition to requiring a special machine for Write-Once / CD-ROM, burning the Write-Once / CD-ROM is time-consuming and costly, albeit better than the premastering process. Met. Also,
As another method, a device for simulating a CD-ROM drive using a large-capacity hard disk could relatively easily perform the simulation, but this requires a special device. there were.
As described above, in the related art, there is no method in which the fine adjustment of the sound and image timing of the created CD-ROM data can be absorbed by the application software side. The present invention has been made in view of the above circumstances, and an object of the present invention is to solve the above-mentioned problems in the prior art by
It is an object of the present invention to provide a multimedia processing system capable of synchronizing, on application software, synchronization inconsistencies during reproduction of media such as sounds, moving images, and still images recorded in a D-ROM.

[0004]

The above object of the present invention is to provide a computer for controlling multimedia data, an internal clock stored in the computer for notifying the computer of the current time,
Storage means for storing a program representing control data to be processed and a control command, and sound source data (music, voice, etc.) output controlled according to the command from the computer, multimedia data such as images, characters, graphics, etc. In a multimedia processing system having a stored read-only storage device, an amplifier for outputting sound, a speaker, and a display for displaying multimedia data, an identification number of display display data is stored in an arbitrary storage device. The present invention is achieved by a multimedia processing system characterized by comprising table data in which display time histories of the display data are associated with each other, and by using the table data, the display time of the display display data and the output time of audio are synchronized.

[0005]

In the multimedia processing system according to the present invention, after instructing the output of sound data, the time when the data is displayed and the animation management table having the number of the data are referred to and compared with the time of the internal clock of the computer. This outputs the animation data that should be displayed at the current time. That is, in an animation that takes a long time to display, what is to be output is skipped and read, and in an animation that displays quickly, the animation is synchronized with the internal clock of the computer while waiting for output. On the other hand, since the sound data is output along the time axis after the performance starts,
In addition to the address information in the CD-ROM, a rewritable sound information management table in which the song number and the song address of the CD-ROM sound information are written is provided on the rewritable storage device in the system. It is possible to easily synchronize the output timing of the sound data with the animation and the moving image by adjusting and recording the address matching the timing in the sound information management table.

[0006]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The principle of the present invention will be briefly described below, and then embodiments of the present invention will be described in detail with reference to the drawings. The principle of the present invention is that a CD-R in which a moving image and a sound are recorded in synchronization with each other.
When creating an OM, first, data for displaying a background image and text data for voice recording is created, and this data is stored in, for example, a hard disk in a multimedia processing system. In the operation of creating the data for creating the CD-ROM, as shown in FIG.
An explanatory diagram (background image) 1403 is displayed on 402, and the explanation corresponding to this explanatory diagram 1403 is pointed to the explanation place using the pointer 1401 in conjunction with the movement of the mouse 1405, and at the same time, the explanation voice Is recorded on the tape recorder 1407 as a voice recording device through the microphone 1406. At this time, the explanation voice (the above-mentioned text data),
A character string 1404 is displayed on the display 1402, and the corresponding portion of the contents to be explained is changed in sequence, for example, in accordance with the internal clock of the computer. The user keeps this intonation while maintaining the intonation of a normal conversation.
Read aloud 4. The data obtained in this way is used as a CD
-By burning onto a ROM, it is possible to create a CD-ROM in which moving images and sounds are recorded almost in synchronization.

However, in reality, the recorded voice is recorded on a CD.
-When recording in the ROM, the start time of the recorded voice is not always synchronized with the moving image due to the timing difference of pressing the start button of the tape recorder. Therefore, in the present invention, C created as described later
Corrected address information (see FIG. 7) corresponding to the address information (see FIG. 6) in the D-ROM is created to handle this. In other words, when perfect synchronization cannot be obtained with the address information (see FIG. 6) in the created CD-ROM, the timing for perfect synchronization is determined by trial and error, and this is used as the correction address information (see FIG. (See 7)
Store it on a hard disk in the multimedia processing system. And at the time of PLAY of CD-ROM,
PLAY is controlled based on the correction address information (see FIG. 7) held in the multimedia processing system described above. As a result, even if there is a slight synchronization shift in the recording in the CD-ROM, it is possible to remove this and obtain a completely synchronized output. The first embodiment of the present invention will be described below with reference to FIGS.

FIG. 1 is a hardware configuration diagram of a multimedia handling system showing an embodiment of the present invention. In this embodiment, the CD-ROM described above is used as the read-only storage device. In FIG. 1, reference numeral 1101 stores a multimedia handling method, a computer for controlling multimedia data, 1
An internal clock 102 informs the computer 1101 of the current time. Peripheral devices of the computer 1101 include a mouse 1108 as an input unit, a keyboard 1109, a hard disk 1103 that stores a program representing control data and control commands to be processed, a display 1104 for displaying multimedia data, It is composed of a CD-ROM drive 1105 for reproducing sound source data of a CD-ROM according to a reproduction command signal, an amplifier 1106 for outputting sound, and a speaker 1107.
Also, in order to synchronize the display of media such as text, figures, and still images with the output of sound, that is, in order to match the timing of displaying text, figures, still images, etc. with the timing of outputting sound, the text to be output, Figures, still images, etc.
Since the load on the computer differs depending on the number of bytes of data, a table (animation management table) 1110 for managing the delay time of issuing the sound output command signal according to the amount of load given to the computer such as text, figures, and still images is created. For example, it is provided in the storage device of the computer 1101.

In the present embodiment, when an output request for a certain medium is made, the output of sound is delayed according to the load amount by referring to the above table 1110, or
Achieve synchronization by speeding up. Hereinafter, as a specific example, an algorithm for synchronizing the display of a medium such as an animation and a moving image with the output of a sound, that is, an algorithm for matching the timing of displaying an animation and a moving image with the timing of outputting a sound will be described. In this case, it is possible to synchronize the two by adjusting their timings with the internal clock of the computer. As a basic concept, the output of animations and moving images is to display a plurality of pictures in sequence and to change their positions, and is the same processing from a time axis viewpoint. However, displaying a plurality of pictures one after another in sequence and changing the position of one picture are different from each other in processing amount of the computer, and thus need to be handled separately. In the following, an embodiment for setting the timing of synchronization in the case where the load on the CPU of the computer is small, such as a simple animation by changing the position of one picture, will be described below. In this case, wait for the display while watching the internal clock of the computer until the time to display, or for animation with fast movement, skip the data according to the internal clock of the computer to synchronize on the time axis. Try.

FIG. 2 shows a multimedia which has a description effect equivalent to that of a presentation using an overhead projector, by moving the pointer to the corresponding part of the description in correspondence with the explanation sound of the illustration displayed on the display. This is an algorithm for creating data for the presentation system. FIG. 3 is a diagram showing a display algorithm of the system shown in this embodiment. FIG. 4 is an image diagram when the pointer animation data and the voice data are input using the system according to the present embodiment. A method of synchronizing the explanation voice and the motion of the pointer animation will be described below with reference to these drawings. First, the process of creating data will be described with reference to FIG. In step 1201 of FIG. 2, text data describing the contents of the explanation voice is read, and in step 1202, a background 1403 which is an explanatory diagram is read.
To draw. Next, at step 1203, the voice reading interval Δt, mouse coordinates (x ₀ , y ₀ ), data counter i, j
Is initialized. In step 1204, input of a mouse button is awaited. At step 1203A, the initial time T ₀ is initialized. Note that time () is a function for reading the current time from the internal clock of the computer. Along with mouse button input, voice recording is started and the mouse coordinates are taken by moving the mouse in accordance with the explanation voice.

[0012] The data is kept waiting for input until the mouse button is pressed again (step 1205). Step
In 1206, the current time is read into t, and in step 1207, the elapsed time from the initial time is stored in flag. next,
In step 1208, the mouse coordinates (x, y) are fetched, and in step 1209, it is determined whether the fetched coordinates are different from the previous mouse coordinates (x ₀ , y ₀ ). If the mouse moves, in step 1210, the data counter i, mouse coordinates (x, y), and elapsed time flag are recorded in the animation management table, and in step 1211, the data counter i is incremented by 1 and the previous mouse is used. The current mouse coordinate (x, y) is substituted for the coordinate (x ₀ , y ₀ ). Note that the animation management table is configured as shown in FIG. 5, and the data number, mouse coordinates, and elapsed time
The flag can be recorded. In step 1212, the voice reading interval Δt is compared with the elapsed time flag,
If the voice reading interval Δt is smaller than the flag and the data counter j is smaller than the number of characters m, in step 1213, the j-th text is discolored to notify the user of the character to be read, and in step 1214, the elapsed time flag is changed. The voice reading interval Δt is subtracted, and in step 1215, the data counter j is incremented by one. Then, in step 1216, the magnitude relationship between the elapsed time flag and the voice reading interval Δt is compared again, and these processes are repeated until the elapsed time flag becomes smaller than the voice reading interval Δt. By this processing, the character data to be displayed at the current time can be displayed without omission.

With this series of processing, the characters to be read are displayed in different colors for the user, so while maintaining the normal intonation of the conversation, the data can be created by moving the mouse as explained to a human opponent. To be done. Next, when the audio data created by the above method, the coordinate data for animation, etc. are stored in a CD-ROM and a presentation is carried out using this, if an output command signal, that is, a PLAY command is executed, Sound is output from the CD-ROM drive. The algorithm for displaying data will be described below with reference to FIG. In step 1301, an explanatory diagram that is a background image is displayed, and in step 1302, the animation management table is read. The animation management table is created in the procedure already described with reference to FIG. 2 and has the format shown in FIG. Step 1303
Then, the PLAY command is issued to the CD-ROM drive, and the explanation voice is output from the CD-ROM drive.
In step 1304, the initial time T ₀ and the data counter i are initialized. As described above, time () is a function for reading the current time from the internal clock of the computer. While the explanation voice is playing, in step 1305, the following process is repeated until the pointer animation data counter i reaches the data number n-1. That is, in step 1306, the current time is read into t
In 1307, the elapsed time from the initial time is stored in flag.

In step 1311, the value of flag is t (i
If it is larger than +1), in step 1308, the data counter i is incremented by 1, and the skipped reading of data is performed without displaying the i-th data. In step 1309, the value of flag is the predetermined time t (i) and t (i + 1) described in the animation management table.
If it is confirmed that the pointer is between the coordinate data (x (i), y) in the table in step 1310.
Move to (i)) and display. By these processes, an animation that matches the progressing speed of the internal clock of the computer, that is, is synchronized with the absolute time is reproduced. On the other hand, CD-RO
As described above, the sound output from M can be synchronized with the start timing of the animation by adjusting the PLAY command issuing time. As long as the start timing is correct, the sound is output according to the absolute time, and thereafter, the sound can be perfectly synchronized with the animation.
The method of starting the timing after burning the CD-ROM will be described below.

The non-rewritable address information of the CD-ROM is described as shown in FIG. 6, while the rewritable sound information management table on the hard disk has the structure as shown in FIG. It describes how many minutes and seconds the sound of the song starts and the timing with another medium, that is, a still image, a moving image, or the like can be taken. After recording the sound as digital data, the sound data is filed by the computer. At this time, a silent state for a certain time at the beginning of the sound data is created. For example, the sound data is described in the directory information so that it starts from 1 minute 1 second 44 frames (1601 in FIG. 6), and the actual sound starts from 1 minute 2 seconds 20 frames. In this way, if PLAY is performed from 1 minute 1 second 44 frames, after a silent state continues for a while, sound starts to be output from 1 minute 2 seconds 20 frames. The timing for synchronizing with other media is determined by trial and error after burning the CD-ROM. For example, from 1 minute 2 seconds 8 frames to PLA
If Y indicates that the timing is correct, this address is described in the sound information management table of the application software (1701 in FIG. 7), and is provided outside the CD-ROM, that is, on the hard disk or the like where writing and updating are possible. ..

The sound information management table is a CD-ROM.
Application software that drives
When you install a program that calls and runs application software, you can write it to the hard disk together. The application software reads the address of the sound information on the CD-ROM from the sound information management table to know the timing for synchronization. According to the above embodiment, by providing the address information of the sound information outside the CD-ROM, the CD is read again in order to correct the timing deviation of the animation.
-No need to burn ROM. That is, the application software absorbs the discrepancy between the sound output of the created CD-ROM data and the display output of the image, and the multimedia processing system that can easily synchronize the media such as voice, animation, and moving image is configured. can do. Next, a second embodiment of the present invention for displaying a plurality of pictures continuously will be described below. It is assumed that the time for displaying one picture is ΔT _0, and as shown in the timing chart of synchronization, synchronization with the internal clock of the computer is attempted at time intervals of ΔT> ΔT ₀ . These data
It is described in the animation management table as shown in FIG. In this animation management table, a time counter t in steps of ΔT and a data number i of an image to be displayed at time t × ΔT are described.

When creating the above-mentioned animation management table, the content to be read out is converted into a character string and displayed on the computer display for the sound synchronized with the moving image or animation. Then, when moving images or animations are continuously displayed, the current time T is read from the internal clock of the computer, and by referring to the animation management table, one screen of the moving image or animation is displayed, and the color of the read characters is changed. .. The voice data input person can synchronize the voice with a moving image or animation by adjusting the reading speed while maintaining the intonation of a normal conversation in accordance with the displayed characters. The audio data created in this way can be completely synchronized if the start timing of the video is matched. The start timing can be adjusted by the method described in the first embodiment by processing the application software after burning the CD-ROM. Since these processes are almost the same as the contents described in the description of FIG. 12 in the first embodiment, the details are omitted, and the audio data, the coordinate data for animation, and the like created in this way are recorded on the CD-RO.
An algorithm for storing in M and giving a presentation using this will be described with reference to FIG.

At step 2201, an explanatory drawing which becomes a background image is displayed, and at step 2202, an animation management table is read. The table is created in the procedure already described with reference to FIG. 2 and has the format shown in FIG. In step 2203, the CD
-Issue the PLAY command to the ROM drive to output the explanation voice from the CD-ROM drive. Step 22
At 04, the initial time T ₀ , the data counter i, and the data register i_old are initialized. As described above, time () is a function for reading the current time from the internal clock of the computer. While the explanation voice is playing, step 2205
Then, the following processing is repeated until the pointer animation data counter i reaches the data number n-1.
That is, in step 2206, the current time is read into t, and in step 2207, the time elapsed from the initial time divided by ΔT is stored in flag.

In step 2211, the value of flag is t (i
If it is larger than +1), in step 2208, the data counter i is incremented by 1, and the skipped data reading is performed without displaying the i-th data. In step 2209, the value of flag is the predetermined time t (i) and t (i + 1) described in the animation management table.
, And if the values of i and i_old are different, that is, if there is a display request for data different from the image currently being displayed, in step 2210 the i-th image data is displayed, and in step 2212 At, i is substituted for i_old. By these processes, an animation that matches the progressing speed of the internal clock of the computer, that is, is synchronized with the absolute time is reproduced. On the other hand, the sound output from the CD-ROM can be synchronized with the start timing of the animation by adjusting the PLAY command issuing time as described in the first embodiment. As long as the start timing is correct, the sound will be output according to absolute time,
After that, it will be perfectly synchronized with the animation.

[0019]

As described above in detail, according to the present invention, the application software absorbs the discrepancy between the sound output of the created CD-ROM data and the display output timing of the image, and the sound, animation, and moving image are absorbed. It is possible to realize a multimedia processing system in which media such as the above can be easily synchronized. In addition, by using the present invention,
The explanation using voice for CAI system etc.
There is also an effect that it can be easily created without requiring a special device.

[0020]

[Brief description of drawings]

FIG. 1 is a hardware configuration diagram of a multimedia handling device showing an embodiment of the present invention.

FIG. 2 is a PAD diagram showing a method of creating an animation management table.

FIG. 3 is a PAD diagram showing a method for synchronizing and outputting voice and animation using an animation management table according to the embodiment.

FIG. 4 is an image diagram when inputting pointer animation data and voice data in the embodiment.

FIG. 5 is a diagram showing an example of a format of an animation management table.

FIG. 6 is a diagram showing a format example of a sound information management table in a CD-ROM.

FIG. 7 is a diagram showing a format example of a sound information management table of a CD-ROM on a hard disk.

FIG. 8 is a diagram showing another example of the format of the animation management table.

FIG. 9 is a PAD diagram showing a method of outputting audio and animation in synchronization with each other using an animation management table according to the embodiment.

[Explanation of symbols]

1101: Computer, 1102: Internal clock, 1103: Hard disk, 1104: Display, 1105: C
D-ROM drive, 1106: amplifier, 1107: speaker, 1108: mouse, 1109: keyboard, 1
110: Animation management table.

Claims (8)

[Claims] 1. A computer that controls multimedia data, an internal clock that is stored in the computer and that informs the computer of the current time, and storage means that stores control data to be processed and programs representing control instructions. A read-only storage device that stores sound source data (music, voice, etc.), multimedia data such as images, characters, and graphics that are output-controlled in accordance with a command from the computer, and an amplifier and a speaker for outputting sound In a multimedia processing system having a display for displaying multimedia data, any storage device is provided with table data in which an identification number of the display data and a display time history of the display data are associated with each other. A multimedia feature characterized by synchronizing the display time of the display data and the output time of the audio using Deer processing system. 2. When outputting sound while displaying data such as a moving image or animation that is heavily loaded on a computer, the internal clock of the computer is compared with the display time of the table data while referring to the table data. By doing so, the target display data and the sound output time are delayed,
2. The multimedia processing system according to claim 1, further comprising means for controlling the timing of the sound and the image by waiting for the skipped reading or output of the display data. 3. The sound data identification number and the address number of the sound data are stored on a rewritable storage means in the system, separately from the address information of the sound data recorded in the read-only storage device. 3. The multimedia processing system according to claim 1, further comprising a rewritable sound information management table that describes the address of sound data. 4. A soundless time region of a small time is recorded at the beginning of each sound data of the read-only storage device, and the output of the sound and image of the read-only storage device to the sound information management table is started. 4. The multimedia processing system according to claim 3, wherein the timing of the sound and the image are matched by changing the address of the sound data within the silent time region so that the timing of the sound timing is matched. 5. A drawing such as a pointer different from the display drawing on the display drawing in the display is moved in synchronization with the reproduction of the voice or song data.
4. The multimedia processing system according to any one of 4 above. 6. When inputting a description voice or music for the read-only storage device to a sound recording device, while moving a drawing such as a pointer on a display drawing in the display,
The multimedia according to any one of claims 1 to 5, wherein synchronization of movement of a drawing such as a pointer corresponding to the explanation voice or music is obtained by recording the explanation voice or music in a sound recording device. Processing system. 7. When inputting the explanation voice or music for the read-only storage device to a sound recording device, the storage device in the system is provided with character string data for the explanation voice, which is stored in the internal clock of the computer. The multimedia processing system according to claim 6, wherein the display mode is sequentially changed according to the display. 8. When inputting the explanation voice for the read-only storage device to a sound recording device, the character string data for the explanation voice is displayed by sequentially changing colors according to the internal clock of the computer, 7. The multimedia processing system according to claim 6, wherein the character string is read aloud and a drawing such as a pointer is simultaneously moved.