JPH09297749A - Multimedia document preparing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To synchronize a sound and an image or between sounds by displaying a sound waveform, so that the reproduction start time and reproduction end time of a prepared document reproduction passage time base can be coincident with the reproduction passage time of the time base. SOLUTION: A reproduction time instruction band 221 or an image reproduction time display band 222 and a sound reproduction time display band 223 are displayed in a reproduction time instruction/display area 220. In this case, the sound reproduction time display band 223 displays the reproduction time of a multimedia document composed only of sounds, that document is displayed in the form of a band over the time corresponding to the reproduction time instruction band 221 and inside that band 225, the output amplitude of that sound corresponding to the passage of time is displayed as a waveform 226. For example, the sound reproduction time display band 223 shows the sound having the waveform 226 is to be reproduced from the time shown by a position 229 of the reproduction time instruction band 221, corresponding to a left end 225s of the band 225 to the time shown by a position 230 of the reproduction time instruction band 221, corresponding to a right end 225e of the band 225.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a multimedia document creating apparatus for creating a multimedia document by appropriately combining ready-made still images, moving images, sounds and the like.

[0002]

2. Description of the Related Art When creating a multimedia document composed of a combination of still images, moving images, sounds, etc., an appropriate one is selected from ready-made still images, moving images, sounds etc. stored in a hard disk device, etc. When an image is selected, the display position on the playback screen and the playback start / end time in the created document, and when audio is selected, the device that creates the multimedia document by specifying the playback start / end time in the created document is already available. Being developed.

[0003]

By the way, during the creation work, it is often the case that the reproduction time is synchronized with the sound and the image. For example, when the sound is a performance sound of a certain song, the reproduction of a certain image is started from a characteristic portion of the song (for example, a climax portion where the volume is maximum).

In the conventional device, when the above-mentioned synchronization is taken, the time at which the characteristic part appears (hereinafter,
The "characteristic portion appearance time") is measured using a clock function or the like of the device, and the measured time is designated as the image reproduction start time. Therefore, according to the conventional device,
The above-mentioned work must be performed every time synchronization is taken, which is very troublesome.

In view of the above-mentioned problems, the present invention is a multimedia document creation apparatus capable of easily synchronizing voice and image or voice and voice without measuring characteristic feature appearance times one by one. The purpose is to provide.

[0006]

In order to achieve the above object, a multimedia document creation apparatus according to the present invention stores data such as still images, moving images, and audio stored in image / audio data storage means. A multimedia document creation apparatus for creating a multimedia document in combination, comprising a data selection acceptance means for accepting selection of data stored in an image / sound data storage means, and a reproduction start time of the selected data in the creation document. And a reproduction time reception means for receiving an instruction of a reproduction end time, and the audio data is selected by the data selection reception means, and when the reproduction start time and the reproduction end time are received by the reproduction time reception means, the selected audio data A voice waveform extracting means for extracting a voice waveform from the reproduction start time to the reproduction end time; Displays created document reproduction elapsed time axis representing the raw elapsed time, the speech waveform with respect to the time axis,
Display means for displaying the reproduction start time and the reproduction end time so as to match the reproduction elapsed time on the time axis.

Further, when the image data is selected by the data selection acceptance means and the reproduction start time and the reproduction end time are accepted by the reproduction time acceptance means, the reproduction start time of the image data on the created document reproduction elapsed time axis. Corresponding to the time length from the playback end time to the playback end time, the image playback time zone creating means for creating a strip-shaped image playback time display zone including the first image to be played back, the display means further comprising the image A reproduction time display band is provided in parallel with the document reproduction elapsed time axis, and an image reproduction time display band display section is provided for displaying the reproduction start time and the reproduction end time so that the reproduction elapsed time on the time axis matches. You can also do it.

Further, when the moving image data is selected by the data selection receiving means, the image reproduction time zone creating means, in addition to the first image to be reproduced, the reproduced images at predetermined time intervals from the first image. It is also possible to have a fixed-interval moving image reproduction time zone creation unit that creates the included image reproduction time display zone. Further, when the audio data is selected by the data selection acceptance means and the reproduction start time and the reproduction end time are accepted by the reproduction time acceptance means, the maximum point and the minimum point of the audio waveform extracted by the audio waveform extraction means appear. Waveform characteristic time extracting means for extracting a time as an audio waveform characteristic time; and the waveform characteristic time in the image reproduction time display band of the already selected moving image having a reproduction time overlapping the reproduction time of the selected audio data. Image reproduction time display band changing means for displaying the image at the voice waveform characteristic time extracted by the extracting means may be provided.

Further, when the moving image data is selected by the data selection receiving means and the reproduction start time and the reproduction end time are received by the reproduction time receiving means, from the reproduction start time to the reproduction end time of the selected moving image. Image change time extraction means for extracting a change time, which is the time after the image has changed significantly, the image reproduction time zone creating means includes the image change time extraction means in addition to the first image to be reproduced. It is also possible to have a changed moving image reproduction time zone creating section for creating an image reproduction time display zone including the reproduced image at the change time extracted by.

Further, when the moving image data is selected by the data selection receiving means and the reproduction start time and the reproduction end time are received by the reproduction time receiving means, from the reproduction start time to the reproduction end time of the selected moving image. Image selection screen display means for displaying an image selection screen composed of a list of all images to be reproduced during the period, and image selection acceptance means for accepting selection of a display band image to be displayed in the image reproduction time display band on the image selection screen. And the image reproduction time zone creating means may have a selected moving image reproduction time zone creating section for creating an image reproduction time display zone including the display zone image received by the image selection receiving means. .

The display band image accepted by the image selection accepting means is stored in the image / sound storage means in association with the moving image, and the moving image data is selected by the data selection accepting means. And referring to the stored contents of the image / sound storage means, it is determined whether or not a display band image is stored for the selected moving image, and if stored, the image selection screen display means and Control stop means for stopping the control of the image selection receiving means, the image reproduction time zone creating means, when the control of the image selection screen display means and the image selection receiving means is stopped, to the selected moving image. It is also possible to have an already-selected moving image reproduction time zone creation unit that creates an image reproduction time display zone including display zone images stored in association with each other.

[0012]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to FIGS. (Embodiment 1) FIG. 1 is a block diagram showing the configuration of Embodiment 1 of a multimedia document creation apparatus according to the present invention.

As shown in the figure, the multimedia document creating apparatus includes an image / sound data storage unit 101 and a display unit 1.
02, input unit 103, file name list extraction unit 104, image extraction unit 105, audio waveform extraction unit 106, time axis synchronization diagram creation unit 107, edit result storage unit 108, edit result storage processing unit 109, and edit / display control unit It is equipped with 110.

The image / sound data storage unit 101 is composed of, for example, a hard disk device, and stores multimedia document data including still images, moving images, sounds and the like, or a combination thereof. Image / sound data storage unit 101
Consists of an index information storage area and a data body storage area. In the index information storage area, a file name identifying each multimedia document data and data type information indicating whether the data is still image data, moving image data, or audio data. And address information indicating the storage location in the data body storage area, and the body of data corresponding to the file name is stored in the data body storage area at the position indicated by the address information. Note that still images are JPEG, TIFF, P
In the format such as ICT, the moving image is Motion J
Audio is compressed and stored in formats such as WAV and AIF in formats such as PEG, AVI, and MPEG.

The display unit 102 is composed of, for example, a CRT and a liquid crystal display, and is roughly divided into an image reproduction position instruction / display area 210 and a reproduction time instruction / display area 220, as shown in FIG. Image playback position indication / display area 2
A file selection screen including a file name list of multimedia documents stored in the image / sound data storage unit 101 is displayed on the screen 10, and a multimedia document corresponding to the file name selected from the file name list is displayed. A predetermined image may be displayed at the position designated by the input unit 103. The file selection screen also displays an edit work end button that is selected when the edit work is finished. The reproduction time indication / display area 220 includes a reproduction time indication band 221 for instructing the reproduction time of the selected multimedia document, an image reproduction time display band 222 indicating the reproduction time of the image data, and audio data. A voice reproduction time display band 223 indicating the reproduction time is displayed. The reproduction time indication band 221 is displayed to indicate the reproduction start time and reproduction end time of the selected multimedia document. The numbers (0, 3, 6) displayed above the playback time indicating band 221.
...) represents the elapsed time (seconds) from the start of reproduction of the created title. The image reproduction time display band 222 displays the reproduction time of the multimedia document including the image data, is displayed in a band shape over the time corresponding to the reproduction time indication band 221, and is displayed at the beginning (left end side) of the band 224. Is a still image, and if it is a moving image, the image of the first frame is displayed. For example, in the image reproduction time display band 222 shown in FIG.
The moving image 4a is the reproduction time indicating band 221 corresponding to the right end 224e of the band 224 from the time indicated by the position 227 of the reproduction time indicating band 221 corresponding to the left end 224s of the band 224.
It indicates that the reproduction is performed until the time indicated by the position 228. Note that the width W of the band 224 is determined by the reproduction time indication band 2
21 is equal to the length of 1 second, and the image of the first frame is
The band 224 is displayed in a square 234 which is separated from the left end 224s by a distance W. Voice playback time display band 223
Indicates a reproduction time of a multimedia document including only voice, and is displayed in a band shape over the corresponding time of the reproduction time indicating band 221. Inside the band 225, the output amplitude of the voice with respect to the elapsed time is displayed. Is waveform 226
Will be displayed as For example, in the voice reproduction time display band 223 shown in FIG. 2, the voice having the waveform 226 is located at the position 2 of the reproduction time indicating band 221 corresponding to the left end 225s of the band 225.
It indicates that the reproduction is performed from the time indicated by 29 to the time indicated by the position 230 of the reproduction time indicating zone 221 corresponding to the right end 225e of the zone 225.

The input unit 103 is, for example, a mouse,
The operator receives an instruction to display a file selection screen, an instruction to end a file name on the file selection screen and an editing operation, and an instruction such as a reproduction position, a reproduction start time, and a reproduction end time of the selected multimedia document. Double click the mouse button to display the file selection screen. The file name is selected by clicking the desired file name on the file selection screen. The edit work end instruction is given by clicking the edit work end button on the file selection screen. The playback position is designated by clicking two points in the image playback position designation / display area 210. For example, in FIG. 2, point 211
If the user clicks at two points, the point 212 and a point 212, a predetermined image of the selected multimedia document is displayed with a square 213 having a diagonal line connecting the two points as a display frame. The reproduction start time is instructed by placing the cursor on the position of the reproduction time instructing zone 221 at which the reproduction is to be started and clicking. Similarly, the reproduction end time is instructed by moving the cursor to the position of the reproduction time instructing band 221 where the reproduction is to be ended and clicking.

The file name list extraction unit 104 includes an input unit 1
The file name and its data type information stored in the image / sound data storage unit 101 are extracted according to the extraction instruction from the editing / display control unit 110 which receives the display instruction of the file selection screen from 03, and the list of file names is extracted. The file selection screen display data is created and output to the editing / display control unit 110.

When a still image is selected, the image extraction unit 105 selects the selected still image from the image / sound data storage unit 1.
01, decompress and bit map by a method according to each compression format, and output the bit map data to the editing / display control unit 110. Further, when a moving image is selected, the first frame of the selected moving image is read from the image / sound data storage unit 101, decompressed and bitmapped by a method according to each compression format,
The bitmap data is output to the edit / display control unit 110.

The voice waveform extraction unit 106 reads the selected voice data from the image / voice data storage unit 101,
Decompression is performed by a method according to each compression format, the audio data is converted into an audio waveform with time on the horizontal axis and output amplitude on the vertical axis, and bitmap data for displaying the audio waveform is created and created. The bitmap data is output to the edit / display control unit 110.

The time-axis synchronization diagram creation unit 107 is a bitmap data and input unit 10 created by the image extraction unit 105.
3, the display data of the image reproduction time display band is created based on the reproduction start time and the reproduction end time information, which are input from the bitmap data created by the audio waveform extraction unit 106 and the input unit 103. Based on the reproduction start time and reproduction end time information, the display data of the audio reproduction time display band is created, and the created display data is edited / display control unit 1
Output to 10.

The edit result holding unit 108 stores the instruction data, which is an instruction for editing from the input unit 103, that is, the display position of the image data, the reproduction start time, the reproduction end time, the reproduction start time of the audio data, and the reproduction end time. Image of each data
The display data is temporarily stored in association with the file name in the voice data storage unit, and the display data to be displayed on the display unit 102 is temporarily stored.

The edit result storage processing unit 109 includes an input unit 10
Editing / display control unit 11 that received an instruction to end editing work from 3
According to the instruction from 0, the instruction data among the data temporarily stored in the edited result holding unit 108 is stored in the image / sound data storage unit 101. The editing / display control unit 110
If the instruction accepted by the input unit 103 is a file selection screen display instruction, the file name list extraction unit 104 is instructed to create the file selection screen display data. The unit 109 is instructed to store the above-described predetermined data in the image / sound data storage unit 101. In addition, when a file is selected on the file selection screen, the type of the selected file is determined. In the case of sound, the reproduction start time and the reproduction end time are accepted from the input unit 103, and in the case of an image, the input unit 103
The image reproduction position, the reproduction start time, and the reproduction end time are received from. When the reception of the above time and position is completed, in the case of voice, the voice waveform extraction unit 106 is instructed to extract the voice waveform and create the bitmap data, and in the case of a still image, it is selected by the image extraction unit 105. An instruction to create bitmap data of an image is issued, and in the case of a moving image, an instruction to create bitmap data of the first frame of the selected moving image is issued to the image extraction unit 105. When the bitmap data is created, the time axis synchronization diagram creation unit 107 is instructed to create the display data of the image reproduction time display band or the audio reproduction time display band. When the display data is created, the display unit 102 is instructed to display the data, and the bitmap data created by the image extraction unit 105 is input to the input unit 10.
The display size is processed into the display size determined by the display frame designated by No. 3, and then the display unit 102 is instructed to display at the designated position.

The operation of the multimedia document creation apparatus having the above configuration will be described with reference to the flowchart shown in FIG. First, when the operator gives a file selection screen display instruction from the input unit 103 (step S3).
02), the image reproduction position instruction / display area 2 of the display unit 102
A file selection screen is displayed at 10 (step S30).
4).

When an editing work end instruction is given on the file selection screen (step S306), an editing result storage process is performed (step S308), and the editing process ends, while the file (name) is selected. When this is done (step S310), the processing from step S312 is performed. If the image file is selected, the process proceeds to step S314. If not, the process proceeds to step S334 (step S3).
12).

If it is determined in step S312 that an image file has been selected, the input of the reproduction position from the input unit 103 is awaited (step S314). When the reproduction position is input, the reproduction start time instruction is awaited ( (Step S316), when the reproduction start time is instructed,
When the reproduction end time is awaited (step S318) and the reproduction end time is instructed, the process proceeds to step S320.

In step S320, it is determined whether a still image file is selected or a moving image file is selected. If a still image file is selected, the image extraction unit 105 is selected. The bitmap data of the image is created (step S322), and the time axis synchronization diagram creation unit 107 creates the image reproduction time display band (step S324). On the other hand, when the moving image file is selected (step S320), the image extraction unit 105
Creates the bitmap data of the first frame of the selected moving image (step S326), and the time axis synchronization diagram creation unit 107 creates the image reproduction time display band (step S3).
28).

When the above processing is completed, step S33
0, the data created by the image extraction unit 105 is displayed in the designated display frame at the designated position in the image playback position designation / display area 210, and the playback time designation / display area 2 is displayed.
An image reproduction time display band is displayed on 20 (step S33).
2), the process returns to step S302. On the other hand, if it is determined in step S312 that the audio file is selected, the process waits for a reproduction start time instruction (step S334), and if the reproduction start time is instructed, a reproduction end time instruction waits (step S336). When the end time is designated, the process proceeds to step S338.

In step S338, the voice waveform extraction unit 1
06 creates bitmap data for displaying the audio waveform of the selected file, and then the time-axis synchronization diagram creation unit 107 creates an audio playback time display band (step S340). The audio reproduction time display band is displayed in the reproduction time instruction / display area 220 (step S34).
2), the process returns to step S302.

FIG. 2 shows a display screen when one moving image and one audio are selected. In the image reproduction position instruction / display area 210, the image 214 of the first frame of the selected moving image is displayed as the two points 211 indicated in step S314.
A rectangle 213 having a diagonal line connecting the line segment and the line 212 is displayed as a display frame, and the image reproduction time display band 222 is displayed in the reproduction time instruction / display area 220.
Further, in the playback time instruction / display area 220, the waveform 22
A voice reproduction time display band 223 of a voice having 6 is displayed.

As described above, according to the present embodiment, the waveform of the selected voice is displayed on the edit screen with the reproduction time indicating band as the elapsed time scale. Therefore, FIG.
When the operator wants to perform editing such as displaying a still image from the peak time of the voice while the voice reproduction time display band 223 shown in FIG.
The peak point 23 of the waveform 226 of the reproduction time indicating band 221
This can be realized by designating the position 231 corresponding to 2 as the reproduction start time of the still image. That is, it is possible to very easily instruct the reproduction start time, the reproduction end time, and the like of another still image, a moving image, or a sound based on the time of a characteristic portion of a certain sound. In addition to the peak point as described above, the characteristic point of the voice may be the lowest point of the sound pressure level, the time zone in which the amplitude fluctuation is severe, the time zone in which the voice with a low sound pressure level continues, and the like.

Further, according to the present embodiment, the reproduction time of the selected image is also displayed on the edit screen as the image reproduction time display band, so that the correlation with the reproduction time of other images and sounds can be grasped. It's easy to do. (Second Embodiment) In the second embodiment, when a moving image is selected, not only the image of the first frame, but also a plurality of images are displayed at equal time intervals over the time period to be reproduced. It is devised to be.

FIG. 4 is a block diagram of the multimedia document creation apparatus of this embodiment. This multimedia document creation apparatus has an editing / display control section 410 in place of the editing / display control section 110 and an image extraction section 40 in place of the image extraction section 105 of the multimedia document creation apparatus according to the first embodiment.
Five. Since other configurations are basically the same as those of the first embodiment, the same reference numerals as those in the first embodiment are attached,
The explanation is omitted.

When a moving image is selected, the editing / display control unit 410 instructs the image extracting unit 405 to add the bitmap data of the first frame of the selected moving image to the bitmap of the frame at a predetermined time. The point of issuing a data creation instruction is different from the editing / display control unit 110 of the first embodiment. Others are the same as those of the editing / display control unit 110 of the first embodiment, and thus the description thereof is omitted. In the present embodiment, the predetermined time is every 1 second from the start of reproduction.

When a moving image is selected, the image extracting unit 405 creates bit map data of the first frame of the selected moving image and the frame at the predetermined time according to the instruction of the editing / display control unit 410. And outputs it to the editing / display control unit 410. Regarding the operation of the multimedia document creation apparatus having the above configuration, in step S326 of the flowchart of the first embodiment shown in FIG. 3, the image extraction unit is instructed not only by the first frame but also by the editing / display control unit. The same applies to a frame of a predetermined time, except that the screen image is generated, and therefore the description thereof will be omitted.

FIG. 5 shows a display screen when one moving image and one audio are selected. The image reproduction time display band 522 shown in this figure is when the reproduction start time of the selected moving image is 1 second later and the reproduction end time is 5.5 seconds later. The image 501 displayed in the image reproduction time display band 522 is the first frame of the selected moving image, the image 502 is the frame one second after the start of reproduction of the selected moving image, and the image 503 is An image 504 of a frame after 2 seconds shows an image of a frame after 3 seconds.

As described above, according to the present embodiment, not only the image of the first frame but also the images of the frames at equal time intervals are displayed in the image reproduction time display band 522 of the moving image over the time period to be reproduced. Is displayed, it is easy to confirm the selected moving image. (Embodiment 3) In the present embodiment, when the moving image is selected and the sound is selected with the time zone overlapping all or a part of the reproducing time zone of the moving image as the instruction reproduction time. In addition, it is devised to display an image of a frame corresponding to the characteristic point (time) of the voice waveform, instead of the image displayed in the image reproduction time display band of the moving image until then.

FIG. 6 is a block diagram of the multimedia document creation apparatus of this embodiment. In addition to the configuration of the multimedia document creation apparatus of the first embodiment, this multimedia document creation apparatus further includes a waveform feature point extraction section 611, and instead of the editing / display control section 110, an editing / display control section 610. Have Other configurations are the same as those of the first embodiment.
Since it is basically the same as that of the first embodiment, the same reference numeral as that of the first embodiment is attached, and the description thereof is omitted.

The waveform feature point extraction unit 611 extracts a feature point of the voice waveform based on the voice waveform extracted by the voice waveform extraction unit 106, and edits the time in the reproduction time indicating zone corresponding to the feature point. It is output to the display control unit 610. Here, the characteristic points basically mean the maximum point and the minimum point of the voice waveform. For example, in the waveform 701 shown in FIG. 7, the maximum point is
In 702, 704, 706 and 708, the minimum point is 70
3, 705 and 707. Of these, the feature points are the maximum or minimum points picked up in the order of elapsed time, the first maximum or minimum point is selected as the feature point, and the maximum or minimum points picked up after that are selected last time. The feature point is selected only when the difference between the feature point and a predetermined value is larger. For example, in the waveform 701 shown in FIG. 7, first, the maximum point 702, which is the first maximum or minimum point, is selected as the feature point, and the minimum point 703 that appears next is selected as the feature point. The next maximum point 704 appears, but the maximum point 704 is not selected as a feature point because the difference D from the previously selected feature point 703 is small. Since the next minimum point 705 also has a small difference from the previously selected feature point 703, it is not selected as the feature point, and the next maximum value 706 is the feature point. The feature points are extracted in this way, and in the waveform 701, the maximum points 702 and 706 and the minimum points 703 and 707 become the feature points. The waveform feature point extraction unit 611 then determines the time (2) of the reproduction time indication band 221 corresponding to the extracted feature point.
72, 273, 276, 277) editing / display control unit 6
Output to 10

The editing / display control unit 610 is the same as in the first embodiment.
In addition to the function of the editing / display control unit 110, if a voice is selected, whether or not a moving image having a time zone overlapping all or a part of the reproduction time zone instructed by the voice is already selected. If it is selected, the waveform feature point extraction unit 611 is instructed to extract the feature points of the selected voice, and the waveform feature point extraction unit 611 outputs the image feature unit 405. It has a function of issuing an instruction to create frame bitmap data at the time of the reproduction time indicating band 221 corresponding to the extracted feature point.

Regarding the operation of the multimedia document creation apparatus having the above-mentioned configuration, as shown in the flowchart of FIG. 8, in the flowchart of the first embodiment, the processing of steps S802 to 810 is inserted after step S342. Other than that, since it is the same as the first embodiment, the same step numbers as those in the flowchart of the first embodiment are attached,
The description is omitted.

The operation (steps S802 to 810) peculiar to this embodiment will be mainly described below with reference to FIG. When the selected audio reproduction time zone is displayed in step S342, the editing / display control unit 610 has a time zone that overlaps all or a part of the instructed reproduction time zone of the selected audio. It is determined whether or not a moving image has already been selected (step S802), and if it has not been selected, the process returns to step S302.

On the other hand, when selected, the waveform feature point extraction unit 611 is instructed to extract the feature points of the selected voice, and the instructed waveform feature point extraction unit 611
The feature point is extracted, and the time in the reproduction time indicating band corresponding to the feature point is output to the editing / display control unit 610 (step S804). Subsequently, the image extraction unit 105 extracts the waveform feature point extraction unit 6 of the moving image determined in step S802.
The bitmap data of the frame at the time output from 11 is created (step S806), the time axis synchronization diagram creation unit 107 creates the image playback time display band (step S808), and the playback time instruction / display area 220 is displayed.
The image reproduction time display band is displayed on the screen (step S33).
2), the process returns to step S302.

In the edit screen shown in FIG. 2, when the moving image is selected first and the image reproduction time display band 222 is displayed, the edit screen when the voice is selected later is shown in FIG.
Shown in When the moving image is selected first, only the image of the first frame is displayed.
Instead of 2, the image reproduction time display band 722 is displayed because the sound is selected in the time band overlapping with the reproduction time band. Image 71 in image reproduction time display band 722
1, 712, 713 and 714 are times 272 and 2 respectively.
73, 276, 277.

As described above, according to the present embodiment, when the audio data is selected with the reproduction time zone overlapping the reproduction time zone of the already selected moving picture as the reproduction time zone, The image reproduction time display band containing the image at the audio waveform characteristic point (time) of the selected audio data is displayed instead of the image reproduction time display band that was displayed until then. The image displayed at the point (time) can be easily confirmed. (Fourth Embodiment) In the fourth embodiment, when a moving image is selected, not only the image of the first frame but also a changed image in a case where the image changes drastically is displayed. There is.

FIG. 9 is a block diagram of the multimedia document creation apparatus of this embodiment. This multimedia document creation apparatus further includes an image change extraction unit 901 in addition to the configuration of the multimedia document creation apparatus according to the first embodiment, and has an edit / display control unit 910 instead of the edit / display control unit 110. ing. The other configurations are basically the same as those in the first embodiment, and therefore, the same reference numerals as those in the first embodiment are attached and the description thereof is omitted.

When a moving image is selected, the image change extraction unit 901 obtains the degree of change between adjacent frames in reproduction time according to an instruction from the editing / display control unit 910, and the degree of change is predetermined. A value larger than the predetermined value set is extracted, and the time after the change at the extracted change (hereinafter referred to as “change time”) is edited / display control unit 910.
Output to As for the degree of change, if the moving image is compressed and stored by MPEG, the motion compensation vector that is a part of the MPEG data is
Or, for those compressed and stored by AVI,
After decompressing the data, the difference between the frames is calculated, and the calculated difference value is used.

When a moving image is selected, the editing / display control unit 910 gives an instruction to the image change extracting unit 901 to extract the change time of the selected moving image, and the image extracting unit 10
5 is different from the editing / display control unit 110 of the first embodiment in that the instruction to create the bitmap data of the frame at the extracted change time is issued in addition to the bitmap data of the first frame. Others are the same as those of the editing / display control unit 110 of the first embodiment, and thus the description thereof is omitted.

With respect to the operation of the multimedia document creating apparatus having the above configuration, except that steps S1002 to 1006 shown in FIG. 10 are replaced with steps S326 and S328 in the flowchart of the first embodiment shown in FIG. Since they are the same as those in the first embodiment, their description will be omitted. Hereinafter, the operation peculiar to the present embodiment will be mainly described with reference to FIG.

In the flowchart shown in FIG. 3, when it is judged that an image has been selected (step S312) and the reception of the reproduction position, the reproduction start time and the reproduction end time from the input unit 103 (steps S314 to 318) is completed, the selection is made. It is determined whether or not the obtained image is a still image or a moving image (step S320), and if it is a moving image, the process proceeds to step S1002 shown in FIG. In step S1002, the image change extraction unit extracts the change time of the selected moving image, and then the image extraction unit creates bitmap data of the first frame and the frame at the extracted change time (step S1002). (S1004), the time axis synchronization diagram creation unit 107 creates an image reproduction time display band (step S1006). When the above process is completed, the process is performed as shown in FIG.
Go to step S330 shown in.

FIG. 11 shows a display screen when one moving image and one audio are selected. The image reproduction time display band 1122 shown in the figure is when the reproduction start time of the selected moving image is 1 second later and the reproduction end time is 5.5 seconds later. Image playback time display band 1122
An image 1101 displayed with is the image of the first frame of the selected moving image. The image 1102 shows an image after the change, although the degree of change between adjacent frames exceeds a predetermined value during 4.5 seconds from the start of reproduction of the moving image to the end thereof. In this example,
Since the degree of change exceeded the predetermined value only once in 4.5 seconds, only one image is displayed in addition to the image of the first frame.

As described above, according to the present embodiment, when a moving image is selected, the image at the time after a large change of the image is displayed in the image reproduction time display band in addition to the image of the first frame. Therefore, it is easy to confirm which moving image is selected. (Fifth Embodiment) In the fifth embodiment, an operator can select an image to be displayed in the image reproduction time display band when a moving image is selected.

FIG. 12 is a block diagram of the multimedia document creation apparatus of this embodiment. This multimedia document creation apparatus has an editing / display control unit 1201 instead of the editing / display control unit 110 of the multimedia document creation apparatus of the first embodiment. The other configurations are basically the same as those in the first embodiment, and therefore, the same reference numerals as those in the first embodiment are attached and the description thereof is omitted. In the present embodiment, in addition to the display content of the first embodiment, the image displayed by the operator in the image reproduction time display band is displayed in the image reproduction position instruction / display area of the display unit 102 as shown in FIG. A list of images of each frame at the designated reproduction time of the selected moving image (hereinafter, referred to as “display frame selection screen”) for selecting is displayed.

When the moving image is selected, the editing / display control unit 1201 issues an instruction to the image extracting unit 105 to create bitmap data of all frames within the instructed reproduction time, and causes the display unit 102 to display. On the other hand, the display instruction of the display frame selection screen is issued, the selection of the display frame input from the input unit 103 is accepted on the display frame selection screen, and the time axis synchronization diagram creation unit 107 is selected from the image of the selected frame. The playback time display band is instructed to be created. The other functions are similar to those of the editing / display control unit 107 of the first embodiment, and thus the description thereof will be omitted.

The image selection instruction operation on the display frame selection screen shown in FIG. 13 is performed by clicking while the cursor is within the display frame of the image to be selected, and the selection end instruction operation is image reproduction. This is done by clicking the selection end button displayed at the lower right of the position pointing / display area 210. Regarding the operation of the multimedia document creation apparatus having the above configuration, steps S326 and S32 in the flowchart of the first embodiment shown in FIG. 3 are performed.
8 instead of step S1402 to 141 shown in FIG.
Since the second embodiment is the same as the first embodiment except that it has 0, the description thereof will be omitted.

Hereinafter, the operation peculiar to the present embodiment will be mainly described with reference to FIG. In the flowchart shown in FIG. 3, it is determined that the image is selected (step S312), and the reproduction position, the reproduction start time, and the reproduction end time are received from the input unit 103 (steps S314-).
When step 318) ends, it is determined whether the selected image is a still image or a moving image (step S320), and if it is a moving image, the process proceeds to step S1402 shown in FIG. In step S1402, the image extraction unit 105 creates bitmap data of all the frames within the instructed reproduction time of the selected moving image according to the instruction of the editing / display control unit 1201, and the created bitmap data A display frame selection screen is displayed based on (step S14)
04). Here, when the operator looks at the display frame selection screen and gives an instruction to select an image to be displayed in the image reproduction time display band (step S1408), the instructed image is selected as a display target of the image reproduction time display band. (Step S1408). The image selection instruction is possible until the selection end instruction operation is performed, and when the selection end instruction operation is performed (step S1410), the process proceeds to step S330 in FIG.

On the display frame selection screen as shown in FIG. 13, images 1302, 1303, 1304, 130 are displayed.
FIG. 15 shows an image reproduction time display band displayed when four of 5 are selected. As described above, according to the present embodiment, since the image selected by the operator is displayed in the image reproduction time display band, it is easy to confirm which moving image is selected. (Embodiment 6) In the present embodiment, with respect to the fifth embodiment, a selected moving image which is once selected in the image reproduction time display band is selected again. Even if it does not (without passing through the display frame selection screen), it is devised so that the image reproduction time display band is displayed.

FIG. 16 is a block diagram of the multimedia document creation apparatus of this embodiment. This multimedia document creating apparatus is an image / audio data storage unit 101 in place of the image / audio data storing section 101 of the multimedia document creating apparatus of the fifth embodiment.
The audio data storage unit 1603 is replaced by the editing result storage processing unit 10.
9, an edit result storage processing unit 1602 is provided, and an edit / display control unit 1601 is provided instead of the edit / display control unit 1201. The other configurations are basically the same as those in the first embodiment, and therefore, the same reference numerals as those in the first embodiment are attached and the description thereof is omitted.

The image / sound data storage unit 1603 stores address information indicating the storage location of the frame selected on the display frame selection screen in the storage contents of the index information storage area of the image / sound data storage unit 101 of the fifth embodiment. The same as the image / sound data storage unit 101 of the fifth embodiment except that is added. That is, the address information (hereinafter referred to as “selected frame address information”) indicating the storage location of the frame selected on the display frame selection screen is stored in association with the file name of the selected moving image.

In addition to the function of the edit result storage processing unit 109 of the first embodiment, the edit result storage processing unit 1602 stores the selected frame address information in the image / audio data storage unit 1603 according to the instruction of the edit / display control unit 1601. It has a function of storing in association with the corresponding file name of. In addition to the functions of the edit / display control unit 1201 of the first embodiment, the edit / display control unit 1601 notifies the edit result storage processing unit 1602 of the selected frame address information when the selection work on the display frame selection screen is completed. When a moving image is selected and the moving image is selected, it is determined whether the selected moving image has the selected frame address information by referring to the image / sound data storage unit 1603.
The image extracting unit 105 has a function of issuing an instruction to create bitmap data of a frame at a storage location corresponding to the selected frame address information.

Regarding the operation of the present multimedia document creating apparatus having the above configuration, in the operation of the fifth embodiment, step S1402 of the flowchart shown in FIG.
Before the processing of step S1, it is determined whether or not the selected moving image has the selected frame address information. If not, the processing of step S1402 and subsequent steps is performed as in the case of the fifth embodiment. Is steps S1402 to S14
Instead of the process of 10, the editing / display control unit 1601 instructs the image extraction unit 105 to create bitmap data of the image stored in the position corresponding to the selected frame address information, and the instructed image is displayed. The process is the same as that of the fifth embodiment except that the extraction unit 105 performs the process of creating the bitmap data and then proceeds to step S1412, and therefore the description thereof will be omitted.

As described above, according to the present embodiment, when a moving image in which an image to be displayed has already been stored in the image reproduction time display band is selected, the image reproduction time display band including the stored image is displayed. Since it is displayed, the operator can save the trouble of selecting the image to be displayed again.

[0062]

In order to achieve the above-mentioned object, the multimedia document creating apparatus according to the present invention is a multimedia document combining apparatus for combining still image data, moving image data and audio data stored in the image / audio data storage means. A multimedia document creation apparatus for creating a document, comprising a data selection acceptance means for accepting selection of data stored in an image / sound data storage means, a reproduction start time and a reproduction end time in a creation document of the selected data. When the audio data is selected by the reproduction time accepting means that accepts the instruction and the reproduction start time and the reproduction end time are accepted by the reproduction time accepting means, from the reproduction start time of the selected audio data, The voice waveform extraction means for extracting the voice waveform between the playback end time and the elapsed playback time of the created document are displayed. A display means for displaying the elapsed time axis of reproduction of the created document, and displaying the audio waveform so that the reproduction start time and the reproduction end time of the audio waveform coincide with the elapsed reproduction time of the time axis. , By this, when the audio data is selected, the audio waveform between the reproduction start and the reproduction end of the selected audio data is displayed at the reproduction start time and reproduction end time and time with respect to the created document reproduction elapsed time axis. It is displayed so that the playback elapsed time on the axis matches. Therefore, the time, etc. at which a characteristic part of the audio waveform appears can be easily known by looking at the corresponding elapsed time on the time axis, without having to measure the time one by one. Can be easily synchronized.

When the image data is selected by the data selection acceptance means and the reproduction start time and the reproduction end time are accepted by the reproduction time acceptance means, the reproduction start time of the image data on the created document reproduction elapsed time axis. Corresponding to the time length from the playback end time to the playback end time, the image playback time zone creating means for creating a strip-shaped image playback time display zone including the first image to be played back, the display means further comprising the image A reproduction time display band is provided in parallel with the document reproduction elapsed time axis, and an image reproduction time display band display section is provided for displaying the reproduction start time and the reproduction end time so that the reproduction elapsed time on the time axis matches. By doing so, the time from the start of reproduction of the selected image to the end of reproduction is displayed as the image reproduction time display band in parallel with the created document reproduction elapsed time axis. Since the elapsed playback time at the reproduction start time and reproduction end time and the time axis are displayed to coincide, the correlation is likely to grasp about reproduction time of audio or other images.

Further, when the moving image data is selected by the data selection accepting means, the image reproduction time zone creating means, in addition to the first image to be reproduced, the reproduced images at predetermined time intervals from the first image. By including the constant-interval moving image reproduction time zone creation unit that creates the included image reproduction time display zone, the image reproduction time display zone includes the first image to be reproduced and a predetermined time from the first image. Since images including reproduced images at intervals are displayed, it is easy to confirm which image is selected.

When audio data is selected by the data selection reception means and the reproduction start time and reproduction end time are received by the reproduction time reception means, the maximum and minimum points of the audio waveform extracted by the audio waveform extraction means are received. Waveform characteristic time extraction means for extracting a time at which a point appears as an audio waveform characteristic time; and the image reproduction time display band of the already selected moving image having a reproduction time overlapping the reproduction time of the selected audio data, By providing the image reproduction time display band changing unit for displaying the image at the voice waveform characteristic time extracted by the waveform characteristic time extracting unit, the time period overlapping with the reproduction time of the already selected moving image is reproduced. When audio data is selected as the time zone, the audio of the selected audio data is displayed in the image playback time display zone of the already selected moving image. Since the image in the form wherein the time is displayed, confirmation of the image can be easily displayed in the speech waveform characteristic time of the moving picture.

When the moving image data is selected by the data selection receiving means and the reproduction start time and the reproduction end time are received by the reproduction time receiving means, from the reproduction start time to the reproduction end time of the selected moving image. Image change time extraction means for extracting a change time, which is the time after the image has changed significantly, the image reproduction time zone creating means includes the image change time extraction means in addition to the first image to be reproduced. By including a change moving image reproduction time zone creation unit that creates an image reproduction time display zone including a reproduction image extracted at the change time, the image reproduction time display zone includes the first image to be reproduced. In addition, since the reproduced image at the time after the image changes greatly is displayed, it is easy to confirm which moving image is selected.

When the moving image data is selected by the data selection receiving means and the reproduction start time and the reproduction end time are received by the reproduction time receiving means, from the reproduction start time to the reproduction end time of the selected moving image. Image selection screen display means for displaying an image selection screen composed of a list of all images to be reproduced during the period, and image selection acceptance means for accepting selection of a display band image to be displayed in the image reproduction time display band on the image selection screen. And the image reproduction time zone creating means has a selected moving image reproduction time zone creating section for creating an image reproduction time display zone including the display zone image received by the image selection receiving means, Since the display band image selected by the operator is displayed in the image reproduction time display band, it is easy to confirm which moving image is selected.

Further, the display band image received by the image selection receiving means is stored in the image / sound storage means in association with the moving image, and the moving image data is selected by the data selection receiving means. And referring to the stored contents of the image / sound storage means, it is determined whether or not a display band image is stored for the selected moving image, and if stored, the image selection screen display means and Control stop means for stopping the control of the image selection receiving means, the image reproduction time zone creating means, when the control of the image selection screen display means and the image selection receiving means is stopped, to the selected moving image. A display band to be displayed in the image reproduction time display band by having an already selected moving image reproduction time band creation unit that creates an image reproduction time display band including display band images stored in association with each other. When a moving image is selected the image is stored, the image reproduction time display period including the display range image stored is displayed, the operator, again, Habukeru the trouble of selecting the displayed band image.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a multimedia document creation device according to a first embodiment.

FIG. 2 is a diagram showing an example of display contents of a display unit according to the first embodiment.

FIG. 3 is a flowchart showing a processing procedure of the first embodiment.

FIG. 4 is a block diagram showing a configuration of a multimedia document creation device according to a second embodiment.

FIG. 5 is a diagram showing an example of display contents of a display unit according to the second embodiment.

FIG. 6 is a block diagram showing a configuration of a multimedia document creation device according to a third embodiment.

FIG. 7 is a diagram showing an example of display contents of a display unit according to the third embodiment.

FIG. 8 is a flowchart showing a processing procedure according to the third embodiment.

FIG. 9 is a block diagram showing a configuration of a multimedia document creation device according to a fourth embodiment.

FIG. 10 is a part of a flowchart showing a processing procedure according to the fourth embodiment.

FIG. 11 is a diagram showing an example of display contents of a display unit according to the fourth embodiment.

FIG. 12 is a block diagram showing the configuration of a multimedia document creation device according to a fifth embodiment.

FIG. 13 is a diagram showing an example of display contents of a display unit according to the fifth embodiment.

FIG. 14 is a part of a flowchart showing a processing procedure according to the fifth embodiment.

FIG. 15 is a diagram showing an example of display contents of a display unit according to the fifth embodiment.

FIG. 16 is a block diagram showing a configuration of a multimedia document creation device according to a fifth embodiment.

[Explanation of symbols]

101, 1603 image / audio data storage unit 102 display unit 103 input unit 104 file name list extraction unit 105, 405 image extraction unit 106 audio waveform extraction unit 107 time axis synchronization diagram creation unit 108 edit result storage unit 109, 1602 edit result storage Processing unit 110, 410, 610, 910, 1201, 1601
Editing / display control unit 611 Waveform feature point extraction unit 901 Image change extraction unit

Claims (7)

[Claims] 1. A multimedia document creation apparatus for creating a multimedia document by combining data such as still images, moving images, and audio stored in the image / audio data storage means, the image / audio data storage means Data selection receiving means for receiving the selection of the data stored in, the reproduction time receiving means for receiving the instruction of the reproduction start time and the reproduction end time in the created document of the selected data, and the audio data is transmitted by the data selection reception means. Selected,
When the reproduction start time and the reproduction end time are received by the reproduction time reception means, the audio waveform extraction means for extracting the audio waveform between the reproduction start time and the reproduction end time of the selected audio data, and the reproduction progress of the created document Display means for displaying a reproduction document elapsed time axis representing time, and displaying the audio waveform with respect to the time axis such that the reproduction start time and the reproduction end time coincide with the reproduction elapsed time on the time axis. A multimedia document creation apparatus comprising: 2. When the image data is selected by the data selection reception means and the reproduction start time and the reproduction end time are received by the reproduction time reception means, the reproduction start time of the image data on the created document reproduction elapsed time axis Corresponds to the length of time from to the playback end time,
An image reproduction time zone creating means for creating a strip-shaped image reproduction time display zone including the first image to be reproduced is provided, and the display means further creates the image reproduction time zone with respect to a document reproduction elapsed time axis. The multimedia document creation apparatus according to claim 1, further comprising an image reproduction time display band display unit that displays the reproduction start time and the reproduction end time in parallel with the reproduction elapsed time on the time axis. . 3. The image reproduction time zone creating means, when the moving image data is selected by the data selection receiving means,
3. A constant-interval moving image reproduction time zone creating unit for creating an image reproduction time display zone including reproduced images at predetermined time intervals from the first image in addition to the first image to be reproduced. The described multimedia document creation device. 4. When the audio data is selected by the data selection receiving means and the reproduction start time and the reproduction end time are received by the reproduction time receiving means, the maximum point and the minimum point of the audio waveform extracted by the audio waveform extracting means. Waveform characteristic time extraction means for extracting a time at which a point appears as an audio waveform characteristic time; and the image reproduction time display band of the already selected moving image having a reproduction time overlapping with the reproduction time of the selected audio data, 3. The multimedia document creation apparatus according to claim 2, further comprising image reproduction time display band changing means for displaying an image at the voice waveform characteristic time extracted by the waveform characteristic time extracting means. 5. When the moving image data is selected by the data selection receiving means and the reproduction start time and the reproduction end time are received by the reproduction time receiving means, from the reproduction start time to the reproduction end time of the selected moving image. Image change time extraction means for extracting a change time, which is the time after the image has changed significantly, between the image reproduction time zone creation means, in addition to the first image to be reproduced, the image change time extraction means 3. The multimedia document creation apparatus according to claim 2, further comprising: a changed moving image reproduction time zone creation unit that creates an image reproduction time display zone including a reproduced image at the change time extracted by. 6. When the moving image data is selected by the data selection receiving means and the reproduction start time and the reproduction end time are received by the reproduction time receiving means, from the reproduction start time to the reproduction end time of the selected moving image. Image selection screen display means for displaying an image selection screen consisting of a list of all the images to be reproduced during the period, and image selection acceptance means for accepting selection of a display band image to be displayed in the image reproduction time display band on the image selection screen. Wherein the image reproduction time zone creating means has a selected moving image reproduction time zone creating section for creating an image reproduction time display zone including the display zone image received by the image selection receiving means. Item 2. The multimedia document creation device according to item 2. 7. A storage unit for storing the display band image received by the image selection receiving unit in an image / sound storage unit in association with the moving image, and moving image data is selected by the data selection receiving unit. And whether or not a display band image is stored for the selected moving image by referring to the stored contents of the image / sound storage unit, and if stored, the image selection screen display unit and Control stop means for stopping the control of the image selection receiving means, the image reproduction time zone creating means, when the control of the image selection screen display means and the image selection receiving means is stopped, to the selected moving image. 7. The multimedia document creation apparatus according to claim 6, further comprising a selected moving image playback time zone creation unit that creates an image playback time display zone including display zone images stored in association with each other.