JP2005318210A - Reproducing device for picture with sound - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a reproducer for reproducing pictures having sounds synchronously at accurate timings, even when a plurality of sound-including picture files in the form of sound data blocks are made to be produced, synchronously attached to pictures frames. <P>SOLUTION: The reproducer reads a plurality of sound-including picture files in the form of sound data blocks synchronously attached to pictures frames, takes the picture frames out of each file and combines them to obtain a composite picture frame, while it combines sound data blocks corresponding to each picture frame to form a composite sound data block. Then it shows the composite picture frames thus obtained on the output with the composite sound data blocks outputted as sounds. Thus, they are outputted, without shifting the timing mutually between the pictures and between the sound data contained in the original file. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  INDUSTRIAL APPLICABILITY The present invention is suitable in the field of packaged video / music playback for viewing in consumer / business applications using CDs / DVDs, and in the field of environmental video / environmental music distributed for commercial purposes by broadcasters / businesses in public facilities. TECHNICAL FIELD OF THE INVENTION

Conventionally, a technique for displaying a plurality of videos on the same screen has been used. Such a technique is often used particularly for surveillance cameras, multi-angle video, multiplexed broadcasting of television programs, and the like (see, for example, Patent Documents 1 to 3). However, these conventional technologies save space for installing a plurality of monitors and allow a single monitor to view a plurality of images at the same time. The audio signal is not played back, or one of the channels is played back when played back.
JP-A-6-6804 Japanese Patent Laid-Open No. 8-125973 JP-A-11-341443

  On the other hand, with the development of computers, video signals can be handled by general-purpose personal computers. In recent years, a file format in which video data and audio data are integrated has been developed, and video and audio can be reproduced by reproducing only one file.

  However, when a video signal in such a file format is synthesized and reproduced, a correct sound cannot be reproduced unless the reproduction timing of the acoustic signal is accurately matched. In the general-purpose video file format, each video frame of the video data and the audio data block are recorded in association with each other. Therefore, in order to match the playback timing of the audio data, the playback timing of the video data must also be accurately matched. . However, since video data has a large amount of data, there is a problem that if the processing CPU is low in capacity, the processing cannot be completed and the reproduction timing is shifted, and the reproduction timing of the audio data is also shifted accordingly.

  Therefore, the present invention reproduces sound-added video that can be played back synchronously at accurate timing even when a plurality of sound-added video files in a format in which audio data blocks are attached to a video frame synchronously. It is an object to provide an apparatus.

  In order to solve the above-mentioned problems, in the present invention, a sound video playback apparatus has a plurality of video files with a plurality of video frames each having a plurality of video frames and an audio data block associated with each video frame. Video file storage means, file selection means for selecting a plurality of sound-added video files to be synthesized and reproduced from the sound-added video file storage means, and video corresponding to each of the selected plurality of sound-added video files Video frame synthesizing means for extracting a frame and synthesizing it into one synthesized video frame, and extracting an audio data block attached to each corresponding video frame from the selected plurality of video files with sound, and producing one synthesized audio A sound data block synthesizing unit that synthesizes the data block; Video frame display means, sound output means for writing the synthesized audio data block to a sound device and reproducing the sound, each of the video frame synthesizing means, the audio data block synthesizing means, the video frame display means, and the sound output means. It is characterized by having a reproduction timing control means for performing control so that the processing is sequentially executed at predetermined time intervals for all the video frames stored in the selected video file with sound.

  According to the present invention, when a plurality of sound-added video files are played back simultaneously, a video frame is extracted from the sound-added video file to generate a composite video frame, and a sound data block corresponding to each video frame is combined to generate a composite audio frame. Since the data block is generated, the synthesized video frame is displayed, and the synthesized audio data block is played back, even if the processing capacity of the CPU of the playback device is low, it is played back synchronously with accurate timing. It becomes possible to do.

(1. Structure of video file with sound)
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. First, the structure of a moving image file with sound to be played back by the playback device for video with sound according to the present invention will be described. FIG. 1 schematically shows the structure of a general-purpose sound-added video file. In FIG. 1, V indicates a video frame, A indicates an audio data block, and F1 to FK indicate frame numbers corresponding to the video frames. In the general-purpose format, it is composed of 30 frames per second. For example, if it is 3 minutes of moving image data, it is composed of 5400 frames. In addition, the video frame V is usually compressed, and a still image may be restored from one video frame V by a compression method. If no other video frame V is used, the still image is restored. Sometimes it is not possible. The acoustic data is divided in units of frames, that is, in units of 1/30 seconds, and recorded as an acoustic data block A. For example, when a stereo sound signal is sampled at a sampling frequency of 48 kHz, 3200 samples corresponding to 1/30 seconds are recorded in one sound data block.

  The sound-added video file as shown in FIG. 1 can be created by a known method. That is, video data and audio data associated with each other obtained by shooting with a video camera or the like may be converted into a sound-added video file format, or video data and audio data obtained separately are associated with each other. It may be recorded. In the present embodiment, it is created by associating and recording separately obtained video data and audio data.

(2. Creation of video file with sound)
Next, creation of a video file with sound in the present embodiment will be described. In the present embodiment, a case will be described in which the present system is used for an apparatus that synthesizes and reproduces audio data recorded as a material in a sound-added video file. In the present system, the number of sound-added video files to be synthesized can be changed as appropriate. In the present embodiment, a case where a maximum of five files can be selected will be described. In this case, the acoustic data is set to five tracks and reproduced. However, if, for example, five songs can be selected for each track, a total of 25 acoustic data are required. Therefore, first, analog sound data obtained by recording or the like is digitized to obtain 25 digital sound data. Digitization of an analog acoustic signal is performed using a conventional general PCM technique. Specifically, an analog audio signal is sampled at a predetermined sampling frequency, and an amplitude is converted into digital data using a predetermined number of quantization bits. The acoustic data digitized in this way becomes a set of time series of samples having a value corresponding to the number of quantization bits. For example, when the sampling frequency is 48 kHz and the number of quantization bits is 16 bits, the analog sound signal for 1 second is converted into digital sound data composed of 48000 samples having values of −32768 to 32767.

  In order to synthesize a plurality of pieces of sound data and reproduce them as one piece of reproduced sound data, it is necessary to process the sound data to be synthesized to have the same reproduction time. This is based on one acoustic data so that each sample of other acoustic data (quantized with a predetermined number of bits at each time) synchronizes temporally and musically with the reference material acoustic signal. Process to adjust to. In this embodiment, the acoustic data is divided into parts such as melody, chord, rhythm, etc. so that the user who reproduces can freely change the music composition. Each acoustic data is obtained by converting an analog acoustic signal into digital data by a technique such as PCM as described above.

  Here, each acoustic data set as five tracks will be described. FIG. 2 schematically shows the signal waveform of the acoustic data of each track. In the example of FIG. 2, a case is shown in which each piece of acoustic data is stereo acoustic data composed of two left and right (LR) channels. In FIG. 2, for simplification of description, a portion where the amplitude value of the signal has a level higher than a certain level, that is, a non-silence portion, shows a waveform with the same amplitude, and a silence portion is shown with no waveform. . When combining and playing music from multiple tracks and making it possible to select multiple songs for each track, to ensure that the combined music is decent regardless of the combination In addition, each acoustic data needs to be created according to a predetermined rule. Therefore, each track is configured such that, regardless of which piece of music is selected, the temporal positions of the silent section and the silent section are basically the same. That is, for example, each of the five acoustic data prepared for the track 1 has a non-silent part (sound part) and a silent part at the same position in order to avoid poor musical change. The lengths of the non-silence portion and the silence portion are also changed somewhat within a range that does not hinder the music rules.

  When the acoustic data having the waveforms as shown in FIG. 2 are synthesized and reproduced, the sound from the track 1 and the track 5 is first heard, and then the sound from the track 5 disappears and the sound from the track 3 is heard. Next, the sound from track 1 and track 3 disappears and the sound from track 2 and track 4 is heard. Next, the sound from tracks 2 and 4 disappears and the sound from tracks 1 and 3 Then, the sound from the track 3 disappears, the sound from the track 5 is heard, and finally the sounds from the track 1 and the track 5 disappear.

  On the other hand, the video data is photographed according to the sound content recorded in the sound data, and the photographed data is compression-coded by a predetermined method. For example, when the acoustic data of a musical instrument representing each country is set for each track, an image of the scenery of the country is captured as image data. Subsequently, the separately obtained video data and audio data are integrated into one sound-added video file, which corresponds to an audio data block corresponding to the time corresponding to each frame of the video data. This is done by recording. For example, when video data is sampled at 30 frames / second and audio data is sampled at 48 kHz stereo, 3200 samples are recorded as one audio data block in association with one video frame. That is, it is recorded in the form as shown in FIG.

  In the present embodiment, data is further processed and recorded for the menu screen. Specifically, the first 10 seconds are extracted from the 25 sound-added video files, and a menu video file and a demo playback sound file are created. The menu video file generates a menu composite frame by synthesizing 25 pieces of video data for 10 seconds extracted from the sound-added video file for each video frame. Thus, a menu video file composed of 300 menu composite frames is obtained. On the other hand, the demo playback audio file is obtained by continuously recording 10-second audio data blocks extracted from the sound-added video file. In addition, when a video is synthesized and played back, a background video file, which is video data for background video, is prepared separately as the background. The background video file is structurally a normal video file of 30 frames per second.

(3. System configuration)
FIG. 3 is a system configuration diagram showing an embodiment of a video signal reproducing apparatus according to the present invention. In FIG. 3, 10 is a video file storage means with sound, 11 is a video file storage means for menu, 12 is an audio file storage means for demonstration reproduction, 13 is a background video file storage means, 20 is a file selection means, and 30 is a video frame. A synthesizing unit, 40 is an audio data block synthesizing unit, 50 is a reproduction timing control unit, 60 is a video frame display unit, and 70 is an audio output unit.

  In FIG. 3, a sound-added video file storage means 10 is a storage device for storing a sound-added video file. The menu video file storage means 11 is a storage device in which a menu video file is recorded. The demo playback sound file storage means 12 is a storage device that records a demo playback sound file. The background video file storage means 13 is a storage device that records a background video file. The video frame synthesizing unit 30 has a function of extracting and synthesizing video frames from the sound-added video file. The sound data block synthesizing unit 40 has a function of extracting and synthesizing the sound data block from the sound-added video file. The reproduction timing control means 50 has a function of controlling the reproduction timing of the synthesized video frame and audio data block. The video frame display means 60 has a function of writing a video frame synthesized in accordance with the instruction timing of the reproduction timing control means 50 into a display memory and displaying it on the screen. The sound output means 70 has a function of outputting the synthesized sound data block as sound according to the instruction timing of the reproduction timing control means 50. Specifically, a process of writing a synthesized sound data block attached to a computer to a sound device and reproducing the sound is performed. The system shown in FIG. 3 is actually realized by hardware such as a computer and its peripheral devices, and dedicated software installed in the computer.

(4. Processing operation)
Next, the processing operation of the system shown in FIG. 3 will be described. First, when the system is activated, a menu screen as shown in FIG. 4 is displayed. In FIG. 4, E is a video display area for displaying a video, and the video is divided into 25 sections of a video section E11 to a video section E55. C1 to C5 are cursors indicating that a video section is selected, and one cursor is prepared for each line. That is, one video can be selected per line, and in the initial state, the leftmost video section is selected as shown in FIG. On this menu screen, it appears that 25 videos are displayed as shown in FIG. 4, but the system only performs the process of playing back the video file for the menu. One video is displayed at a pace of 30 frames per second. However, since 25 different video frames are originally synthesized, the content of the video is different for each video section when viewed from the viewing side. Specifically, this display processing is performed by the video frame display means 60 extracting the menu video file from the menu video file storage means 11 and playing it back.

  On such a menu screen, the user selects a video file with sound to be synthesized and reproduced. Specifically, this is performed by the user clicking on the displayed video section. The user selects one video section for each row, and a total of five video sections are selected. When the user clicks on another video section, the cursor moves on the clicked video, and a demo playback sound file corresponding to the video section is extracted from the demo playback sound file storage means 12 and is output as a sound output. Reproduced and output by means 70. That is, for the user, the sound corresponding to the clicked video section can be heard on the spot. As described above, the demo playback sound file has a length of about 10 seconds and is used by the user to confirm the contents of a performance or the like to be synthesized and played back. In this way, the user selects one video section for each row. The state of the screen after the user has selected is shown in FIG. In the example of FIG. 5, the video segment E12 is selected in the first row, the video segment E23 is selected in the second row, the video segment E35 is selected in the third row, the video segment E41 is selected in the fourth row, and the video segment E53 is selected in the fifth row. It is shown that.

  Here, the structure of such a menu screen will be described. The menu screen is composed of three layers of a material selection button group, a menu video frame, and a cursor overlay window as shown in FIG. Then, a menu video frame is superimposed on the material selection button group, and a cursor overlay window is further superimposed on the menu video frame, whereby display in the video display area shown in FIGS. 4 and 5 is performed. When the user clicks on the video section, the material selection button below it reacts, and the cursor placed on the corresponding line on the cursor layer moves. For example, when the user clicks the video section in the third row and the third column as shown in FIG. 6D on the menu screen, the cursor in the third row starts from the first column as shown in FIG. It will move to the third row. At this time, the audio file for demo reproduction corresponding to the selected video section is reproduced. When the play button is clicked in a state where five video sections are selected, a sound-added video file corresponding to the selected video section is synthesized and played back.

  Hereinafter, the combined reproduction of the selected video file with sound will be described. When the synthesis / playback instruction is given, the video frame synthesizing unit 30 extracts the background video frame from the background video file stored in the background video file storage unit 13 and starts from the selected five video files with sound. A video frame is extracted and superimposed on the background video frame to generate a composite video frame. At this time, if the video frame is compressed, the video frame is extracted and then decoded and synthesized into the synthesized video frame. The synthesized video frame is displayed on the screen by the video frame display means 60. The state of the composite video display screen at this time is shown in FIG. As shown in FIG. 7, the selected video is synthesized and displayed. At this time, the video frame synthesizing unit 30 performs a process of synthesizing the selected video frame of each sound-added video file with a predetermined pixel address and synthesizing it into one synthesized video frame. A composite video frame is created so that two video frames do not overlap each other.

  On the other hand, the audio data block synthesizing unit 40 extracts and synthesizes an audio data block corresponding to the first video frame from the selected five video files with sound. Specifically, one synthesized audio data block is created by calculating the sum of values corresponding to the same time for the audio data block attached to the video frame of the video file with sound. Then, according to the instruction from the reproduction timing control means 50, the sound output means 70 reproduces the synthesized sound data block as sound. The processing by the video frame synthesizing unit 30, the video frame display unit 60, the audio data block synthesizing unit 40, and the audio output unit 70 as described above is performed on all frames of the sound-added video file according to the instruction of the reproduction timing control unit 50. It is repeated until it is done. As a result, the synthesized video is changed and displayed on the synthesized video display screen shown in FIG. 7, and the corresponding acoustic data is reproduced as sound. The reproduction timing control means 50 controls to execute video frame synthesis, display processing, audio data block synthesis, and output processing at 1/30 second intervals corresponding to the video frame intervals. As described above, in order to synthesize and display the video of the selected video file with sound, even if the processing capacity of the CPU is low, there is a video file with sound between the video files with sound. Is no longer delayed in display processing compared to other video files with sound. Therefore, the sound data blocks recorded in association with the frames in each sound-added video file are also reproduced simultaneously without shifting between the sound-added video files. For this reason, the present invention is particularly effective when a performance by each instrument constituting an ensemble is recorded in each sound-added video file.

  In addition, in this system, since processing is performed on a sound-added video file that is recorded by associating an audio data block with a video frame, the user can add a sound-added video in a format commonly used by computers. If even a file is prepared and stored in the sound-added video file storage means 10 of this system, it can be used in this system. In other words, as a user, it is possible to give variations to video and music content to be played by simply preparing a video file with sound as a material and replacing multiple video files with sound to be synthesized and played as needed. It is also possible to provide environmental images and BGM that change over a long period of time.

  The preferred embodiment of the present invention has been described above. However, the present invention is not limited to the above embodiment, and various modifications can be made. For example, in the above embodiment, when the selected video is synthesized and displayed, it is synthesized with the background video, but the background video may not be used. Further, in the above embodiment, when displaying a menu screen for selecting a sound-added video file, a menu video frame file in which a plurality of videos are combined is created and displayed. May be displayed as it is. For example, in the above example, 25 videos are processed and displayed independently, so depending on the processing power of the CPU, the last processed video may not be in time, and the top There may be a deviation from the video processed by the. However, when the menu is displayed, the reproduction of the audio data for demonstration reproduction is performed independently without synchronizing with the video frame, so there is no problem in the deviation between the displayed videos.

  In the above embodiment, the audio data is not compressed, but the compressed data may be recorded in a video file with sound in association with each video frame. In this case, the sound output means 70 is required to have a function of decoding and reproducing in accordance with the compression encoding method. However, since the amount of audio data is smaller than that of video data, there is not much problem in this system that handles a large amount of video frames.

  Further, in the above embodiment, when the video frames are combined and played back, no processing other than the synthesis is performed, and the images are displayed side by side as shown in FIG. 7, but processing such as masking of each video is performed. May be. In this case, mask data for identifying whether to overwrite each pixel is attached to the video frame of the video file with sound. Then, when synthesizing the synthesized video frame, the video frame synthesizing means 30 synthesizes the synthesized video frame with respect to some pixels of the selected video frame of the video file with sound based on the mask data. Do not control.

It is a figure which shows the structure of the moving image file with a sound made into the object of composition reproduction by this invention. It is a figure which shows the signal waveform of each acoustic signal made into a synthetic | combination object. It is a functional block diagram which shows one Embodiment of the moving image reproduction system with sound which concerns on this invention. In this system, it is a figure which shows the initial state of the menu screen for selecting the object to synthesize | combine and reproduce. It is a figure which shows the state of the menu screen in the state as which the object of synthetic | combination reproduction was selected by the user. It is a figure which shows the structure of each layer which comprises the video display area of a menu screen. It is a figure which shows the state which synthesize | combined and displayed the selected moving image.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Sound-added video file storage means 11 ... Menu video file storage means 12 ... Demo reproduction sound file storage means 13 ... Background video file storage means 20 ... File selection means 30 ... Video frame synthesizing means 40 ... sound data block synthesizing means 50 ... reproduction timing control means 60 ... video frame display means 70 ... sound output means

Claims (10)

Sound-added video file storage means having a plurality of video frames and storing a plurality of sound-added video files each having an audio data block associated with each video frame;
File selection means for selecting a plurality of sound-added video files to be synthesized and reproduced from the sound-added video file storage means;
Video frame synthesizing means for extracting corresponding video frames from the plurality of selected video files with sound and synthesizing them into one synthesized video frame;
An audio data block synthesizing unit that extracts an audio data block attached to each corresponding video frame from the plurality of selected video files with sound, and synthesizes the audio data block into one synthesized audio data block;
Video frame display means for writing the synthesized video frame to a display memory and displaying it on a screen;
Sound output means for writing the synthesized sound data block to a sound device and reproducing the sound;
Each process in the video frame synthesizing unit, the audio data block synthesizing unit, the video frame displaying unit, and the audio output unit is performed on all video frames stored in the selected video file with sound. Reproduction timing control means for controlling to execute sequentially at time intervals;
An apparatus for reproducing video with sound, comprising: In claim 1,
The file selection means is configured to display a menu composite frame created by combining video frames extracted from a predetermined sound-added video file among sound-added video files stored in the sound-added video file storage means. A sound-added video reproduction apparatus, wherein a sound-added video file to be synthesized and reproduced is selected by displaying on a writing screen in a memory and selecting a video section on the screen. In claim 2,
It has a function of preparing an audio file for demo reproduction in which only audio data is extracted from a video file with sound in advance, and reproducing an audio file for demo reproduction corresponding to the video section selected on the screen. To play video with sound. In claim 1,
The image size of each video frame of the selected video file with sound and the number of video frames to be combined are set so as to fit in a display memory when combined, and the video frame combining means An apparatus for reproducing a video with sound, characterized in that the video frame of the selected video file with sound is offset by a predetermined pixel address and synthesized into one composite video frame. In claim 1,
The video frame synthesizing unit captures a background video frame stored in a background video file which is a background video file as an initial image when synthesizing the synthesized video frame, and then selects the selected video file with sound. A video-with-sound reproduction apparatus characterized in that the video frames are sequentially superimposed on a single composite video frame. In claim 1,
Mask video data for identifying whether to overwrite each pixel is attached to the video frame of the selected video file with sound, and the video frame synthesis means is configured to synthesize the synthesized video frame. A part of pixels of the video frame of the selected video file with sound is controlled not to be combined with the composite video frame based on the mask data. Playback device. In claim 1,
The sound data block synthesizing unit calculates the sum of the corresponding values of the sound data blocks with respect to the sound data blocks attached to the video frames of the selected plurality of sound-added video files. An apparatus for reproducing video with sound, which creates two synthetic audio data blocks. In claim 1,
The video frames of the audio video file stored in the audio video file storage means are recorded in a compressed and encoded state by a predetermined method, and the video frame synthesis means An apparatus for reproducing a video with sound, wherein a video frame is extracted from an attached video file, decoded and then synthesized into a synthesized video frame. In claim 1,
The sound data block recorded in the sound-added video file stored in the sound-added video file storage means is recorded in a state compressed and encoded by a predetermined method, and the sound data block combining means The sound-added video is characterized in that one audio data block attached to the video frames of the selected plurality of sound-added video files is extracted and then decoded to create one composite audio data block. Playback device. A program for causing a computer to function as the playback device according to claim 1.

Images