JP2010086615A - Multiplexing device, program, and multiplexing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a multiplexing device for generating a moving image file which can efficiently perform post edition processing, in a non-fragment system file of a moving image-encoding file format. <P>SOLUTION: When video data and voice data are multiplexed and the moving image encoding file of a non-fragment system is generated, mdat boxes equivalent to the number of scenes of video data are generated as shown in Fig. 202. The video data and voice data are divided for each scene of video data, divided video data and voice data are multiplexed, and media data are stored in order from the beginning, that is, from mdat box of the head. Also, moov box is generated and meta-data are stored. The moving image-encoding file can be cut off for each mdat box defined for each scene while keeping the non-fragment form as it is. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a multiplexing apparatus that creates media data such as video data and audio data suitable for later editing processing, and more particularly to a technology for multiplexing media data in a non-fragment format.

  In recent years, with the increase in capacity of communication networks and the advancement of transmission technology, video distribution services that distribute video files including multimedia content such as video, audio, text, or still images to personal computers over the Internet have become widespread. It is remarkable. In addition, the video distribution service is expected to expand to mobile terminals such as mobile phones and PDAs in the future.

  Along with this, video and audio coding formats have diversified, and there is an increasing need to be able to process various formats in a form that can be interconnected within a uniform framework. Therefore, the ISO / IEC (International Standardization Organization / International Engineering Consortium) JTC1 / SC29 / WG11 uses a general-purpose file format called “ISO Base Media file format” to record video and audio content data such as MPEG. It has been standardized.

For example, in the case of MP4, which is an extended format, as an editing method in the case of performing an editing process in which moving images of different files are cut out and combined into one MP4 file, each is multiplexed in the mdat box of the MP4 file. Demultiplexing (demux) was performed on the actual data, and the data was extracted as video data and audio data. Then, the data was cut out and rearranged, and the edited data was multiplexed again into the MP4 format file. In this process, there are many detour procedures such as separating previously multiplexed content data and then multiplexing again, and the data processing load during editing is large.

In order to solve the above-mentioned problem, a data file editing method has been proposed that can efficiently perform editing processing on a fragment-type MP4 file in which content data is divided in advance (see Patent Document 1).
JP2006-129081

  When performing edit processing to extract a predetermined moving image from an MP4 file that has not been fragmented in advance, or to extract a moving image of different MP4 files and link them to one MP4 file, the multiplexed content data is temporarily multiplexed. It must be separated and multiplexed again after editing. In the fragment method, an editing method in divided units has been proposed, but it cannot be applied to non-fragment content data.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a multiplexing apparatus for creating an MP4 file capable of efficiently performing subsequent editing processing in a non-fragmented file of the MP4 file format. And

A multiplexing device according to the present invention is a multiplexing device for multiplexing video data and audio data into a non-fragmented moving image encoded file format format having metadata and media data, the video data And means for acquiring the audio data; means for dividing the video data and the audio data for each scene included in the video data at a randomly accessible point of the video data; and Means for defining the number of mdat boxes; means for multiplexing the divided video data and audio data; means for storing the multiplexed media data in order from the first mdat box from the first data; It is characterized by comprising.

A program according to the present invention is a computer-executable program that multiplexes video data and audio data into a non-fragmented moving image encoded file format format having metadata and media data. A procedure for obtaining media data composed of audio data; a procedure for dividing the video data and the audio data for each scene included in the video data at a randomly accessible point of the video data; and the divided video A procedure for defining mdat boxes for the number of data, a procedure for multiplexing the divided video data and audio data, and the multiplexed video data and audio data from the first data to the first mdat box It is characterized by having a computer execute the procedure of storing in order.

A multiplexing method according to the present invention is a multiplexing method for multiplexing video data and audio data into a non-fragmented moving image encoded file format format having metadata and media data, the video data and Media data consisting of audio data is acquired, and the video data and the audio data are divided for each scene included in the video data at a random accessible point of the video data, and the number of the divided video data The mdat box of minutes is defined, and the encoded data obtained by multiplexing the divided video data and audio data is stored in order from the first data to the first mdat box.

  According to the present invention, when MP4 files are multiplexed, media data is divided for each scene, and each media data is multiplexed and stored in different mdat boxes. By doing so, editing for each scene can be cut out in units of desired mdat boxes and rearranged. Therefore, the MP4 file editing process can be efficiently performed without performing a detour process such as multiplexing and remultiplexing of media data in the mdat box in the non-fragment format.

Embodiments of the present invention will be described below with reference to the drawings.

  The above-mentioned general-purpose file format called “ISO Base Media file format” is defined as a basic file format that does not assume a specific encoding format. In order to adapt to a predetermined encoding format and purpose, there is a feature that a standard partially expanded from this standard is separately defined. A typical example of this extension is the MP4 file format.

  When distributing a media file in a distribution service, first, the multiplexing device captures media data such as moving images, still images, audio, and text, and the header information and media data required for reproducing the media data. Although it is necessary to multiplex the entity data with the media file data, the MP4 file format is currently attracting attention as a moving image file format and is expected to be widely used in the future. In this embodiment, an MP4 file format will be described as an example.

  Here, the data structure of the MP4 file will be described. FIG. 1 is a diagram showing an example of a data structure in the MP4 format. As shown in FIG. 1, the MP4 format file data is largely composed of two data structures of mdat box and moov box.

Here, an area for storing a single piece of data is expressed in units of box. In this box, the size (Size) of the box in bytes is stored in the first 4 bytes, the type (Type) of the box is stored in the subsequent 4 bytes, and the total of 8 bytes is used as a header, and It has a structure added to the head. Therefore, the size and type of the first box can be known from the first 8 bytes of the MP4 file, and only the size of the first box is moved from the beginning of the file (only the position obtained by adding the size from the beginning position of the file). The subsequent box can be accessed.

An mdat box is an area that stores encoded video and audio data entities, and a moov box stores metadata such as physical and temporal positions of video and audio data, and characteristic information. It is an area. Boxes can be recorded not only continuously in a file, but also can have several boxes in a box, such as the track box in the moov box shown in Fig. 1. Can do. Here, the track box is an area that exists in the moov box and shares and holds management information necessary for reproducing sound, images, and the like.

The MP4 file format allows not only the form of recording all the metadata in the moov box, but also the form of recording the metadata by dividing it into a plurality of areas in chronological order. This format is called "Fragmented Movie".

When creating media data in MP4 file format, the video and audio data is captured and multiplexed by the multiplexing device as described above, the media data is stored in the mdat box, and the metadata is stored in the moov box. create.

FIG. 2 shows an example of a conventional MP4 file 201 that has been multiplexed using a conventional multiplexer, and an MP4 file 202 that has been multiplexed using the present invention.

When multiplexing processing is performed using a conventional multiplexing processing apparatus, a non-fragmented MP4 file is output in a format of one moov box and one mdat box as in the MP4 file 201. On the other hand, the MP4 file 202 output from the multiplexing device in the present embodiment is composed of one moov box and one or more mdat boxes. Media data stored in this mdat box is stored in chronological order from the top mdat box. When normal playback of the MP4 file 202 is performed, the media data stored in the first mdat box is sequentially played back to the media data in the next mdat box. By adopting the MP4 file 202 format, each mdat box can be cut and edited without being demultiplexed.

FIG. 3 is a block diagram showing an example of the configuration of the information processing apparatus for realizing the multiplexing processing in the present embodiment. FIG. 3 shows a CPU 301, M memory 302, HDD 303, multiplexing device 304, ODD 305, network I / F 306, audio I / F 307, video I / F 308, and bus 309.

A CPU 301 is a central processing unit and controls the entire information processing apparatus. It also has a function of executing a program and executing a predetermined process corresponding to the program.

The M memory 302 is composed of a semiconductor memory, and is used as a program and data storage area when the CPU 301 processes the program.

The HDD 303 is a magnetic disk device, for example, and is used as a non-volatile area for storing data. The stored program and data can be read out according to an instruction from the CPU 301.

The multiplexing device 304 is a module that multiplexes video data and audio data in accordance with the MP4 file format to generate an MP4 format file. The multiplexing device 304 acquires video data and audio data via the bus 309 and outputs the multiplexed MP4 data to the HDD 303. In this embodiment, the multiplexed MP4 data is output to the HDD 303. However, the MP4 data may be written to the ODD 305 or transmitted to another information processing apparatus via the network I / F 306. Also good.

The ODD 305 is, for example, an optical disk device, and writes and reads data and programs to and from an inserted optical disk such as a CD (Compact Disc) or a DVD (Digital Versatile Disc).

A network I / F 306 transmits data to other information processing apparatuses and the like via a network such as a LAN or a WAN.

The audio I / F 307 is an interface for acquiring external sound such as a microphone, and has a function of converting the external sound into an electric signal and inputting it to the information processing apparatus.

A video I / F 308 is an interface for acquiring an external video such as a digital camera, and has a function of converting the external video into an electric signal and inputting it to the information processing apparatus.

Each module is connected to the bus 309 so that the modules can communicate with each other.

The configuration of each unit of the multiplexing device 304 may be realized by software. In this case, a program that can be executed by the CPU 301 that realizes the function of each configuration is stored in the HDD 303 in advance, and is read out and executed on the memory 302 during processing. In the case of being configured by software, the program is not only stored in the HDD 303, but the program stored in the optical disk may be directly read from the ODD 305. Or you may make it acquire from network I / F306.

In the present embodiment, the video data and audio data to be multiplexed using the multiplexing device 304 may be data obtained separately from the audio I / F 307 and the video I / F 308. In the case of data acquired separately from the audio I / F 307 and the video I / F 308, each is independently input to the multiplexer 304 and multiplexed. In addition, data that has already been multiplexed is demultiplexed (demultiplexed), video data and audio data are separated and then input to the multiplexer 304. These data input to the multiplexing device 304 may be data stored in the HDD 303 or the ODD 305, or may be acquired from the network I / F 305. The data acquisition means is not limited to the above example.

FIG. 4 is a block diagram showing an example of the configuration of the multiplexing device 304 in the present embodiment. FIG. 4 shows a multiplexing device 304, a video data specifying unit 401, a video data analyzing unit 402, an audio data specifying unit 403, an audio data analyzing unit 404, an MP4 multiplexing unit 405, and an MP4 data output unit 406. .

The video data designation unit 401 has a function of acquiring video data from the outside of the multiplexing device 304 via the bus 309 and outputting the video data to the video data analysis unit 402.

The video data analysis unit 402 has a function of analyzing the video data generated by the video data specifying unit 401 and extracting information necessary for subsequent multiplexing processing. Also, scene information, encoding information for each scene, and the like are extracted from the video data, and the extracted scene information and information necessary for the multiplexing process are output to the MP4 multiplexing unit 405. Here, the scene is obtained by dividing video data at random accessible points.

The audio data designation unit 403 has a function of acquiring audio data from the outside of the multiplexing device 304 via the bus 309 and outputting the audio data to the audio data analysis unit 404.

The voice data analysis unit 404 analyzes the voice data acquired from the voice data designation unit 403 and extracts information necessary for the multiplexing process. Also, it has a function of outputting the extracted information to the MP4 multiplexing unit 405.

The MP4 multiplexing unit 405 has a function of multiplexing video data and audio data into the MP4 file format and outputting the multiplexed MP4 data to the MP4 data transmitting unit. When creating the MP4 file, the MP4 multiplexing unit 405 divides the video data and audio data into scene units extracted from the video data. After that, create mdat boxes for the number of scenes included in the video data, multiplex the video data and audio data in the previously divided units, and store the multiplexed video data and audio data for each scene from the top mdat box Store in order. With this process, an MP4 file 202 is generated.

In addition, the MP4 multiplexing unit 405 defines a sample description box for storing a sample description, which is encoding information of media data to be multiplexed, for the number of mdat boxes (for the number of scenes) in the stsd box in the moov box. To do. This is because the input video data may have different encoding information for each scene.

At this time, the sample description describing the scene stored in the first mdat box is stored in the first sample description box, and the sample description describing the scene stored in the Nth mdat box is stored in the Nth sample description box. To do. In other words, the first sample description is sequentially stored in the first sample description box. This Sample description box is stored in the stsd box in the moov box. Here, the Sample description box and the stsd box are one of the trak boxes described above and are of other layers.

Sample_description_index is also created corresponding to the number of mdat boxes and stored in the stsc box in the moov box. For example, if there are N mdat boxes and they are not common to all sample descriptions describing each scene, “Sample_description_index = N” is defined. If Sample description is common in all scenes, it is defined as “Sample_description_index = 1”.

The MP4 data output unit 406 has a function of outputting the MP4 data multiplexed by the MP4 multiplexing unit 405 to the outside.

FIG. 5 is a schematic diagram illustrating an example of the structure of the moov box after multiplexing and stored information when the number of scenes in the present embodiment is N. Create and store url boxes indicating the location of each mdat box in the dref box stored in the moov box (the number of scenes). FIG. 5 shows a schematic diagram of the structure of the moov box and stored information when the number of scenes is N. In FIG. 5, the hierarchical structure in the moov box is based on the MP4 file format, and the upper hierarchy is omitted.

FIG. 6 is a diagram showing an example of a scene dividing method of video data and audio data before being multiplexed. In general, encoded video data has a limited number of points that can be randomly accessed according to a frame encoding unit, and scene division here is always performed at a point where random access is possible. The actual scene division method depends on the implementation at the time of implementation, and if it is short video data, it is possible to divide at all points where random access is possible, and video with a certain length If it is data, it is conceivable to divide at a point where random access is possible at every elapse of a predetermined time, or to divide a scene at a point where random access is possible where information of video data is largely switched. Then there is no particular limitation. However, no matter what division method is used, the scene is different at the point where the sample description changes, and division is performed.

  Regarding audio data, there is no point at which random access is impossible, and any point can be divided. Therefore, scene division is performed at a location corresponding to a scene division location of video data as shown in the figure.

  FIG. 7 is a flowchart showing an example of a processing flow of the multiplexing device 304 in the present embodiment.

  First, the video data designating unit 401 and the audio data designating unit 403 acquire the video data and the audio data from the outside of the multiplexing device 304 in separate data formats (S701). Next, the video data analysis unit 402 and the audio data analysis unit 404 respectively analyze the received video data and audio data (S702), and MP4 multiplexing of the scene information obtained by the analysis and information necessary for the multiplexing process is performed. The data is transmitted to the processing unit 405. Based on the received information, the MP4 multiplexing processing unit first determines whether the number of scenes in the video data is two or more (S703). If the number of scenes is 1 (No), scene division is not performed, so one mdat box is created and all media data is stored therein. If the number of scenes is 1, since there is only one Sample description, Sample_description_index = 1 is set for Sample_description_index (S704). If the number of scenes is 2 or more in S703 (Yes), it is next determined whether or not the Sample description is common to all scenes (S705). If the sample description is common to all scenes (Yes), mdat boxes are created for the number of scenes, and media data for each scene is stored in time series from the first mdat box. Also create a corresponding url box. Furthermore, since Sample description is common to all scenes, Sample_description_index = 1 is set (S706). If there are scenes having different sample descriptions in the media data in S705 (No), mdat boxes and url boxes for the number of scenes are created as in S706. Sample description corresponding to the media data stored in each mdat box is prepared for each mdat box, and Sample_description_index = N is set (S707). With these processes, the correspondence between the media data stored for each mdat box and the sample description stored in the moov box can be determined. Next, as the next flow of S704, S706, and S707, the multiplexing apparatus 304 constructs MP4 data in accordance with the MP4 file format (S708). Finally, the constructed MP4 data is output from the MP4 data output unit 406 to the outside of the multiplexing device 304, and a series of multiplexing processing flow ends.

  By the above method, there are several mdat dox scenes, and mMP4 data storing each scene can be created. Here, an editing processing method in the case where new MP4 data is created using the MP4 data obtained by the present embodiment as original data will be described.

  FIG. 8 is a diagram showing an example of a method for editing MP4 data obtained by the multiplexing process of this embodiment. In the present embodiment, consider an editing process in which mdat boxes in the obtained MP4 data (a), (b), and (c) are cut out and combined into one MP4 data (d).

As shown in FIG. 8, mdat dox [A], mdat dox [C] from MP4 data (a), mdat dox [B] from MP4 data (b), mdat dox [D] from MP4 data (c) When the editing process is performed to combine the data into one MP4 data (d), it is not necessary to perform a process such as demultiplexing and multiplexing again like the MP4 data multiplexed by the conventional multiplexing apparatus. . When the same editing is performed on the MP4 data obtained in the present embodiment, mdat dox [A], [B], [C], and [D] are cut out from the respective MP4 data and stored simply by arranging them. The editing process related to the mdat box to be completed is completed. Later, the edited MP4 data is completed by creating metadata in the moov box so as to correspond to the entity data. Here, since the size of the box is stored as header information at the top of the mdat box as well as other boxes, this may be used when creating metadata.

In this embodiment, it is not necessary to perform re-multiplexing or the like on the entity data in the mdat box, it can be edited only by arranging the mdat boxes, and only the metadata in the moov box is created. Since editing is possible, the processing is simple and the data processing burden on the information processing apparatus is light.

Since the corresponding sample description exists in the mdat box cut out from the original MP4 data, the sample description is the MP4 data (a), (b), The sample description in (c) can be used as it is. FIG. 9 shows the correspondence between the mdat box of the original MP4 data and the sample description stored in the moov box when extracted during editing.

Here, the correspondence between the mdat box of the original MP4 data and the extraction when editing the sample description will be described with reference to FIG. In FIGS. 9A to 9C, the MP4 data of the editing source is classified and shown.

FIG. 9A shows the correspondence between the mdat box and the sample description when “the number of mdat boxes = 1”. Since the number of mdat boxes = 1 (number of scenes = 1), the sample description is naturally one, so when using the MP4 data mdat box, mdat 1 is associated with Sample description 1 and used for editing. .

FIG. 9-B shows a case where “the number of mdat boxes = N, common sample description”. This is the case where the number of mdat boxes = N and the number of scenes = N, but the sample description is common in all scenes (Sample_description_index = 1). At this time, when any mdat box is cut out, Sample description 1 is associated and used for editing.

FIG. 9C shows a case where “the number of mdat boxes = N, there is a non-common Sample description”. This is the case where the number of mdat boxes = N and the number of scenes = N, and there is a sample description that is not common to each scene (Sample_description_index = N). At this time, since the sample description corresponding to the mdat box is stored in the moov box in the same order, the mdat box and the sample description in the same order are associated with each other and used for the editing work.

As described above, by using the sample description corresponding to each mdat box from the original MP4 data, editing can be performed without worrying about the encoding information for each mdat box. In this embodiment, at the time of multiplexing MP4 data, only the sample description and the like are created and illustrated corresponding to each mdat box. However, the present invention is not limited to this, and other metadata is also included in the mdat box. If it is created so that it corresponds, it can be used in the same way as Sample description when editing.

  Further, in the present embodiment, the editing work for cutting out the MP4 multiplexed by the multiplexing device 304 as the editing work and creating new MP4 data is illustrated, but the present invention is not limited to this, and the multiplexed MP4 data is used. Since it is possible to move data freely in units of mdat boxes, such as deletion of mdat boxes from, and addition of mdat boxes to other MP4 data, various editing processes can be applied.

  In the present invention, it is possible to create an MP4 file that can be efficiently processed later in a non-fragmented file in the MP4 file format.

Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

The figure which shows an example of the data structure of MP4 format. The figure which shows an example of the data structure of the conventional MP4 file which performed the multiplexing process using the conventional multiplexing apparatus, and the MP4 file which performed the multiplexing process in this embodiment. 1 is a block diagram showing an example of the configuration of an information processing apparatus for realizing multiplexing processing in the present embodiment The block diagram which shows an example of a structure of the multiplexing apparatus in this embodiment. Schematic showing an example of the structure of moov box after multiplexing and stored information when the number of scenes in this embodiment is N The figure which shows an example of the division | segmentation method of the scene of the video data before multiplexing processing in this embodiment, and audio | voice data Flow chart showing an example of the processing flow of the multiplexing device in the present embodiment The figure which showed an example of the edit method of MP4 data obtained by the multiplexing process of this embodiment The figure which shows an example of a response | compatibility at the time of editing of mdat box and Sample description.

Explanation of symbols

201: MP4 file 202 in the conventional format 202: MP4 file 301 multiplexed in the present embodiment 301: CPU
302: M memory 303: HDD
304: Multiplexer 305: ODD
306: Network I / F
307: Voice I / F
308: Video I / F
309: Bus 401: Video data specifying unit 402: Video data analyzing unit 403: Audio data specifying unit 404: Audio data analyzing unit 405: MP4 multiplexing unit 406: MP4 data output unit

Claims (9)

A multiplexing device for multiplexing video data and audio data into a non-fragmented moving image encoded file format format having metadata and media data,
Means for obtaining the video data and the audio data;
Means for dividing the video data and the audio data for each scene included in the video data at a randomly accessible point of the video data;
Means for defining mdat boxes for the number of divided video data;
Means for multiplexing the divided video data and audio data;
Means for sequentially storing the multiplexed media data from the first data to the first mdat box;
A multiplexing apparatus comprising: If the encoding information of the media data stored in each of the mdat boxes is common, determine one Sample description corresponding to the media data,
Means for determining a plurality of sample descriptions corresponding to each of the media data if the encoding information of the media data is not common;
2. The multiplexing apparatus according to claim 1, further comprising means for defining a Sample description box for storing the Sample description by the number of Sample descriptions.   The multiplexing apparatus according to claim 2, further comprising means for defining the number of Sample description boxes as a value of Sample_description_index. A computer-executable program that multiplexes video data and audio data into a non-fragmented moving image encoded file format format having metadata and media data,
A procedure for acquiring media data composed of video data and audio data;
A procedure of dividing the video data and the audio data for each scene included in the video data at a randomly accessible point of the video data;
A procedure for defining mdat boxes for the number of divided video data;
A procedure for multiplexing the divided video data and audio data;
A procedure for storing the multiplexed video data and audio data in order from the first data to the first mdat box;
A program characterized by causing a computer to execute. If the encoding information of the media data stored in each of the mdat boxes is common, determine one Sample description corresponding to the media data,
A procedure for determining a plurality of sample descriptions corresponding to each of the media data if the encoding information of the media data is not common,
5. The program according to claim 4, further causing a computer to execute a procedure for defining a Sample description box for storing the Sample description for the number of Sample descriptions. 6. The program according to claim 5, further causing a computer to execute a procedure for defining the number of Sample description boxes as a value of Sample_description_index. A multiplexing method for multiplexing video data and audio data into a non-fragmented moving image encoded file format format having metadata and media data,
Obtain media data consisting of video data and audio data,
Dividing the video data and the audio data for each scene included in the video data at a randomly accessible point of the video data,
Define mdat boxes for the number of divided video data, and store encoded data obtained by multiplexing the divided video data and audio data in order from the first data to the first mdat box. Feature multiplexing method. If the encoding information of the media data stored in each of the mdat boxes is common, determine one Sample description corresponding to the media data,
If the encoding information of the media data is not common, determine a plurality of sample descriptions corresponding to each media data,
8. The multiplexing method according to claim 7, wherein after that, a number of sample description boxes for storing the sample description are defined. The multiplexing method according to claim 8, wherein the number of Sample description boxes is defined as a value of Sample_description_index.

Images