JP4114868B2 - Multiplexer and multiplexing method - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a multiplexing device that multiplexes media data such as moving image data and audio data, and a demultiplexing device that reads and demultiplexes a bit string in which media data such as moving image data and audio data is multiplexed. About.
[0002]
[Prior art]
In recent years, with the increase in communication network capacity and advances in transmission technology, the spread of video distribution services that distribute video files including multimedia content such as video, audio, text, or still images to personal computers over the Internet Is remarkable. Also, the international standardization organization 3GPP (Third Generation Partnership Project), which aims to standardize the standard of so-called third generation mobile communication systems such as mobile terminals, is the TS26.234 (Transparent The end-to-end packet switched streaming service has been established, and the video distribution service is expected to be expanded to mobile communication terminals such as mobile phones and PDAs.
[0003]
When distributing a moving image file in a moving image distribution service, first, the multiplexing device captures media data such as a moving image, a still image, audio and text, and header information and media data necessary for reproducing the media data. However, the MP4 file format is attracting attention as a multiplexed file format of the moving image file data.
[0004]
This MP4 file format is a multiplexed file format that is being standardized in ISO / IEC (International Standardization Organization / International Engineering Consortium) JTC1 / SC29 / WG 11 which is an international standardization organization. Because it is adopted, it is expected to be widely spread.
[0005]
Here, the data structure of the MP4 file will be described. The data structure of this MP4 file is disclosed in Non-Patent Document 1.
The MP4 file stores entity information such as header information and media data in units of objects called boxes, and is configured by arranging a plurality of boxes hierarchically.
[0006]
FIG. 18 is a diagram for explaining the structure of a box constituting a conventional MP4 file.
The box 901 stores a box header portion 902 in which header information of the box 901 is stored, and data included in the box 901 (for example, a box in a hierarchy below the box and a field for describing information). A box data storage unit 903.
[0007]
The box header portion 902 has fields of a box size 904, a box type 905, a version 906, and a flag 907.
The box size 904 is a field in which the size information of the entire box 901 including the byte size assigned to this field is described.
[0008]
The box type 905 is a field in which an identifier for identifying the type of the box 901 is described. This identifier is usually represented by four alphabetic character strings. In the following description, each box may be indicated by this identifier.
[0009]
The version 906 is a field in which a version number indicating the version of the box 901 is described, and the flag 907 is a field in which flag information set for each box 901 is described. Since this version 906 and flag 907 are not mandatory fields for all boxes 901, there may be boxes 901 that do not have these fields.
[0010]
An MP4 file composed of a plurality of boxes 901 having such a structure can be roughly divided into a basic part indispensable for the structure of the file and an extended part used as necessary. First, the basic part of the MP4 file will be described.
FIG. 19 is a diagram for explaining a basic part of a conventional MP4 file.
[0011]
The basic part 911 of the MP4 file 910 includes a file header part 912 and a file data part 913.
The file header portion 912 is a portion in which header information of the entire file, for example, information such as a compression encoding method for moving image (video) data, is stored, and includes a file type box 914 and a movie box 915.
[0012]
The file type box 914 is a box identified by an identifier of “ftyp”, and stores information for identifying an MP4 file. As to what kind of media data is stored in the MP4 file, and what kind of compression encoding method is used to store moving image (video) data, audio (audio) data, etc., standardization organizations and services Since the service provider can uniquely define the information, the file type box 914 stores information for identifying which of the MP4 files has been created.
[0013]
The movie box 915 is a box identified by an identifier of “moov”, and stores header information of entity data stored in the file data portion 913, for example, information such as a display time length.
The file data portion 913 includes a movie data box 916 identified by an identifier “mdat”. Instead of the file data portion 913, an external file different from the MP4 file 910 can be referred to. In this way, when referring to an external file, the basic part 911 of the MP4 file 910 is composed of only the file header part 912. In this specification, a case will be described in which entity data is included in the MP4 file 910 instead of referring to the external file.
[0014]
The movie data box 916 is a box for storing media data entity data in units called samples. This sample is the smallest access unit in the MP4 file, and corresponds to the VOP (Video Object Plane) of video data encoded by the MPEG (Moving Picture Experts Group) -4 Visual compression encoding method and the frame of audio data To do.
[0015]
Here, the structure of the movie box 915 will be described by digging down the hierarchy of the structure of the basic part of the conventional MP4 file.
FIG. 20 is a diagram for explaining the structure of a movie box in a conventional MP4 file.
[0016]
As shown in FIG. 20A, the movie box 915 includes the box header portion 902 and the box data storage portion 903 described above. The size information of the movie box 915 is described in the box size 904 field constituting the box header portion 902 (in FIG. 20A, “xxxx”), and the box type 905 field contains An identifier “moov” of the movie box 915 is described.
[0017]
The box data storage unit 903 of the movie box 915 stores header information for each track such as a movie header box 917 storing the header information of the basic unit 911 of the MP4 file 910 and a video track or an audio track. A track box 918 and the like are stored. Here, the track means the entire sample data of each medium included in the MP4 file 910, and tracks such as moving images, audio, and text are referred to as video tracks, audio tracks, text tracks, and the like, respectively. . In addition, when there are a plurality of data of the same medium in the MP4 file 910, a plurality of tracks exist for the same medium. More specifically, for example, when two types of moving image data are included in the MP4 file 910, there are two video tracks.
[0018]
The movie header box 917 also includes the box header portion 902 and the box data storage portion 903 described above, and the size information of the movie header box 917 is stored in the box size 904 field constituting the box header portion 902. In the box type 905 field, the identifier “mvhd” of the movie header box 917 is described. The box data storage unit 903 of the movie header box 917 stores information related to the length of time required to reproduce the content included in the basic unit 911 of the MP4 file 910.
[0019]
Further, in the box size 904 field constituting the box header portion 902 of the track box 918, the size information of the track box 918 is described (in FIG. 20A, “xx”), and the box type 905 is displayed. In the field, an identifier “trak” of the track box 918 is described. A track header box 919 is stored in the box data storage unit 903 of the track box 918.
[0020]
The track header box 919 is a box having a field for describing header information for each track, and is identified by an identifier of “tkhd”. In the box data storage section 903 of the track header box 919, a field for describing a track ID for identifying the type of track, information on a time length required for reproducing the track, and the like are described.
[0021]
As described above, the boxes 901 are hierarchically arranged in the movie box 915, and header information for each track such as video and audio is stored in the track box 918 identified by “trak”. In the lower box included in the track box 918, header information for each track sample is stored.
[0022]
When the structure of the movie box 915 shown in FIG. 20A is shown in a tree shape, a diagram as shown in FIG. 20B is obtained.
That is, a movie header box 917 and a track box 918 are arranged as a lower box group of the movie box 915, a track header box 919 is arranged as a lower box group of the track box 918, and the boxes 901 are arranged hierarchically. You can see that
[0023]
At the beginning of standardization of the MP4 file format, the MP4 file 910 was composed only of the basic unit 911. However, as the amount of media data increases, the size increases, so there are various problems such as difficulty in application to streaming playback, and the use of an extension unit with a plurality of pairs of header boxes and data boxes is required. Improvements have been made.
[0024]
FIG. 21 is a diagram showing a structure of a conventional MP4 file including an extension unit. As shown in FIG. 21, the MP4 file 920 to which the above improvements are added is composed of a basic part 911 and an extension part 921. Since all media data can be stored in the extension unit 921 in the MP4 file 920 including the extension unit 921, the movie data box 916 of the MP4 file basic unit 911 may be omitted.
[0025]
The extension unit 921 is configured by a plurality of packets 922 separated in predetermined units.
This packet 922 includes a pair of a movie fragment box 923 and a movie data box 916, and is also referred to as a movie fragment.
[0026]
The movie data box 916 stores a sample for each track in the predetermined unit divided above, and the movie fragment box 923 is a box for storing header information corresponding to the movie data box 916, and is called “moof”. Identified by an identifier. The structure of the movie fragment box 923 will be described in more detail.
[0027]
FIG. 22 is a diagram for explaining the structure of a conventional movie fragment box.
As shown in FIG. 22, a movie fragment header box 924 and a plurality of track fragment boxes 925 are stored in the box data storage unit 903 of the movie fragment box 923.
[0028]
The movie fragment header box 924 is a box identified by the identifier “mfhd”, and stores header information of the entire movie fragment box 923.
The track fragment box 925 is a box identified by an identifier “traf”, and stores header information for each track.
[0029]
Normally, one track fragment box 925 is prepared for header information of one track, but a plurality of track fragment boxes 925 may be prepared for header information of one track. As described above, when the header information of one track is divided into a plurality of track fragment boxes 925 and stored, the decoding time of the first sample in the track fragment box 925 is arranged in ascending order.
[0030]
In the box data storage unit 903 of the track fragment box 925, a track fragment header box 926 and one or more track fragment run boxes 927 are stored.
The track fragment header box 926 is a box identified by an identifier of “tfhd”, and stores a field describing a track ID for identifying a track type, information on a default value such as a playback time length of a sample, and the like. To do.
[0031]
The track fragment run box 927 is a box identified by an identifier of “trun” and stores header information in units of samples. The track fragment run box 927 will be described in detail with reference to FIG.
FIG. 23 is a diagram for explaining the structure of a conventional track fragment run box 927.
[0032]
The flag 907 is a field in which flag information set for each box 901 is described. Here, each field from the data offset 929 to the sample composition time offset 936 is the track fragment run box 927 following the flag 907. Flag information indicating whether or not the file exists is described.
[0033]
The sample count 928 is a field in which information indicating how many samples of header information are stored in the track fragment run box 927 is described.
The data offset 929 is stored in the movie data box 916 in which the actual data of the sample located at the head of the track fragment run box 927 among the samples whose header information is stored in the track fragment run box 927 is stored. This is a field in which pointer information indicating whether or not
[0034]
The head sample flag 930 is a field in which the value of the field of the sample flag 935 described later can be overwritten when the head sample of the track fragment run box 927 is a randomly accessible sample. Here, random access means, for example, a processing operation of moving the data playback position after 10 seconds in the middle of playback or starting playback from the middle of data in an MP4 file playback device. The randomly accessible sample is a frame that can be decoded independently without referring to the data of other frames in the MP4 file playback device among video samples, that is, an intra-frame encoded frame (so-called intra frame). Means sample. In addition, since any sample can be decoded independently, it can be said that all audio samples are randomly accessible samples.
[0035]
In the table 931, entries 932 indicating header information for each sample are accumulated for the number indicated in the sample count 928.
The entry 932 is a collection of fields indicating header information for each sample, and which field is included is indicated by the flag 907. In the field included in the entry 932, a sample duration 933 in which the playback time length of the sample is described, a sample size 934 in which the sample size is described, and flag information indicating whether the sample is randomly accessible are described. In order to handle samples using bi-directional prediction, there is a sample composition time offset 936 in which the difference value between the sample decoding time and the display time is described.
[0036]
If these fields are not included in the entry 932, the header information of each sample is stored in the track fragment header box 926 and the movie extend box (identifier “mvex”) in the movie fragment box 915. Since default values are described, these default values are used.
[0037]
In the track fragment run box 927, header information is described in order from the sample with the earliest decoding time. Therefore, when the apparatus that plays the MP4 file searches for the header information of the sample, the track ID in the track fragment header box 926 is referred to in order from the first track fragment box 925 in the file, thereby obtaining the track information to be acquired. The track fragment box 925 including the header information is searched, and within the track fragment box 925, the sample header information is searched in order from the first track fragment run box 927.
[0038]
Even in the case of the MP4 file 920 including the extension unit 921, information necessary for the entire track, such as initialization information at the time of decoding, is stored in the movie box 915.
Next, a configuration example of an MP4 file including the extension unit 921 having such a structure will be described.
[0039]
FIG. 24 is a diagram illustrating a configuration example of an MP4 file extension unit including a conventional extension unit.
In FIG. 24, the content storage method will be described with reference to two examples, and the playback time length of the content is assumed to be 60 seconds.
[0040]
The MP4 file 940 shown in FIG. 24A is configured to store media data in both the basic unit 941 and the extension unit 942. That is, media data of 0 to 30 seconds is stored in mdat_1 (reference numeral 945) of the basic unit 941, and media data of 30 to 45 seconds is stored in mdat_2 (reference numeral 947) of the extension section 942, and mdat_3 (reference numeral 949). ) Stores media data for 45 to 60 seconds. The header information of mdat_1 (reference 945) is stored in moov 944, the header information of mdat_2 (reference 947) is stored in moof_1 (reference 946), and the header information of mdat_3 (reference 949) is stored in moof_2 (reference 948). Has been.
[0041]
On the other hand, the MP4 file 950 shown in FIG. 24B is configured to store media data only in the extension unit 952. That is, the basic unit 951 includes ftyp 953 and moov 954 and does not include mdat. Media data from 0 to 30 seconds is stored in mdat_1 (reference numeral 956) of the extension section 952, and 30 to mdat_2 (reference numeral 958). Media data up to 60 seconds is stored. The header information of mdat_1 (reference numeral 956) is stored in moof_1 (reference numeral 955), and the header information of mdat_2 (reference numeral 958) is stored in moof_2 (reference numeral 957).
[0042]
Here, how the extension part of the MP4 file is created will be described with reference to FIGS.
FIG. 25 is a block diagram showing a configuration of a conventional multiplexing apparatus.
The multiplexing device 960 is a device that multiplexes media data and creates MP4 file extension data. Here, it is assumed that the extension data of the MP4 file is created by multiplexing the video data and the audio data.
[0043]
The first input unit 961 takes video data into the multiplexing device 960 and stores it in the first data storage unit 962, and the second input unit 964 takes audio data into the multiplexing device 960 and sends it to the second data storage unit 965. To accumulate.
The first analysis unit 963 reads the video data sample by sample from the first data storage unit 962 and analyzes it, and outputs the header information of the video sample to the packet unit determination unit 967. The second floor seat 966 reads audio data sample by sample from the second data storage unit 965 and analyzes it, and outputs the header information of the audio sample to the packet unit determination unit 967. The video sample header information and the audio sample header information include information indicating the sample size and the playback time length, and the video sample header information also includes information indicating whether the video sample is an intra frame. include.
[0044]
The packet unit determination unit 967 determines packet units of video data and audio data so that the number of samples included in the packet is constant, and creates header information of each packet based on the acquired sample header information.
FIG. 26 shows a processing operation flow of the conventional packet unit determination unit. Here, the number of samples stored in one packet is N, and this value is determined in advance and held in the memory of the multiplexing apparatus 960 or the like.
[0045]
First, when the first analysis unit 963 acquires one video sample (S901) and outputs the video sample header information to the packet unit determination unit 967, the packet unit determination unit 967 stores the video sample header information in the packet creation table. Add (S902).
Next, the packet unit determination unit 967 updates the number of video samples included in the packet (S903), and determines whether the number of video samples included in the packet has become N (S904).
[0046]
Here, when the number of video samples included in the packet is less than N (No in S904), the processes from S901 to S903 are repeated, and the number of video samples included in the packet becomes N (S904). In step S905, the packet unit determination unit 967 packetizes the N video samples and ends the processing operation.
[0047]
Similarly, the packet unit determination unit 967 converts audio samples into packets by performing the processing operations from S901 to S905. Then, the packet unit determination unit 967 repeats the processing operation of this flow until packetization of all samples is completed.
[0048]
FIG. 27 shows an example of a packet creation table for storing conventional video sample header information. In the packet creation table 968a, for each video sample, information on the size of the sample, the playback time length of the sample, and the intra-frame encoded frame flag indicating whether or not the video sample is an intra frame is described. Here, it is shown that the first video sample stored in the packet has a size of 300 bytes, a playback time length of 30 ms, and is not an intra-frame encoded frame. The second video sample is an intra-frame encoded frame. It is shown that. This packet creation table 968a is output to the packet creation table storage unit 968 when these pieces of information are sequentially added by the packet unit determination unit 967 and created up to the Nth sample as the last sample included in one packet. The
[0049]
Referring to FIG. 25 again, subsequently, the packet unit determination unit 967 describes the header information of N samples in the packet creation table 968a, and then outputs the packet creation table 968a to the packet creation table storage unit 968. The packet creation signal is output to the packet header creation unit 969.
[0050]
When the packet header creation unit 969 obtains the packet creation signal, the packet header creation unit 969 reads the packet sample header information from the packet creation table 968a held in the packet creation table storage unit 968, and creates moof data. In addition, the packet header creation unit 969 outputs the created moof data to the packet combining unit 971, and the actual data of the sample included in the packet is stored in the first data storage unit 962 and the second data storage unit 965. Mdat information including the pointer information indicating whether or not and the sample size information are output to the packet data creation unit 970.
[0051]
Based on the acquired mdat information, the packet data creation unit 970 reads sample substance data from the first data storage unit 962 and the second data storage unit 965 to create mdat data, and outputs the mdat data to the packet combining unit 971. To do.
[0052]
Then, the packet combining unit 971 combines the moof data and the mdat data, and outputs mp4 extension unit data for one packet.
Eventually, the output mp4 extension data for one packet is captured by the apparatus that creates the MP4 file, and the mp4 extension data that is sequentially created is arranged in order, so that the extension part of the MP4 file is Created. Thereafter, the MP4 file is created by combining the basic part and the extension part of the MP4 file with this file creation device.
[0053]
[Non-Patent Document 1]
ISO / IEC JTC1 / SC29 / WG11 MPEG, N4854 “Proposed Revised Common Text Multimedia File Format Specification”, March 21, 2002
[0054]
[Problems to be solved by the invention]
However, when reproducing the extended portion of the MP4 file multiplexed by such a conventional multiplexing device, there are the following problems.
As one of them, first, in the conventional multiplexing apparatus, multiplexing is performed without considering the reproduction start time of the sample included in the packet, so that, for example, synchronization with a video sample at a certain reproduction start time is achieved. Audio samples may be stored in different packets than video samples. Therefore, there is a problem that the efficiency of data access at the time of reproduction deteriorates on the reproduction apparatus side of the MP4 file.
[0055]
In addition, in the conventional multiplexing apparatus, multiplexing is performed based on the number of samples included in the packet. Therefore, the random accessible sample, that is, the video sample corresponding to the intra frame is stored in the packet. In many cases, it becomes mixed. Therefore, when searching for a randomly accessible sample on the MP4 file playback device side, all video samples included in the packet must be searched, and the amount of calculation required for searching the sample becomes enormous. There is also a problem.
[0056]
These problems will be described in more detail with reference to FIG.
FIG. 28 is a diagram for explaining problems of a conventional multiplexing apparatus.
FIG. 28A clarifies the first problem that the efficiency of data access during reproduction deteriorates.
[0057]
The header information of the sample included in each mdat is stored in the immediately preceding moof, and the header information regarding the video sample of the playback start time 20 s stored in mdat_1 is stored as the first sample in moof_1, and is stored in mdat_10. The stored header information related to the audio sample of the reproduction start time 20 s is stored as the final sample in moof_10.
[0058]
Therefore, if the MP4 file playback apparatus tries to play back the portion of the content playback time of 20 s, moof_10 from the acquisition of the video sample header information stored in moof_1 to the acquisition of the audio sample header information. The data access efficiency becomes worse.
[0059]
FIG. 28B clarifies the second problem that the amount of calculation required to search for a randomly accessible sample becomes enormous.
The header information related to the i-th randomly accessible video sample stored at the end of mdat_1 is stored as the last sample in moof_1, and is related to the i + 1th randomly accessible video sample stored at the end of mdat_3. The header information is stored as a final sample in moof_3.
[0060]
Therefore, if the MP4 file playback device tries to perform random access, it must search to the last sample of moof, and the amount of calculation required for the search becomes enormous.
Further, in addition to the first and second problems, the configuration of the MP4 file extension unit created by the conventional multiplexing device increases the number of seeks for obtaining sample data, so that the optical disk playback device There is also a problem that it is not suitable for random access reproduction in a device having a low seek speed such as the above.
[0061]
This problem will be described again with reference to FIG. When trying to randomly access the i-th randomly accessible video sample of moof_1, the playback apparatus first reads the pointer to the top position of moof_1 in order to obtain header information of the i-th randomly accessible video sample. , And the inside of moof_1 is analyzed in order. At this time, the first seek is required.
[0062]
After that, the playback apparatus obtains where in mdat_1 the actual data of the i-th randomly accessible video sample is stored, and moves the read pointer to the start position of the actual data. At this time, since the actual data of the i-th randomly accessible video sample is stored at the end of mdat_1, the actual data of the sample cannot be acquired by continuously moving the read pointer from the top position of moof_1. A second seek is required.
[0063]
That is, since the seek operation is performed when the read pointer is moved to the start position of moof_1 and the start position of the actual data, if the playback device is a device with a slow seek speed, it takes time for random access playback. End up. In particular, when the actual data such as an audio sample that is synchronized with the i-th randomly accessible video sample is stored separately from the actual data of the video sample such as a different packet, the seek operation is further performed. It becomes necessary, and it becomes difficult to perform random access reproduction quickly.
[0064]
Therefore, the present invention has been made in view of these problems, and media data is multiplexed so that a multiplexed file of media data is excellent in data access efficiency at the time of reproduction and the calculation amount required for sample search is reduced. An object of the present invention is to provide a multiplexing device that can multiplex.
[0065]
It is another object of the present invention to provide a multiplexing device capable of multiplexing media data so that the multiplexed file is suitable for random access reproduction in a device having a low seek speed.
[0066]
[Means for Solving the Problems]
In order to achieve the above object, a multiplexing apparatus according to the present invention creates a multiplexed data by packet multiplexing media data including image data and at least one of audio data and text data. The media data acquisition means for acquiring the media data, the media data acquired by the media data acquisition means is analyzed, and the minimum of the image data, audio data, and text data included in the media data is analyzed. Analyzing means for obtaining reproduction start time information indicating the reproduction start time of the sample for the sample that is an access unit, and the image data included in the media data based on the reproduction start time information obtained by the analyzing means, Align the playback start time of each sample of audio data and text data A packet unit determining unit for determining a unit for packetizing the media data; a packet header unit generating unit for generating a packet header unit for storing a header of the media data in a packetization unit determined by the packet unit determining unit; A packet data part creating means for creating a packet data part for storing the actual data of the media data in a packetization unit determined by the packet unit determining means; a packet header part created by the packet header part creating means; and the packet Packetizing means for creating a packet by combining the packet data part created by the data part creating means The packet unit determining means determines the unit by aligning the reproduction start times of the samples of the image data, audio data, and text data for all the packets necessary for storing the media data. It is characterized by that.
[0067]
As a result, the playback start times of the image data, audio data, and text data included in the media data are aligned and stored in the packet, so that the playback device can improve the efficiency of data access during playback. Can do.
[0068]
In the multiplexing apparatus according to the present invention, the image data is moving image data, and the analyzing unit further analyzes the moving image data acquired by the media data acquiring unit, and the moving image data is displayed on a screen. When at least one sample including intra frame information indicating that it is an intra-coded sample is included, the intra frame information is acquired, and the packet unit determining means is configured to acquire the intra frame information by the analyzing means. When acquired, the unit for packetizing the media data is determined based on the intra frame information and the playback start time information, and the sample of the moving image data including the intra frame information is determined as the packetization unit. It is preferable to arrange at the head.
[0069]
As a result, since the first video sample included in the packet becomes an intra-frame video sample, the amount of calculation required for searching for the sample at the time of random access on the playback device side can be greatly reduced.
Furthermore, in the multiplexing device according to the present invention, the packet data section creation means stores the media data samples included in the packetization unit in an interleaved manner so that the playback start times of the samples are in ascending order. More preferably, the packet data part is created.
[0070]
As a result, the video sample and audio sample are stored in mdat in ascending order of playback start time, so the number of seek operations during random access on the playback device can be reduced, and playback with a slow seek speed is possible. Even a device can realize quick random access reproduction.
[0071]
The present invention can be realized not only as such a multiplexing apparatus, but also as a multiplexing method using steps characteristic of the multiplexing apparatus as a step, or by performing these steps as a computer. It can also be realized as a program to be executed. Needless to say, such a program can be distributed via a recording medium such as a CD-ROM or a transmission medium such as the Internet.
[0072]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that MPEG-4 Visual encoded data is used as video data in the present embodiment, and MPEG-4 Audio encoded data is used as audio data in the present embodiment. In the present embodiment, an apparatus for multiplexing video data and audio data will be mainly described. However, it is not intended to exclude multiplexing of other media data such as text data.
[0073]
(Embodiment 1)
First, a multiplexing apparatus according to Embodiment 1 of the present invention will be described with reference to FIGS.
FIG. 1 is a block diagram showing a functional configuration of a multiplexing apparatus according to Embodiment 1 of the present invention.
The multiplexing device 100 is a device that multiplexes video data and audio data to create MP4 file extension data, and includes a first input unit 101, a first data storage unit 102, a first analysis unit 103, a second analysis unit 103, and a second analysis unit 103. An input unit 104, a second data storage unit 105, a second data analysis unit 106, a packet unit determination unit 107, a packet creation table storage unit 111, a packet header creation unit 112, a packet data creation unit 113, and a packet combination unit 114 are provided.
[0074]
The first input unit 101 is an interface for fetching encoded video data from the image encoding device or the like into the multiplexing device 100, and causes the first data storage unit 102 to sequentially store the acquired video input data.
The first data storage unit 102 is a cache memory or RAM (Random Access Memory) that temporarily holds video input data.
[0075]
The first analysis unit 103 reads out and analyzes video sample data that is data for one video sample from the video input data held in the first data storage unit 102, and outputs header information of the video sample Which is realized by a CPU and a memory. Note that the video sample header information output from the first analysis unit 103 includes information indicating the size of the video sample, the playback time length, and whether it is an intra frame. Further, in the case of a sample using bi-directional prediction, the video sample header information includes difference information between the decoding time and the display time.
[0076]
The second input unit 104 is an interface for importing encoded audio data from the speech encoding device or the like into the multiplexing device 100, and causes the second data storage unit 105 to sequentially store the acquired audio input data.
The second data storage unit 105 is a cache memory, RAM, or the like that temporarily holds audio input data.
[0077]
The second analysis unit 106 reads and analyzes audio sample data, which is data for one audio sample among the audio input data held in the second data storage unit 105, and outputs header information of the audio sample Which is realized by a CPU and a memory. Note that the audio sample header information output from the second analysis unit 106 includes information indicating the size and playback time length of the audio sample.
[0078]
The packet unit determination unit 107 accumulates the header information of the video sample and the audio sample included in the packet so that the playback start time of the video sample and the playback start time of the audio sample included in the packet are aligned. A processing unit that determines a packet unit of audio data, and is realized by a CPU and a memory. The packet unit determination unit 107 outputs a collection of sample header information for the determined packet units to the packet generation table storage unit 111 as a packet generation table, and creates a packet for instructing the creation of a packet header after the packet unit is determined The signal is output to the packet header creation unit 112. The packet unit determining unit 107 includes a time adjusting unit 108 that adjusts the packet unit in time units, a video packet unit determining unit 109 that determines a packet unit of video data, and an audio packet that determines a packet unit of audio data. A unit determining unit 110.
[0079]
The time adjustment unit 108 is a processing unit that adjusts the end time of a packet so that the packet is within a predetermined time unit. First, the time adjustment unit 108 outputs a predetermined time (target time) to the video packet unit determination unit 109. The target time may be designated by the user. In this case, the multiplexing apparatus 100 acquires the target time specification via an input device such as a keyboard, and outputs a target time input signal indicating the target time specified from the input device to the time adjustment unit 108. Become.
[0080]
The video packet unit determination unit 109 is a processing unit that acquires video sample header information from the first analysis unit 103 and determines a packet unit of video data.
The video packet unit determination unit 109 acquires the target time from the time adjustment unit 108 and the video sample header information from the first analysis unit 103 so that each video is stored in a packet within the target time. While counting the playback time length of each video sample included in the sample header information, the header information of the last video sample included in the packet is sequentially added to the video packet creation table. When the video packet unit determination unit 109 adds the header information of the last video sample included in the packet to the video packet creation table, the playback start time of the first video sample included in the packet and the video sample included in the packet Video sample playback time information indicating the total playback time length is output to the audio packet unit determination unit 110.
[0081]
The audio packet unit determination unit 110 is a processing unit that acquires the audio sample header information acquired from the second analysis unit 106 and determines the packet unit of audio data.
The audio packet unit determination unit 110 acquires video sample playback time information from the video packet unit determination unit 109 and audio sample header information from the second analysis unit 106, and is included in the head of the packet. The audio sample included in the packet is arranged while arranging the audio sample of the playback start time that is the same as or close to the playback start time of the first video sample, and counting the playback time length of each audio sample included in each audio sample header information The last audio sample included in the packet is arranged so that the total sum of the playback time lengths of the packets is the same as or approximate to the total playback time length of the video samples included in the packets.
[0082]
Here, the audio sample with the reproduction start time approximate to the video sample reproduction start time is the audio sample with the earliest reproduction start time or the video sample reproduction start after the video sample reproduction start time. It means an audio sample with the latest playback start time before the time.
[0083]
Thereafter, the audio packet unit determination unit 110 sequentially adds audio sample header information from the first audio sample to the last audio sample included in the packet to the audio packet creation table.
The packet creation table storage unit 111 is a cache memory, a RAM, or the like that temporarily holds the video packet creation table and the audio packet creation table output from the packet unit determination unit 107.
[0084]
The packet header creation unit 112 is a processing unit that creates a packet header part (moof) in which packet header information is stored, and is realized by a CPU or a memory.
When the packet header creation unit 112 acquires the packet creation signal from the packet unit determination unit 107, the packet header creation unit 112 reads the packet sample header information by referring to the packet creation table from the packet creation table storage unit 111, creates moof data, Output to the unit 114.
[0085]
The packet header creation unit 112 also includes pointer information indicating where in the first data storage unit 102 and the second data storage unit 105 the actual data of the video sample and the audio sample included in the packet is stored, Sample size information indicating the size and mdat information including a signal instructing creation of the packet data portion (mdat) are output to the packet data creation portion 113.
[0086]
Note that the packet header creation unit 112 was encoded by a coding method such as AMR (Advanced Multi Rate CODEC) in which coding rate switching occurs in the middle of data when creating a moof. For media data, header information can also be stored in different trafs according to the encoding rate.
[0087]
The packet data creation unit 113 is a processing unit that creates a packet data part (mdat) in which packet actual data is stored, and is realized by a CPU or a memory.
When the packet data creation unit 113 acquires the mdat information from the packet header creation unit 112, the packet data creation unit 113 receives the video sample contained in the packet from the first data storage unit 102 based on the pointer information and the sample size information contained in the mdat information. The video substance data is read, the audio substance data of the audio sample included in the packet is read from the second data storage unit 105 to generate mdat data, and is output to the packet combining unit 114.
[0088]
The packet combining unit 114 is a processing unit that combines the moof data and the mdat data to create mp4 extension data for one packet, and is realized by a CPU or a memory. The packet combining unit 114 acquires moof data from the packet header generation unit 112, acquires mdat data from the packet data generation unit 113, combines the moof data and mdat data, and mp4 expansion unit data for one packet. Are generated, and the sequentially created mp4 extension data is output to an apparatus for creating an MP4 file.
[0089]
With reference to FIG. 2, a description will be given of a processing procedure for creating an MP4 file extension in the multiplexing apparatus 100 configured as described above.
FIG. 2 is a flowchart showing the processing operation of the multiplexing apparatus 100.
First, when the first input unit 101 and the second input unit 104 capture video data and audio data, respectively, in the multiplexing apparatus 100 (S100), the first input unit 101 stores the video input data in the first data storage unit 102. The second input unit 104 causes the second data storage unit 105 to store the audio input data.
[0090]
Next, the first analysis unit 103 reads and analyzes the video sample data from the first data storage unit 102 and outputs the video sample header information to the video packet unit determination unit 109 of the packet unit determination unit 107. Then, the video packet unit determination unit 109 determines a packet unit of video data based on the video sample header information acquired from the first analysis unit 103 and the target time acquired from the time adjustment unit 108 (S110). The processing operation in which the video packet unit determination unit 109 determines the video data packet unit will be described in detail later.
[0091]
Thereafter, the video packet unit determination unit 109 outputs the reproduction time information of the video sample included in the packet whose packet unit is determined to the audio packet unit determination unit 110 (S120).
Then, the audio packet unit determination unit 110 determines the packet unit of the audio data based on the reproduction time information of the video sample acquired from the video packet unit determination unit 109 (S130). At this time, the audio packet unit determination unit 110 sets the packet unit so that the reproduction start time of the first audio sample included in the packet is the same as or close to the reproduction start time of the first video sample included in the packet. decide.
[0092]
When the audio packet unit determination unit 110 determines the packet unit of the audio data, the packet unit determination unit 107 outputs the packet creation table to the packet creation table storage unit 111 and outputs the packet creation signal to the packet header creation unit 112.
[0093]
Thereafter, the packet header creation unit 112 creates the moof data in the determined unit and outputs it to the packet combining unit 114, and the packet data creation unit 113 creates the mdat data in the determined unit and combines the packets. The packet combining unit 114 combines the moof data and the mdat data to create one packet in the determined unit (S140), and outputs it as mp4 extension data for one packet.
[0094]
When the creation of one packet is completed, the multiplexing apparatus 100 determines whether there is still input data from the first input unit 101 and the second input unit 104 (S150). Here, when there is input data (No in S150), the multiplexing apparatus 100 sets the data held in the buffer memory, that is, the first data storage unit 102, the second data storage unit 105, and the packet creation table storage unit 111. Among them, data that has already been packetized is cleared (S160), and the processing operations from S110 to S150 are repeated.
[0095]
On the other hand, if there is no input data (Yes in S150), the multiplexing apparatus 100 ends the MP4 file extension processing.
As described above, the multiplexing apparatus 100 first determines the packet unit of the video data, then determines the packet unit of the audio data, and multiplexes the media data, thereby creating the extension portion of the MP4 file.
[0096]
Here, the processing operation in which the video packet unit determination unit 109 determines the packet unit of video data in step S110 of FIG. 2 will be described in detail.
FIG. 3 is a flowchart showing the processing operation of the video packet unit determination unit 109. Prior to this flow, the video packet unit determination unit 109 acquires the target time from the time adjustment unit 108.
[0097]
When the video packet unit determination unit 109 acquires the video sample header information from the first analysis unit 103 (S111), the video packet unit determination unit 109 adds the video sample header information to the video packet creation table (S112).
At this time, the video packet unit determination unit 109 determines whether the total playback time length of the video samples included in the video sample header information, that is, the total playback time of the video data included in the packet has reached the previously acquired target time, Alternatively, it is determined whether or not the target time has been exceeded (S113).
[0098]
When the total reproduction time of the video data included in the packet has not reached the target time (No in S113), the video packet unit determination unit 109 acquires the next video sample header information (S111), and performs the processing in S112 and S113. Repeat the operation.
[0099]
When the total playback time of the video data included in the packet has reached the target time (Yes in S113), the video packet unit determination unit 109 determines the video sample indicated by the video sample header information added last to the video packet creation table, The last video sample included in the packet is determined (S114), and the processing operation for determining the packet unit is terminated.
[0100]
Next, the processing operation in which the audio packet unit determination unit 110 determines the audio data packet unit in step S130 of FIG. 2 will be described in detail.
FIG. 4 is a flowchart showing the processing operation of the audio packet unit determination unit 110.
[0101]
Prior to this flow, the audio packet unit determination unit 110 acquires video sample playback time information from the video packet unit determination unit 109.
When the audio packet unit determination unit 110 acquires the audio sample header information from the second analysis unit 106 (S131), the audio packet unit determination unit 110 refers to the previously acquired video sample playback time information (S132), The reproduction start time of the video sample is read, and an audio sample having a reproduction start time that is the same as or close to the reproduction start time of the first video sample included in the packet is determined as the audio first sample of the packet (S133).
[0102]
When the audio packet unit determination unit 110 determines the audio head sample included in the packet, the audio packet unit determination unit 110 sequentially acquires the audio sample header information (S134), and adds the audio sample header information to the audio packet creation table (S135).
[0103]
Thereafter, the audio packet unit determination unit 110 refers to the video sample playback time information, reads the sum of the playback time lengths of the video samples included in the packet (S136), and sums the playback time lengths of the audio samples included in the packet. Determines the last audio sample included in the packet so that it becomes equal to or close to the sum of the playback time lengths of the video samples included in the packet (S137), and ends the processing operation for determining the packet unit To do.
[0104]
The MP4 file extension created through such processing by the multiplexing device 100 is excellent in data access efficiency on the playback device side. The reason will be described with reference to FIG. 5 showing an example of the data structure of the MP4 file extension created by the multiplexing apparatus 100.
[0105]
The MP4 file extension unit 200 shown in FIG. 5A is composed of a plurality of packets and is coupled to the basic part of the MP4 file.
Each packet constituting the MP4 file extension unit 200 is composed of a packet header part moof and a packet data part mdat. Here, the packet_1 means the first packet of the MP4 file extension unit 200, the moof included in the packet_1 is indicated as moof_1, and the mdat included in the packet_1 is indicated as mdat_1. Also, “V” shown in each mdat in FIG. 5A indicates that it is a video sample, and “A” shown in each mdat in FIG. 5A is an audio sample. (Hereinafter, the same applies to other drawings).
[0106]
In mdat_1 of the MP4 file extension unit 200, a video sample with a playback start time of 20 seconds is stored as a video head sample, and an audio sample with a playback start time of 20 seconds is stored as an audio head sample. In mdat_2, a video sample with a playback start time of 30 seconds is stored as a video head sample, and an audio sample with a playback start time of 30 seconds is also stored as an audio head sample.
[0107]
In this way, by storing video samples and audio samples in one packet with their respective playback start times aligned, the amount of calculation required for data access when the MP4 file extension unit 200 is played back on the playback device side. Can be greatly reduced.
[0108]
Further, since the reproduction start times of the respective media data are aligned and stored in the packet, it is possible to divide the data by an arbitrary number of packets and adjust the size of the MP4 file data to a desired size.
Here, the MP4 file extension created by the multiplexing apparatus 100 may have the data structure shown in FIG.
[0109]
FIG. 5B is a diagram illustrating a second example of the data structure of the MP4 file extension created by the multiplexing apparatus 100.
In the mdat_1 of the MP4 file extension unit 210 shown in FIG. 5B, a video sample with a playback start time of 20 seconds is stored as a video head sample, and an audio sample with a playback start time of 20 seconds is stored in mdat_2. Stored as the first audio sample. In mdat_3, a video sample with a playback start time of 30 seconds is stored as a video head sample, and in mdat_4, an audio sample with a playback start time of 30 seconds is stored as an audio head sample.
[0110]
In this way, by storing either video or audio data in one packet, and alternately arranging a packet for storing video data and a packet for storing audio data with the same reproduction start time. However, when the MP4 file expansion unit 200 is played back on the playback device side, the amount of calculation required for data access can be greatly reduced.
[0111]
As described above, according to multiplexing apparatus 100 according to the first embodiment, each media data is packetized by aligning the playback start time of each media data, so that the efficiency of data access on the playback device side is increased. Can be achieved.
[0112]
(Embodiment 2)
Next, a multiplexing apparatus according to Embodiment 2 of the present invention will be described with reference to FIGS.
The multiplexing apparatus according to the second embodiment is common in the main components to the multiplexing apparatus 100 according to the first embodiment, but has a characteristic configuration in the packet unit determination unit. 3 differs from the multiplexing apparatus 100 according to the first embodiment. Hereinafter, this difference will be mainly described. In addition, about the same component as the said Embodiment 1, the same code | symbol shall be used and description is abbreviate | omitted.
[0113]
FIG. 6 is a block diagram showing a functional configuration of the packet unit determination unit of the multiplexing apparatus according to the second embodiment.
The packet unit determination unit 117 accumulates the header information of the video sample and audio sample included in the packet so that the respective reproduction start times are aligned, and the first video sample included in the packet becomes an intra frame. As described above, the processing unit determines a packet unit of video data and audio data, and includes a time adjustment unit 108, a video packet unit determination unit 119, and an audio packet unit determination unit 110.
[0114]
The video packet unit determination unit 119 is a processing unit that acquires video sample header information from the first analysis unit 103 and determines a packet unit of video data based on either time or an intra frame. Unit 120 and an I-frame reference unit adjustment unit 121.
[0115]
The time reference unit adjustment unit 120 is a processing unit that adjusts the packet unit of video data based on the target time output from the time adjustment unit 108, counts the reproduction time length of each video sample header information, The packet unit is adjusted to be a predetermined time unit.
[0116]
The I frame reference unit adjustment unit 121 adjusts the packet unit of video data based on whether or not the video sample header information output from the first analysis unit 103 includes information indicating an intra frame. If the video sample header information that contains information indicating that it is an intra frame is acquired, the packet unit is switched by the video sample of the intra frame, and the video sample of the next packet is the video sample of the intra frame. The packet unit is adjusted so that
[0117]
The processing operation in which the video packet unit determining unit 119 determines the packet unit of video data in the multiplexing apparatus according to the second embodiment having the packet unit determining unit 117 configured as described above will be described in detail.
FIG. 7 is a flowchart showing the processing operation of the video packet unit determination unit 119.
[0118]
Prior to this flow, the video packet unit determination unit 119 acquires the target time from the time adjustment unit 108 and stores the target time in the time reference unit adjustment unit 120.
As in the first embodiment, when the video packet unit determination unit 119 acquires the video sample header information from the first analysis unit 103 (S201), the video packet unit determination unit 119 adds the video sample header information to the video packet creation table (S202). ).
[0119]
At this time, the video packet unit determination unit 119 determines in the I frame reference unit adjustment unit 121 whether the acquired video sample header information includes information indicating an intra frame (S203).
When information indicating an intra frame is included (Yes in S203), the video packet unit determination unit 119 causes the time reference unit adjustment unit 120 to calculate the total playback time of all video samples included in the packet. It is determined whether or not the acquired target time is exceeded (S205).
[0120]
Here, when information indicating an intra frame is not included (No in S203) or when the target time is not exceeded (No in S205), the video packet unit determination unit 119 includes the time reference unit adjustment unit 120. In step S204, the sum of the playback time lengths of the video samples included in the packet is updated by adding the playback time length of the video samples included in the video sample header information (S204), and the next video sample header information is acquired. (S201) The above processing operation is repeated.
[0121]
On the other hand, when the target time is exceeded (Yes in S205), the video packet unit determination unit 119 determines that the last video sample included in the packet is an intra frame in the I frame reference unit adjustment unit 121. The video sample immediately before the sample is determined (S206), and the video data packet unit determination processing operation is terminated.
[0122]
In the extension part of the MP4 file created through the processing operation of the video packet unit determination unit 119, the video sample stored at the head of the packet is always an intra-frame video sample. Sometimes, playback can be started from the first video sample of the packet, and the amount of calculation required to search for a randomly accessible video sample can be greatly reduced.
[0123]
Also, since the video sample stored at the head of the packet is always an intra-frame video sample, the packet header portion (moof) has a trun head sample flag located at the head of the traf storing the video track header information. It is only necessary to describe information indicating that random access is possible only in the field. Since the sample flag field of each trun can be omitted by using the default value, the load at the time of creating the moof data is reduced. The file size of the entire MP4 file can also be reduced.
[0124]
According to this processing operation, when the interval between intra frames included in the video data is increased, the reproduction time length per packet may be increased. Therefore, the packet unit determination unit 117 may perform processing operations as described below.
[0125]
FIG. 8 is a flowchart showing the second processing operation of the video packet unit determination unit 119.
Similar to the first processing operation, prior to this flow, the video packet unit determination unit 119 acquires the target time from the time adjustment unit 108 and holds it in the time reference unit adjustment unit 120.
[0126]
When the video packet unit determination unit 119 acquires the video sample header information from the first analysis unit 103 (S211), the video packet unit determination unit 119 adds the video sample header information to the video packet creation table (S212).
At this time, the video packet unit determination unit 119 determines in the time reference unit adjustment unit 120 whether or not the total playback time of all video samples included in the packet exceeds the previously acquired target time (S213). .
[0127]
When the target time is exceeded (Yes in S213), the video packet unit determination unit 119 indicates the video indicated by the video sample header information immediately before the video sample header information acquired this time as the last video sample included in the packet. The sample is determined (S214), and the processing operation for determining the video data packet unit is terminated.
[0128]
On the other hand, when the target time has not been exceeded (No in S213), the video packet unit determination unit 119 includes information indicating that it is an intra frame in the acquired video sample header information in the I frame reference unit adjustment unit 121. It is determined whether or not (S215).
[0129]
Here, when information indicating an intra frame is included (Yes in S215), the video packet unit determination unit 119 determines the last video sample included in the packet in the I frame reference unit adjustment unit 121. The video sample immediately before the video sample determined to be a frame is determined (S214), and the processing operation for determining the video data packet unit is terminated.
[0130]
On the other hand, when information indicating an intra frame is not included (No in S215), the video packet unit determination unit 119 causes the time reference unit adjustment unit 120 to reproduce the playback time of the video sample included in the video sample header information. By adding the length, the sum of the playback time lengths of the video samples included in the packet is updated (S216), the next video sample header information is acquired (S211), and the above processing operation is repeated.
[0131]
The extension unit of the MP4 file created through the second processing operation of the video packet unit determination unit 119 creates a packet by setting a predetermined time limit and keeps the packet size below a desired size. If there is an intra-frame video sample, it can be stored at the beginning of the packet, so the playback device determines whether or not only the top video sample of the packet is randomly accessible during random access. This can reduce the amount of calculation required to search for a randomly accessible video sample.
[0132]
When the video packet unit determination unit 119 completes the video data packet unit determination processing operation, the video packet unit determination unit 119 outputs the video sample playback time information to the audio packet unit determination unit 110, and determines the audio data packet unit in the audio packet unit 110. The processing operation is performed in the same manner as in the first embodiment.
[0133]
The MP4 file extension unit created through such processing operations by the packet unit determination unit 117 reduces the search load during random access on the playback device side. The reason will be described with reference to FIG. 9 showing an example of the data structure of the MP4 file extension created by the multiplexing apparatus according to the second embodiment.
[0134]
In the mdat_1 of the MP4 file extension unit 220 shown in FIG. 9A, an intra-frame video sample is stored as a video head sample, and an intra-frame video sample is also stored in mdat_2 as a video head sample. .
[0135]
In this way, by storing the intra-frame video sample in the packet as the first video sample, only the first video sample of the packet is acquired in order to obtain a randomly accessible video sample at the time of random access on the playback device side. Since searching is sufficient, there is no need to search all video samples included in the packet, and the sample search load during random access can be greatly reduced.
[0136]
At this time, also in the MOOF_1 and MOOF_2 of the MP4 file extension unit 220, information indicating that random access is possible only in the leading sample flag field of trun located at the beginning of the traf storing the header information of the video track. By describing, the size of moof_1 and moof_2 can also be reduced.
[0137]
Here, the MP4 file extension created by the multiplexing apparatus according to the second embodiment may have the data structure shown in FIG.
In the mdat_1 of the MP4 file extension unit 230 shown in FIG. 9B, the intra-frame video sample is stored as the video head sample, and the intra-frame video sample is also stored in the mdat_3 as the video head sample. . Audio samples are stored in mdat_2 and mdat_4.
[0138]
As described above, the playback device can also be configured by storing either video or audio data in one packet and storing the video sample of the intra frame as the first video sample in the packet storing the video data. The sample search load during random access can be greatly reduced.
[0139]
Note that in any of these data structure examples of the MP4 file extension unit, the data on the playback device side is obtained by aligning the playback start time of the first video sample stored in the packet with the playback start time of the first audio sample. The amount of computation required for access can be greatly reduced.
[0140]
As described above, according to the multiplexing apparatus according to the second embodiment, since a packet is created using a randomly accessible video sample as the first video sample, it is necessary for the playback apparatus to search for a sample during random access. The amount of calculation can be reduced.
[0141]
(Embodiment 3)
Furthermore, a multiplexing apparatus according to Embodiment 3 of the present invention will be described with reference to FIGS.
The multiplexing apparatus according to the third embodiment is common to the multiplexing apparatuses according to the first and second embodiments in the main components, but has a characteristic configuration in the packet data creation unit. In this respect, the multiplexing apparatus according to the first and second embodiments is different. Hereinafter, this difference will be mainly described. In addition, about the same component as the said Embodiment 1 and 2, the same code | symbol shall be used and description is abbreviate | omitted.
[0142]
FIG. 10 is a block diagram showing a functional configuration of the packet data creation unit of the multiplexing apparatus according to the third embodiment.
The packet data creation unit 130 is a processing unit that creates a packet data portion (mdat) by interleaving and storing video sample entity data and audio sample entity data. The mdat information acquisition unit 131, A video entity data reading unit 132, an audio entity data reading unit 133, and an interleave arrangement unit 134 are provided.
[0143]
The mdat information acquisition unit 131 is a processing unit that acquires the mdat information from the packet header creation unit 112 and outputs an instruction to read the actual data and reproduction time information to the other units constituting the packet data creation unit 130.
When the mdat information acquisition unit 131 acquires the mdat information from the packet header creation unit 112, the mdat information analysis unit 131 analyzes the mdat information and acquires reproduction time information indicating the reproduction start time and the reproduction end time of the video sample and the audio sample. Based on the reproduction time information, all video samples and audio samples included in the packet are rearranged so that the reproduction start time is in ascending order.
[0144]
Then, the mdat information acquisition unit 131 outputs a video reading instruction for instructing the video entity data reading unit 132 to read out the actual data of the video sample in order from the sample with the lowest playback start time according to the rearranged order, or the audio An audio read instruction for instructing the substance data reading unit 133 to read the substance data of the audio sample is output. This video read instruction includes pointer information indicating where the actual data of the video sample is stored in the first data storage unit 102 and video sample size information. The audio read instruction includes the audio sample. Pointer information indicating where in the second data storage unit 105 the actual data is stored, and audio sample size information.
[0145]
The video entity data reading unit 132 is a processing unit that acquires a video reading instruction from the mdat information acquisition unit 131 and reads the video entity data from the first data storage unit 102. The video entity data reading unit 132 reads the video entity data from the first data storage unit 102 with reference to the pointer information and the size information included in the video reading instruction, and stores the read video entity data in the interleave arrangement unit 134. Output.
[0146]
The audio entity data reading unit 133 is a processing unit that acquires an audio reading instruction from the mdat information acquisition unit 131 and reads audio entity data from the second data storage unit 105. The audio entity data reading unit 133 reads the audio entity data from the second data storage unit 105 with reference to the pointer information and the size information included in the audio reading instruction, and the read audio entity data to the interleave arrangement unit 134. Output.
[0147]
The interleaving array unit 134 sequentially acquires the read video data and the read audio data output from the video substance data reading unit 132 and the audio substance data reading unit 133 in the order of output, and creates mdat data by arranging them in an interleaved manner. And a processing unit that outputs to the packet combining unit 114.
[0148]
In the multiplexing apparatus according to the third embodiment provided with the packet data creation unit 130 configured as described above, a processing operation in which the packet data creation unit 130 creates mdat will be described in detail.
FIG. 11 is a flowchart showing the processing operation of the packet data creation unit 130.
[0149]
First, the packet data creation unit 130 obtains mdat information from the packet header creation unit 112 in the mdat information acquisition unit 131 (S301). The mdat information acquisition unit 131 analyzes the acquired mdat information and extracts sample pointer information, size information, and reproduction time information. Then, the mdat information acquisition unit 131 rearranges all the video samples and audio samples included in the packet so that the playback start times are in ascending order based on the extracted playback time information of the samples. Subsequently, the mdat information acquisition unit 131 outputs a video reading instruction including pointer information and size information of the extracted video sample to the video entity data reading unit 132 in order from the sample with the smallest playback start time according to the rearranged order. Alternatively, an audio reading instruction including pointer information and size information of the extracted audio sample is output to the audio entity data reading unit 133.
[0150]
When acquiring the video read instruction, the video entity data reading unit 132 reads the video entity data from the first data storage unit 102 with reference to the pointer information and the size information, and outputs the video entity data to the interleave array unit 134. When acquiring the audio reading instruction, the unit 133 reads the audio entity data from the second data storage unit 105 with reference to the pointer information and the size information, and outputs it to the interleave arrangement unit 134 (S302).
[0151]
When the interleaving arrangement unit 134 receives the read entity data from the video entity data reading unit 132 and the audio entity data reading unit 133, the interleaving arrangement unit 134 sequentially arranges the received entity data in the order received (S303).
Here, the interleave arrangement unit 134 continues the arrangement of the entity data until all the arrangement of the video entity data and the audio entity data, that is, all of the entity data stored in one packet is completed (No in S304). S303).
[0152]
When all arrangements of entity data stored in one packet are completed (Yes in S304), the interleave arrangement unit 134 outputs the arranged entity data as mdat data to the packet combining unit 114 (S305). The processing for creating mdat is terminated.
[0153]
The MP4 file extension unit created through the processing operation of the packet data creation unit 130 is suitable for random access reproduction in an optical disk device or the like that takes time to seek. The reason for this will be described with reference to FIG. 12 showing an outline of the data structure of the MP4 file extension created by the multiplexing apparatus according to the third embodiment.
[0154]
The MP4 file extension unit 240 shown in FIG. 12 stores packet 1 for storing content data for 4 to 8 seconds, packet 2 for storing content data for 8 to 12 seconds, and content data for 12 to 16 seconds. A packet 3 is configured by arranging a plurality of packets.
[0155]
Each packet is composed of a moof 241 and an mdat 242. The moof 241 includes tfhd (V) and traf (V-1, V-2) relating to a video track, and tfhd (A) and traf (A-) relating to an audio track. 1, A-2) are stored. The sample entity data indicated by the header information stored in traf (V-1) and traf (A-1) is stored in mdat_1, and stored in traf (V-2) and traf (A-2). Sample entity data indicated by the header information is stored in mdat_2. In the mdat 242, the actual data of the video sample and the actual data of the audio sample are alternately interleaved and stored.
[0156]
At this time, if the read pointer is moved to the top position of moof_1 in the random access process in which the playback starts from the position where the playback time is 4 seconds on the playback device side, the moof_1 is analyzed thereafter, and the read pointer is continuously set. Therefore, the entity data necessary for reproduction can be acquired from mdat_1 that is continuous with moof_1.
[0157]
That is, according to the MP4 file extension unit 240, the playback device can realize random access playback by only one seek operation for moving the read pointer to the top position of moof_1, so it takes time to seek. It can be said that it is effective for optical disk devices.
[0158]
Here, in mdat242, the audio sample entity data stored immediately after the video sample entity data is aligned with the playback start time of the immediately preceding video sample, so that synchronized playback of video data and audio data is guaranteed. ing. FIG. 13 shows a state in which entity data is stored in mdat_1 of the MP4 file extension unit 240.
[0159]
As shown in FIG. 13, the playback start time of the video sample 1 stored at the head of mdat_1 is 4000 ms, and the playback start time of the audio sample 1 stored immediately after the video sample 1 is 4000 ms. The playback start times of the video sample 1 and the audio sample 1 are the same.
[0160]
Usually, the sample rate of the video sample and the audio sample is often different, and here, the playback time length of the video sample is 500 ms, and the playback time length of the audio sample is 100 ms.
Therefore, in mdat_1 of the MP4 file extension unit 240, audio samples 1 to 5 are interleaved and stored immediately after the video sample 1, and thereafter, the video sample 2, the audio samples 6 to 10, the video sample 3,. They are stored in order.
[0161]
At this time, the playback start time of the video sample 2 is 4500 ms, the playback start time of the audio sample 6 stored immediately after the video sample 2 is also 4500 ms, and the playback of the video sample and the audio sample immediately after the video sample is played back. The start times are always set to be the same.
[0162]
Further, since the sample rates of the video sample and the audio sample are different, the playback start time of the video sample and the playback start time of the audio sample immediately after that may not be the same. Even in such a case, synchronized playback of video data and audio data can be ensured by using an audio sample immediately after the video sample as an audio sample having a playback start time approximate to the playback start time of the video sample.
[0163]
FIG. 14 is a diagram illustrating a second data structure showing a state in which entity data is stored in mdat_1 of the MP4 file extension unit.
As shown in FIG. 14, the playback start time of the video sample 1 stored at the head of mdat_1 of the MP4 file extension unit 250 is 4000 ms, and the playback start of the audio sample 1 stored immediately after the video sample 1 is started. The time is 4050 ms, and the audio sample 1 having the earliest reproduction start time after the reproduction start time of the video sample 1 is stored as an audio sample stored immediately after the video sample 1.
[0164]
Here, similarly to the case described above, the playback time length of the video sample is 500 ms, and the playback time length of the audio sample is 100 ms.
Therefore, the audio samples 1 to 5 are interleaved and stored immediately after the video sample 1 in the mdat_1 of the MP4 file extension unit 250, and then the video sample 2, the audio samples 6 to 10, the video sample 3. Are stored in the order of.
[0165]
At this time, the playback start time of the video sample 2 is 4500 ms, the playback start time of the audio sample 6 stored immediately after the video sample 2 is 4550 ms, and the video sample and the audio sample immediately after the video sample are The playback start times are always aligned so as to approximate each other.
[0166]
Here, as an audio sample stored immediately after the video sample, an audio sample having the latest playback start time before the video sample playback start time may be stored. In this case, the audio sample 1 stored immediately after the video sample 1 has a playback time of 3950 ms.
[0167]
As described above, according to the multiplexing apparatus of the third embodiment, an audio sample having a playback start time that is the same as or close to the playback start time of the video sample is arranged immediately after the video sample, and the video sample And the audio sample are interleaved so that the playback start time is in ascending order and stored in mdat. Therefore, even in a playback device with a low seek speed, it is possible to create an MP4 file extension portion having a data structure that can be quickly accessed at random. it can.
[0168]
(Embodiment 4)
Subsequently, a demultiplexing apparatus according to Embodiment 4 of the present invention will be described with reference to FIG. 15 and FIG.
FIG. 15 is a block diagram showing a functional configuration of the demultiplexing apparatus according to the fourth embodiment.
The demultiplexing apparatus 300 acquires and analyzes MP4 file data including the MP4 file extension created by the multiplexing apparatus according to the first, second, and third embodiments, demultiplexes the media data, and plays back the data. A file input unit 301, a file data storage unit 302, a header separation analysis unit 303, a moov analysis unit 304, a moof analysis unit 305, a traf analysis unit 306, a run analysis unit 307, an RA search unit 308, and a sample An acquisition unit 309 is provided.
[0169]
The file input unit 301 is an interface for acquiring MP4 file data, and causes the file data storage unit 302 to store the acquired MP4 file input data sequentially.
The file data storage unit 302 is a cache memory or RAM that temporarily holds MP4 input data.
[0170]
The header separation analysis unit 303 reads and analyzes the MP4 file header data out of the MP4 input data held in the file data storage unit 302, and the MP4 file basic part header moov data and the extension part header moof data. And processing units that are output to the moov analysis unit 304 and the moof analysis unit 305, respectively, and are realized by a CPU or a memory.
[0171]
The moov analysis unit 304 is a processing unit that analyzes the moov of the MP4 file and acquires media information necessary for the analysis of the media data such as the encoding rate of the media data and the playback time length of the content. Realized. The moov analysis unit outputs the acquired media information to the moof analysis unit 305.
[0172]
The moof analysis unit 305 is a processing unit that analyzes the moof of the MP4 file based on the media information acquired from the moov analysis unit 304, and outputs the traf data that is header data for each track to the traf analysis unit 306. And memory.
[0173]
The traf analysis unit 306 is a processing unit that analyzes the traf of the MP4 file and outputs the run data, which is header data for each sample included in the traf, to the run analysis unit 307, and is realized by a CPU or a memory.
The run analysis unit 307 is a processing unit that analyzes the run of the MP4 file, acquires information described in each field in the run, and outputs the run analysis information to the sample acquisition unit 309. It is realized by. The trun analysis information includes, for example, the size of the sample, data offset information indicating where the sample is stored in the file data storage unit 302, and whether it is an intra frame in the case of a video sample. The flag information indicating that is included.
[0174]
Also, the trun analysis unit 307 indicates a reproduction start position after random access from the RA search unit 308 described below, and obtains a reproduction start instruction that is an output signal instructing the start of reproduction. The analysis is sequentially performed from the “run”, and the run analysis information is output to the sample acquisition unit 309.
[0175]
The RA search unit 308 acquires target playback time information indicating the playback start time after random access, and plays start time for the first sample included in the first trun in the first traf that stores the header information regarding the video track. And a processing unit that reads out the head sample information, which is information indicating whether the frame is an intra frame, and searches for a video sample as a playback start position after random access, and is realized by a CPU or a memory. When the RA search unit 308 acquires the target playback time information from the input device of the demultiplexer 300 that accepts a random access instruction from the user, the RA search unit 308 sequentially acquires only the top sample information from the run analysis unit 307, and acquires the target playback time. A video sample having a playback start time that is the same as or similar to the information is searched, and a playback start instruction is output to the run analysis unit 307.
[0176]
The sample acquisition unit 309 is a processing unit that reads and decodes the actual data of the sample based on the trun analysis information, and outputs the reproduction data to a display device such as a display. When the sample acquisition unit 309 acquires the run analysis information from the run analysis unit 307, the sample acquisition unit 309 refers to the data offset information included therein and reads the actual data of the sample from the file data storage unit 302. Here, it is assumed that the reproduction start is instructed when the acquisition of the trun analysis information is started.
[0177]
A random access processing operation in the demultiplexing apparatus 300 configured as described above will be described with reference to FIG.
FIG. 16 is a flowchart showing the random access processing operation of the demultiplexer 300. Prior to this flow, it is assumed that the demultiplexer 300 receives a random access instruction from the user via the input device.
[0178]
First, when the file input unit 301 acquires the data of the MP4 file created in the multiplexing device according to the first, second, or third embodiment (S400), the demultiplexing device 300 sequentially stores the file data storage unit 302. To accumulate.
[0179]
Next, the demultiplexing apparatus 300 separates and analyzes only the file header part of the MP4 file in the header separation analysis unit 303 (S410), and further separates the basic part header and the extension part header into moov. The analysis unit 304 analyzes the basic part header, and the moof analysis unit 305 analyzes the extension part header (S420).
[0180]
Subsequently, in the demultiplexing apparatus 300, the moof analysis unit 305 further separates the extension unit header into headers for each track, and the traf analysis unit 306 analyzes the track fragment, that is, traf (S430). At this time, the demultiplexing apparatus 300 further separates the track fragments in the traf analysis unit 306 and analyzes the trun in the run analysis unit 307.
[0181]
Here, when the target reproduction time information is input in the RA search unit 308, the demultiplexing apparatus 300 outputs the head sample information from the run analysis unit 307 to the RA search unit 308, and the RA search unit 308 outputs the target reproduction time. It is determined whether or not the first sample information indicates the reproduction start time that is the same as or similar to the time information (S440).
[0182]
At this time, if the target sample is not found (No in S450), the demultiplexing apparatus 300 acquires the first sample information in the extension unit header arranged next in the storage order in the file in the RA search unit 308. Then, it is determined whether or not the head sample information indicates the reproduction start time that is the same as or similar to the previously acquired target reproduction time information (S440).
[0183]
On the other hand, if the target sample is found (Yes in S450), the demultiplexer 300 generates a reproduction start instruction in the RA search unit 308 and outputs the reproduction start instruction to the run analysis unit 307. When receiving the playback start instruction from the RA search unit 308, the run analysis unit 307 outputs the run analysis information to the sample acquisition unit 309 in order from the trun that has received the playback start instruction. Here, the trun that has received the instruction to start reproduction refers to the trun that includes the sample instructed to start reproduction by the RA search unit 308.
[0184]
Thereafter, the demultiplexer 300 refers to the data offset information included in the run analysis information in the sample acquisition unit 309, acquires the actual data of the target sample from the file data storage unit 302 (S460), decodes it. The reproduction data is output and the random access processing operation is terminated.
[0185]
As described above, according to the demultiplexing apparatus 300 according to the fourth embodiment, random access is performed on the MP4 file including the MP4 file extension generated by the multiplexing apparatus according to the first, second, or third embodiment. When performing playback, it is possible to determine the video sample that should be the playback start position after random access by searching only the video sample stored at the beginning of each packet. The load will be greatly reduced.
[0186]
(Application example)
Here, an application example of the multiplexing apparatus according to the present invention will be described with reference to FIG.
FIG. 17 is a diagram illustrating an application example of the multiplexing device according to the present invention.
The multiplexing device according to the present invention can be applied to a cellular phone 403 with a recording function and a personal computer 404 that acquire and multiplex media data such as video data and audio data and create MP4 file data. Further, the demultiplexing apparatus according to the present invention can be applied to a mobile phone 407 that reads and reproduces the created MP4 file data.
[0187]
Here, the MP4 file data created in the cellular phone with recording function 403 and the personal computer 404 is stored in a recording medium such as the SD memory card 405 or the DVD-RAM 406 or stored in the image distribution server 401 via the communication network 402. And transmitted from the image distribution server 401 to another mobile phone 407 or the like.
[0188]
As described above, the multiplexing device and the demultiplexing device according to the present invention are used as an MP4 file creation device or playback device in an image distribution system or the like.
As described above, the multiplexing apparatus and the demultiplexing apparatus according to the present invention have been described based on the respective embodiments and the like, but the present invention is not limited to these embodiments and the like.
[0189]
For example, in each of the above embodiments, MPEG-4 Visual encoded data is used as the video data, but other moving images such as MPEG-4 AVC (Advanced Video Coding) and H.263 are used as the video data. You may use the encoding data by an image compression encoding system. In MPEG-4 AVC (Advanced Video Coding) and H.263 encoded data, one picture corresponds to one sample.
[0190]
Similarly, MPEG-4 Audio encoded data is used as the audio data, but encoded data according to another audio compression encoding method such as G.726 may be used as the audio data.
In each of the above embodiments, the video data and the audio data are described. However, even when text data or the like is included, the processing is performed in the same manner as the packetization of the audio data. The effects of the invention can be obtained.
[0191]
Further, in the second embodiment, when packetization is performed for each intra frame, the time reference unit adjustment unit 120 is omitted from the components of the packet unit determination unit 117, and the process of step S205 in FIG. It may be omitted.
[0192]
Furthermore, in the third embodiment, when an MP4 file is to be played in accordance with a buffer model set in advance on the playback device side of the MP4 file, the video sample is set so as to satisfy the buffer model. Data and audio sample data may be interleaved and stored in mdat. Here, the buffer model means that when encoded data is input in accordance with the conditions defined in the standard, the buffer is emptied by providing the playback apparatus with a buffer having a size defined in the standard (underscore). This is a model for ensuring that the playback apparatus can perform decoding without overflowing (overflow) from the buffer.
[0193]
In the first, second, and third embodiments, the number of trafs stored in the moof of the extension part of the MP4 file to be created is not mentioned, but the traf stored in the moof is one traf per track. Is preferably stored. In this way, if only the first traf in the moof is analyzed for each track, the header information for all the samples of the track stored in the moof can be obtained, so the efficiency at the time of obtaining the header information Will be further improved.
[0194]
Furthermore, in Embodiments 1, 2, and 3, sample entity data in which header information is stored in the moof of the extension part of the MP4 file to be created is stored in one mdat continuous with moof. It is good also as dividing | segmenting and storing in several mdat continuous with moof. More specifically, sample entity data in which header information is stored in moof_1 is stored in the order of mdat_1, mdat_2, and mdat_3, and sample entity data in which header information is stored in moof_2 is stored in mdat_4, mdat_5, and mdat_6. You may store in order.
[0195]
In Embodiments 2 and 3, when an intra frame of moving image data is included in a packet, it is arranged at the head of the packet. However, if random access is possible, P (Predictive) Video samples other than intra frames, such as frames and B (Bidirectionally predictive) frames, may be arranged at the head of the packet. Hereinafter, this will be described by taking as an example the case of using MPEG-4 AVC encoded data as video data.
[0196]
In MPEG-4 AVC, there is a case where a correct decoding result cannot be obtained even when decoding from an intra picture. More specifically, there are two types of MPEG-4 AVC intra pictures: IDR (Instantaneous Decoder Refresh) pictures and other pictures (hereinafter referred to as non-IDR intra pictures), which are decoded from IDR pictures. When decoding is started, a correct decoding result can always be obtained, but when decoding is started from a non-IDR intra picture, a non-IDR intra picture and a plurality of pictures after the non-IDR intra picture in display order A correct decoding result may not be obtained.
[0197]
Therefore, in MPEG-4 AVC, in order to obtain a correct decoding result from a non-IDR intra picture, auxiliary information indicating which picture should be decoded (Recovery Point Supplemental Enhancement Information, hereinafter referred to as “Recovery Point”).
SEI ”) can be added.
[0198]
For example, five pictures indicated by Pic_1, Pic_2, Pic_3, Pic_4, and Pic_5 are included in the video data in this order, Pic_5 is a non-IDR intra picture, and Pic_5 and Pic_5 and subsequent pictures are correctly decoded in display order. If decoding is to be started from Pic_1, the Recovery Point SEI is placed immediately before Pic_1, so that Pic_5, which is the fourth picture in the storage order in the video data, and the display are displayed. In order to correctly decode the subsequent pictures in order, it can be shown that it is necessary to start decoding from Pic_1.
[0199]
That is, in this case, it can be said that Pic_1 is a randomly accessible sample. Therefore, in the case of MPEG-4 AVC encoded data, a sample of a picture to which an IDR picture or a recovery point SEI is added can be randomly accessed. It is good also as arrange | positioning at the head of a packet as a sample. Note that Recovery Point SEI can be added to pictures other than intra pictures.
[0200]
At this time, the sample of the picture to which the Recovery Point SEI is added and the picture sample from which a correct decoding result can be obtained by starting decoding from the picture to which the Recovery Point SEI is added are stored in the same packet. As a result, the processing amount at the time of obtaining sample data can be reduced.
[0201]
Furthermore, an IDR picture and a sample of a picture to which Recovery Point SEI is added can be identified by a specific flag value (hereinafter referred to as a non-sync sample flag) in the head sample flag 930 or the sample flag 935. . In MP4, the non-sync sample flag can be set to 0 only for samples in which random access samples and samples for which correct decoding results are obtained among samples that can be accessed randomly. For this reason, the IDR picture sample can be identified by setting the nonsync sample flag to 0, and the picture sample to which the Recovery Point SEI is added by setting the nonsync sample flag to 1.
[0202]
By using the identification method as described above, it is possible to identify randomly accessible samples having different properties, not limited to pictures to which IDR pictures and Recovery Point SEI are added. In practice, it can be used as follows.
[0203]
The first is a case where fast-forward playback is performed by playing back only a specific sample. At this time, since it is desirable that the decoded sample can be displayed immediately, only the sample whose non-sync sample flag is 0 is decoded and reproduced.
[0204]
The second is a case where playback is started from the middle of the content, or a specific section is skipped and playback of the next section is started. At this time, there is a possibility that the sample from which decoding is started and the sample from which a correct decoding result is obtained are only at the start of reproduction. Therefore, it is assumed that reproduction can be started from either a sample whose nonsync sample flag is 0 or a randomly accessible sample whose nonsync sample flag is 1.
[0205]
Note that such a storage method is not limited to MPEG-4 AVC Recovery Point SEI, and can be applied when a sample for starting decoding is different from a sample from which a correct decoding result is obtained. It can be applied to a structure such as Open GOP (Group Of Pictures) in MPEG2-Video.
[0206]
Further, when identification information indicating that the sample is randomly accessible is added, the sample indicated by the identification information as being randomly accessible may be arranged at the head of the packet. .
[0207]
【The invention's effect】
As is apparent from the above description, according to the multiplexing device of the present invention, the reproduction start times of the image data, audio data, and text data included in the media data are aligned and stored in the packet. The efficiency of data access during playback on the device side can be realized.
[0208]
In addition, by using the first video sample included in the packet as an intra-frame video sample, it is possible to significantly reduce the amount of calculation required for sample search at random access on the playback device side. The
In addition, video samples and audio samples contained in the packet are stored in ascending order of playback start time, so the number of seek operations during random access on the playback device can be reduced, and playback with a slow seek speed is possible. Multiplexing that enables quick random access reproduction can be realized even in the apparatus.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a functional configuration of a multiplexing apparatus according to Embodiment 1 of the present invention.
FIG. 2 is a flowchart showing a processing operation of the multiplexing apparatus.
FIG. 3 is a flowchart showing a processing operation of a video packet unit determination unit.
FIG. 4 is a flowchart showing a processing operation of an audio packet unit determination unit.
FIG. 5A is a diagram illustrating a first example of a data structure of an MP4 file extension unit created by a multiplexing device, and FIG. 5B is a data structure of an MP4 file extension unit created by a multiplexing device; It is a figure which shows the 2nd example.
FIG. 6 is a block diagram showing a functional configuration of a packet unit determination unit of the multiplexing apparatus according to the second embodiment.
FIG. 7 is a flowchart showing a first processing operation of a video packet unit determination unit.
FIG. 8 is a flowchart showing a second processing operation of the video packet unit determination unit.
FIG. 9A is a diagram illustrating a first example of a data structure of an MP4 file extension created by a multiplexing device, and FIG. 9B is a data structure of an MP4 file extension created by a multiplexing device; It is a figure which shows the 2nd example.
FIG. 10 is a block diagram showing a functional configuration of a packet data creation unit of the multiplexing apparatus according to the third embodiment.
FIG. 11 is a flowchart showing a processing operation of a packet data creation unit.
FIG. 12 is a diagram showing an outline of a data structure of an MP4 file extension created by a multiplexing device.
FIG. 13 is a diagram illustrating a first example of a data structure of an MP4 file extension created by the multiplexing device.
FIG. 14 is a diagram illustrating a second example of the data structure of the MP4 file extension created by the multiplexing device.
FIG. 15 is a block diagram showing a functional configuration of a demultiplexing apparatus according to the fourth embodiment.
FIG. 16 is a flowchart showing the processing operation of the demultiplexer.
FIG. 17 is a diagram illustrating an application example of a multiplexing device according to the present invention.
FIG. 18 is a diagram for explaining the structure of a box constituting a conventional MP4 file.
FIG. 19 is a diagram for explaining a basic part of a conventional MP4 file.
20A is a diagram for explaining the structure of a movie box in a conventional MP4 file, and FIG. 20B is a diagram showing the structure of a movie box in a conventional MP4 file in a tree shape.
FIG. 21 is a diagram illustrating a structure of a conventional MP4 file including an extension unit.
FIG. 22 is a diagram for explaining the structure of a conventional movie fragment box.
FIG. 23 is a diagram for explaining the structure of a conventional track fragment run box.
24A is a diagram illustrating a first configuration example of an MP4 file including a conventional extension unit, and FIG. 24B is a diagram illustrating a second configuration example of an MP4 file including a conventional extension unit. is there.
FIG. 25 is a block diagram showing a configuration of a conventional multiplexing apparatus.
FIG. 26 is a flowchart showing a processing operation of a conventional packet unit determination unit.
FIG. 27 is a diagram showing an example of a packet creation table for storing conventional header information of video samples.
FIG. 28A is a diagram for explaining a first problem of the conventional multiplexing device, and FIG. 28B is a diagram for explaining a second problem of the conventional multiplexing device. FIG.
[Explanation of symbols]
100, 960 Multiplexer
101, 961 1st input part
102, 962 First data storage unit
103, 963 First analysis unit
104, 964 second input section
105, 965 Second data storage unit
106,966 Second analysis unit
107, 117, 967 Packet unit determination unit
108 Time adjustment section
109, 119 Video packet unit determination unit
110 Audio packet unit determination unit
111, 968 packet creation table storage unit
112,969 Packet header creation unit
113, 130, 970 Packet data creation unit
114, 971 packet combining part
120 Time base unit adjustment section
121 I frame reference unit adjustment unit
131 mdat information acquisition unit
132 Video substance data reading unit
133 Audio entity data reading unit
134 Interleaved array part
200, 210, 220, 230, 240, 250 MP4 file extension
241, 923, 946, 948, 955, 957 Movie fragment box
242, 916, 945, 947, 949, 956, 958 Movie data box
300 Demultiplexer
301 File input section
302 File data storage unit
303 Header separation analysis unit
304 moov analysis part
305 moof analysis part
306 traf analysis unit
307 run analysis part
308 RA search unit
309 Sample acquisition unit
401 Image distribution server
402 Communication network
403 Mobile phone with recording function
404 Personal computer
405 SD memory card
406 DVD-RAM
407 mobile phone
901 box
902 Box header
903 Box data storage
904 box size
905 box type
906 version
907 flag
910, 920, 940, 950 MP4 file
911, 941, 951 Basic part
912 File header
913 File data part
914, 943, 953 File type box
915, 944, 954 Movie box
917 Movie header box
918 Track box
919 Track header box
921, 942, 952 Expansion part
922 packets
924 Movie fragment header box
925 track fragment box
926 Track fragment header box
927 truck fragment runbox
928 sample count
929 data offset
930 First sample flag
931 table
932 entries
933 Sample duration
934 sample size
935 sample flag
936 Sample composition time offset
968a packet creation table

Claims (16)

A multiplexing apparatus that packet-multiplexes media data including image data and at least one of audio data and text data to create multiplexed data,
Media data acquisition means for acquiring the media data;
Analysis of the media data acquired by the media data acquisition means, and a playback start indicating the playback start time of the sample for the sample that is the minimum access unit of the image data, audio data, and text data included in the media data An analysis means for acquiring time information;
Based on the reproduction start time information acquired by the analysis means, a unit for packetizing the media data is determined by aligning the reproduction start times of the samples of the image data, audio data, and text data included in the media data. A packet unit determining means to perform,
A packet header part creating means for creating a packet header part for storing a header of the media data in a packetization unit determined by the packet unit determining means;
A packet data part creating means for creating a packet data part for storing the actual data of the media data in a packetization unit determined by the packet unit determining means;
A packetizing unit for creating a packet by combining the packet header part created by the packet header part creating unit and the packet data part created by the packet data part creating unit ;
The packet unit determining means determines the unit by aligning the reproduction start times of the samples of the image data, audio data, and text data for all packets necessary for storing the media data. Multiplexer to do. The packet unit determining means includes
The reproduction start times of the audio data and text data samples arranged at the beginning of the packetization unit are aligned with the reproduction start times of the image data samples arranged at the beginning of the packetization unit. The multiplexing apparatus according to claim 1. The packet unit determining means includes
The audio data and the text data sample arranged at the beginning of the packetization unit are after the reproduction start time of the image data sample arranged at the beginning of the packetization unit, and the sample of the image data The multiplexing apparatus according to claim 2, wherein a reproduction start time sample closest to the reproduction start time is used. The packet unit determining means includes
The audio data and the text data sample arranged at the head of the packetization unit are before the reproduction start time of the image data sample arranged at the head of the packetization unit, and the image data sample The multiplexing apparatus according to claim 2, wherein a reproduction start time sample closest to the reproduction start time is used. The image data is video data,
The analysis means further includes:
When the moving image data acquired by the media data acquisition unit is analyzed, and the moving image data includes one or more samples including intra frame information indicating that it is an intra-screen encoded sample, the intra Get frame information
The packet unit determining means includes
The unit for packetizing the media data is determined based on the intra frame information and the reproduction start time information when the analyzing unit acquires the intra frame information. Multiplexer. The packet unit determining means includes
The multiplexing apparatus according to claim 5, wherein a sample of the moving image data including the intra frame information is arranged at a head of the packetization unit. The packet unit determining means includes
The playback start time of the audio data and text data samples placed at the beginning of the packetization unit at the playback start time of the video data samples including the intra frame information placed at the beginning of the packetization unit The multiplexing apparatus according to claim 6, wherein: The packet data part creation means includes:
2. The multiplexing apparatus according to claim 1, wherein the packet data unit that stores and interleaves the media data samples included in the packetization unit so that the reproduction start times of the samples are in ascending order is created. . The packet data part creation means includes:
The multiplexing apparatus according to claim 8, wherein the packet data unit that stores the media data samples included in the packetization unit in an interleaved manner so as to satisfy a predetermined rule is created. A multiplexing method for creating multiplexed data by packet multiplexing media data including image data and at least one of audio data and text data,
A media data acquisition step of acquiring the media data;
The media data acquired in the media data acquisition step is analyzed, and a playback start indicating the playback start time of the sample for the sample that is the minimum access unit of the image data, audio data, and text data included in the media data is started. An analysis step to obtain time information;
Based on the reproduction start time information acquired in the analysis step, a unit for packetizing the media data is determined by aligning the reproduction start times of the samples of the image data, audio data, and text data included in the media data. A packet unit determination step to be performed;
A packet header part creating step for creating a packet header part for storing a header of the media data in the packetization unit determined in the packet unit determiner step;
A packet data part creating step for creating a packet data part for storing the actual data of the media data in the packetization unit determined in the packet unit determining step;
Seen containing a packet header portion created in the packet header part generation step, and a packetizing step of creating a packet by combining the packet data unit that was created in the packet data unit generating step,
In the packet unit determination step, the unit is determined by aligning the reproduction start times of the samples of the image data, audio data, and text data for all packets necessary for storing the media data. Multiplexing method to do. In the packet unit determination step,
The reproduction start time of the audio data and the text data sample arranged at the head of the packetization unit is aligned with the reproduction start time of the image data sample arranged at the head of the packetization unit. The multiplexing method according to claim 10. The image data is video data,
In the analyzing step,
When the moving image data acquired in the media data acquisition step is analyzed, and the moving image data includes one or more samples including intra frame information indicating that it is an intra-screen encoded sample, the intra Get frame information
In the packet unit determination step,
The unit for packetizing the media data is determined based on the intra frame information and the reproduction start time information when the intra frame information is acquired in the analyzing step. Multiplexing method. In the packet unit determination step,
The multiplexing method according to claim 12, wherein a sample of the moving image data including the intra frame information is arranged at the head of the packetization unit. In the packet unit determination step,
The playback start time of the audio data and text data samples placed at the beginning of the packetization unit at the playback start time of the video data sample including the intra frame information placed at the beginning of the packetization unit The multiplexing method according to claim 13, wherein: In the packet data part creation step,
11. The multiplexing method according to claim 10, wherein the packet data unit is generated to interleave and store the media data samples included in the packetization unit so that the reproduction start times of the samples are in ascending order. . A program for a multiplexing device that creates multiplexed data by packet multiplexing media data including image data and at least one of audio data and text data,
A media data acquisition step of acquiring the media data;
The media data acquired in the media data acquisition step is analyzed, and a playback start indicating the playback start time of the sample for the sample that is the minimum access unit of the image data, audio data, and text data included in the media data is started. An analysis step to obtain time information;
Based on the reproduction start time information acquired in the analysis step, a unit for packetizing the media data is determined by aligning the reproduction start times of the samples of the image data, audio data, and text data included in the media data. A packet unit determination step to be performed;
A packet header part creating step for creating a packet header part for storing a header of the media data in the packetization unit determined in the packet unit determiner step;
A packet data part creating step for creating a packet data part for storing the actual data of the media data in the packetization unit determined in the packet unit determining step;
Seen containing a packet header portion created in the packet header part generation step, and a packetizing step of creating a packet by combining the packet data unit that was created in the packet data unit generating step,
In the packet unit determining step, in the multiplexing method for determining the unit by aligning the reproduction start times of the samples of the image data, audio data, and text data for all the packets necessary for storing the media data A program characterized by causing a computer to execute each step.

Images