JP2012257078A - Image and sound data processor and data multiplexing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce the overflow of image data and sound data.SOLUTION: An image and sound data processor includes: a first storage part for storing encoded image data; a second storage part for storing encoded sound data; and a chunk determination part for monitoring the respective occupation amounts of the first storage part and the second storage part and reduces the number of frames of a chunk to be smaller than the number of frames of a GOP (Group Of Picture) when at least one condition is satisfied between the first condition that the occupation amount of the first storage part is equal to or more than a predetermined first threshold and the second condition that the occupation amount of the second storage part is equal to or more than a predetermined second threshold.

Description

  The present invention relates to a video / audio data processing apparatus and a data multiplexing method.

  In recent years, the MP4 file format has become widespread as a file format for handling video data and audio data. The MP4 file format is defined in the MPEG4 system standard (ISO / IEC 14496-1). The MP4 file format is widely used in the field of PC applications because it can be easily processed by a personal computer (hereinafter also referred to as a PC). For example, video and audio stream data can be easily edited by using the MP4 file format.

  The MP4 file has a data portion in which video data and audio data are stored, and a header portion in which attached information is stored. In the MP4 file, video data and audio data are managed in units of samples and chunks. A sample is a minimum unit for managing video data and audio data. For example, in video data, one sample is one video frame. In the audio data, one sample is one audio frame. A plurality of samples arranged together in the data portion are managed as chunks. In the MP4 file, samples in the same chunk are continuously arranged in the data portion.

  The number of samples in a chunk is not specified in the MP4 file format. Therefore, the number of chunk samples is arbitrarily determined by the user. In a general method, frames of 1 GOP (Group of Picture) of video data are combined into one chunk. For example, the number of samples (number of frames) of each chunk is set so that the first frame of the chunk is an I picture (I frame). The GOP is a collection of pictures including at least one I picture, and may include a P picture (frame) and a B picture (frame) in addition to the I picture. In this method, when the interval between I pictures is long, the number of chunk samples increases. A method has been proposed in which the maximum number of frames (samples) in a chunk is set in advance and the number of frames in a chunk is prevented from exceeding a preset maximum number (see, for example, Patent Document 1).

  In general, in a video / audio data processing apparatus such as a codec LSI, the capacity of an internal memory is limited, and therefore it is not realistic to generate an MP4 file internally. Therefore, for example, in a system that stores video data and audio data encoded by a codec LSI in an MP4 file, the MP4 file is generated by a module outside the codec LSI. For example, the codec LSI individually outputs data stored in the data portion of the MP4 file and attached information stored in the header portion. Then, for example, a module external to the codec LSI synthesizes the data and the attached information to generate an MP4 file.

JP 2004-128938 A

  A video / audio data processing apparatus such as a codec LSI alternately outputs video data and audio data in units of chunks. For this reason, in the period in which one of the video data and the audio data is output, the other is kept waiting. When the output waiting period is long, either the memory for temporarily storing the video data or the memory for temporarily storing the audio data may overflow. When overflow occurs, valid data is lost.

  The output waiting period becomes longer, for example, when the number of chunk samples is large. Note that with the method of setting the maximum number of samples in a chunk in advance, there is a risk that the chunk will be unnecessarily subdivided even when the free capacity of the memory is large. The attached information stored in the header part of the MP4 file is generated for each chunk. For this reason, when chunks are subdivided unnecessarily, the size of the header portion of the MP4 file may increase. As the size of the header portion of the MP4 file increases, the size of the entire MP4 file increases. A large MP4 file is difficult to handle in a PC application.

  An object of the present invention is to reduce overflow of video data and audio data.

  In one aspect of the present invention, a video / audio data processing device includes a first storage unit that stores encoded video data, a second storage unit that stores encoded audio data, a first storage unit, Each occupancy of the second storage unit is monitored, the first condition that the occupancy of the first storage unit is equal to or greater than a predetermined first threshold, and the occupancy of the second storage unit is equal to or greater than a predetermined second threshold A chunk determination unit that reduces the number of chunk frames to less than the number of GOP frames when at least one of the two conditions is satisfied.

  Overflow of video data and audio data can be reduced.

1 illustrates an example of a video / audio data processing device according to an embodiment. An overview of the MP4 file format is shown. 2 shows an example of data output from the video / audio data processing apparatus shown in FIG. An example of the relationship between chunks and random access points is shown. 4 shows another example of the relationship between chunks and random access points. An example of the chunk when the memory occupation amount is equal to or greater than the threshold is shown. The other example of the chunk when the memory occupation amount is more than a threshold is shown. The other example of the chunk when the memory occupation amount is more than a threshold is shown. The other example of the chunk when the memory occupation amount is more than a threshold is shown. 3 shows an example of a video / audio data processing apparatus according to another embodiment. 11 shows an example of the operation of the video / audio data processing apparatus shown in FIG.

  Hereinafter, embodiments will be described with reference to the drawings.

  FIG. 1 shows an example of a video / audio data processing apparatus 10 according to an embodiment.

  The video / audio data processing apparatus 10 is mounted on, for example, a system that generates an MP4 file. The MP4 file format is defined in the MPEG4 system standard (ISO / IEC 14496-1). For example, the video / audio data processing apparatus 10 alternately outputs video data and audio data encoded by an encoding method compliant with MPEG-4 (ISO / IEC 14496) or the like to the external device in units of chunks. Then, for example, the external device stores the video data and audio data received from the video / audio data processing device 10 in an MP4 file, and generates an MP4 file.

  The video / audio data processing apparatus 10 includes a chunk determination unit 20 and a memory 30. The memory 30 sequentially stores, for example, video data and audio data encoded by an encoding method compliant with MPEG-4 or the like. Note that the video data and audio data stored in the memory 30 are sequentially output. That is, the memory 30 temporarily stores the encoded video data and audio data.

  For example, the memory 30 includes a video area 32 that temporarily stores encoded video data and an audio area 34 that temporarily stores encoded audio data. That is, the video area 32 of the memory 30 functions as a storage unit that temporarily stores the encoded video data. The audio area 34 of the memory 30 functions as a storage unit that temporarily stores encoded audio data. Note that the video area 32 and the audio area 34 may be provided in different memories.

  The chunk determination unit 20 monitors the occupation amount of the video area 32 and the occupation amount of the audio area 34, respectively. For example, the chunk determination unit 20 calculates the occupation amount of the video area 32 and the audio area 34 using information related to encoding of video data and audio data. The information related to encoding is, for example, the size of encoded video data or audio data, and is generated when encoding the video data or audio data. Note that the chunk determination unit 20 may calculate the occupation amount of the video area 32 and the audio area 34 based on the write address and the read address for the video area 32 and the audio area 34 of the memory 30.

  Then, for example, the chunk determination unit 20 compares the occupation amount of the video area 32 with a predetermined first threshold value, and compares the occupation amount of the audio area 34 with a predetermined second threshold value. Then, when the chunk determination unit 20 satisfies at least one of the first condition in which the occupation amount of the video area 32 is equal to or greater than the first threshold and the second condition in which the occupation amount of the audio area 34 is equal to or greater than the second threshold, The number of frames (number of samples) is made smaller than the number of frames of GOP (Group of Picture). The GOP is a data unit having a plurality of pictures, and has at least one I picture (I frame).

  As described above, the chunk determination unit 20 makes the number of chunk frames smaller than the number of GOP frames when the free capacity of at least one of the video area 32 and the audio area 34 is small. For example, the video / audio data processing device 10 multiplexes the encoded video data and audio data based on the number of chunk frames set by the chunk determination unit 20.

  The encoding of the video data and the audio data may be performed by the video / audio data processing device 10 or may be performed outside the video / audio data processing device 10. Therefore, the video / audio data processing apparatus 10 may have a function of encoding video data and audio data, or may not have a function of encoding video data and audio data. For example, when the video / audio data processing apparatus 10 does not have a function of encoding video data and audio data, the video / audio data processing apparatus 10 sequentially receives video data and audio data encoded by an external codec LSI or the like. Then, the video / audio data processing device 10 sequentially stores the encoded video data and audio data in the video area 32 and the audio area 34.

  FIG. 2 shows an outline of the MP4 file format. The MP4 file FIL has, for example, boxes B10, B20, and B30. There is only one file type box (ftyp) B10 at the head of the file. A brand name indicating the structure of the MP4 file FIL is stored in the file type box (ftyp) B10. There is only one movie box (moov) B20 in one file. The movie box (moov) B20 stores all header information related to mdat. For example, the size of each sample, the playback period of each sample, the playback period of the entire file, the position information of each chunk, and the like are stored in the movie box (moov) B20. In the media data box (mdat) B30, for example, encoded video data and audio data are stored. Note that a plurality of media data boxes (mdat) B30 may exist in one file.

  FIG. 3 shows an example of data output from the video / audio data processing apparatus 10 shown in FIG. The shaded portions in the figure indicate the chunk CNK of video data and the message MES of video data. The numbers in parentheses of the message MES and the numbers in parentheses of the chunk CNK indicate the output order.

  The video / audio data processing apparatus 10 alternately outputs the video data and audio data stored in the video area 32 and the audio area 34 to the external apparatus 100 in units of chunks CNK. The chunk CNK has a plurality of samples. A sample is a minimum unit for managing video data and audio data. For example, one sample of the chunk CNK of video data is one video frame FRM. Further, for example, one sample of the chunk CNK of the audio data is one audio frame.

  That is, the chunk CNK has a plurality of frames FRM. For example, the chunk CNK (1) of the video data has a frame FRM (1) -FRM (n). Further, for example, the chunk CNK (2) of the audio data has a frame FRM (n + 1) −FRM (m).

  Also, the video / audio data processing device 10 outputs a message MES corresponding to each chunk CNK to the external device 100. For example, the video / audio data processing apparatus 10 outputs the message MES from an interface different from the interface that outputs the video data and audio data.

  In the MP4 file, video data and audio data are managed in units of sample (frame FRM) and chunk CNK. For this reason, for example, the video / audio data processing device 10 generates a message MES for each chunk CNK. The message MES is, for example, header information related to the chunk CNK. Therefore, for example, the video / audio data processing apparatus 10 outputs each message MES in association with each chunk CNK. The broken lines in the figure indicate that each message MES is associated with each chunk CNK.

  The external device 100 generates an MP4 file based on the video data and audio data received from the video / audio data processing device 10 in units of chunks CNK and the message MES. For example, the external device 100 stores the message MES received from the video / audio data processing device 10 in the movie box (moov) B20 shown in FIG. For example, the external device 100 stores the video data and audio data received from the video / audio data processing device 10 in the media data box (mdat) B30 shown in FIG. In the MP4 file, samples (frame FRM) in the same chunk CNK are continuously arranged in the media data box (mdat) B30.

  Therefore, the video / audio data processing apparatus 10 alternately outputs the video data and audio data stored in the video area 32 and the audio area 34 in units of chunks CNK. Therefore, the output waiting time (output waiting period) of the video data and audio data is shortened by, for example, reducing the number of chunk frames. For example, as described with reference to FIG. 1, the video / audio data processing apparatus 10 reduces the number of chunk frames to be smaller than the number of GOP frames when the free space of at least one of the video area 32 and the audio area 34 is small.

  That is, in this embodiment, when the free space of at least one of the video area 32 and the audio area 34 is small, the number of chunk frames is made smaller than the number of GOP frames, thereby reducing the output waiting period of video data and audio data. Can be shortened. As a result, in this embodiment, the overflow of the other of the video area 32 and the audio area 34 during the period in which data is output from one of the video area 32 and the audio area 34 (the other output waiting period) is reduced. it can. That is, in this embodiment, overflow of video data and audio data can be reduced.

  FIG. 4 shows an example of the relationship between the chunk CNK and the random access point RAP. FIG. 4 shows an example in which two GOP frames FRMv are combined into one chunk CNKv. A frame FRMv indicated by a bold line in the figure indicates an IDR picture and an I picture. The frame FRMv indicates the frame FRM of the video data, and the frame FRMa indicates the frame FRM of the audio data. The chunk CNKv indicates the chunk CNK of the video data, and the chunk CNKa indicates the chunk CNK of the audio data.

  The numbers in parentheses of the frame FRMa indicate that the numbers in the parentheses correspond to the same frame FRMv. For example, the frame FRMa (1) is a frame FRM of audio data corresponding to the frame FRMv (1). The number in parentheses of chunk CNKa indicates that the numbers in parentheses correspond to the same chunk CNKv. For example, chunk CNKa (1) is a chunk CNK of audio data corresponding to chunk CNKv (1).

  The random access point RAP means a return point and a playback point when playing back an MP4 file from a specific position. For example, a random access point RAP is searched for during fast forward playback. The frame FRMv (sample) of the random access point RAP is a frame FRM corresponding to an IDR picture or an I picture that does not require a reference image of video data. The frame FRMa (sample) of the random access point RAP is an audio data frame FRM corresponding to the IDR picture or I picture of the video data.

  For example, frames FRMv (1), FRMv (n + 1), and FRMv (m + 1) are random access points RAP for video data. The frames FRMa (1), FRMa (n + 1), and FRMa (m + 1) are voice data random access points RAP. Information on the random access point RAP is stored in a box called “STSS” in the movie box (moov) B20 shown in FIG.

  The information (random access point RAP information) stored in the box “STSS” is the sample number (the order counted from the top of the file) of the frame FRM corresponding to the random access point RAP. For example, in the video data, sample numbers (“1”, “n + 1”, etc.) such as frames FRMv (1), FRMv (n + 1) are stored in the box “STSS”. Also, for example, in the audio data, the sample numbers of frames FRMa (1), FRMa (n + 1), etc. (“number of frames of chunk CNKv (1) +1”, “number of frames of chunk CNKv (1) + n + 1”, etc.) are boxes. Stored in “STSS”.

  FIG. 5 shows another example of the relationship between the chunk CNK and the random access point RAP. FIG. 5 shows an example when the frame FRMv of one GOP is combined into one chunk CNKv. The meaning of the frame FRMv indicated by the bold line in the figure is the same as that in FIG. Further, the meanings of the numbers in parentheses of the frame FRM and the numbers in parentheses of the chunk CNK are the same as those in FIG.

  In the example of FIG. 5, since the frame FRMv of one GOP is combined into one chunk CNKv, the first frame FRM of the chunk CNK corresponds to the random access point RAP. In the MP4 file, information indicating the position of the top frame FRM of the chunk CNK (offset address from the top of the file) is stored in a box called “STCO” in the movie box (moov) B20 shown in FIG. For this reason, in the MP4 file in which one chunk CNK corresponds to one GOP, the random access point RAP can be efficiently searched.

  Therefore, for example, when the free space of the video area 32 and the audio area 34 is large, the chunk determination unit 20 sets the number of frames FRM of each chunk CNK so that one chunk CNK corresponds to one GOP. When the number of frames FRM of chunk CNK is made smaller than the number of frames of GOP, the chunk determination unit 20 counts the number of frames FRM of each chunk CNK so that the random access point RAP corresponds to the first frame FRM of the chunk CNK. Set.

  For example, the chunk determination unit 20 sets the number of frames FRM of each chunk CNK so that the I picture or IDR picture becomes the first frame FRM of the chunk CNK. Thereby, in this embodiment, the search efficiency of the random access point RAP can be improved. For example, an apparatus that plays an MP4 file can efficiently search for a random access point RAP.

  In addition, when the number of chunk CNK frames FRM is smaller than the number of GOP frames, the chunk determination unit 20 closes the relationship between the display order and the encoding processing order when reproducing the MP4 file in the chunk CNK. Is set to the number of frames FRM of each chunk CNK. For example, the chunk determination unit 20 sets the number of frames FRM of the chunk CNK so that the order of the frames FRM can be changed within each chunk CNK to the display order when reproducing the MP4 file. Thereby, in this embodiment, the search efficiency of the frame FRM subject to the order change can be improved. For example, an apparatus that reproduces an MP4 file can efficiently search for a frame FRM whose order is to be changed when the order of the frames FRM is changed to the display order.

  FIG. 6 shows an example of the chunk CNK when the occupation amount of the memory 30 is equal to or larger than the threshold value. That is, FIG. 6 shows an example of the chunk CNK when at least one of the first condition in which the occupation amount of the video area 32 is equal to or larger than the first threshold and the second condition in which the occupation amount of the audio area 34 is equal to or larger than the second threshold is satisfied. Is shown. FIG. 6 shows an example of the chunk CNKv when the GOP has an IBBP structure. A frame FRMv indicated by a bold line in the figure indicates an I picture. The numbers in parentheses of the frame FRMv indicate the encoding processing order.

  In the example of FIG. 6, the GOP has 15 frames FRM. In the GOP having the IBBP structure, first, a frame FRMv (1) of an I picture is encoded. Next, the frames FRMv (2) and FRMv (3) of the B picture are sequentially encoded. Then, the process of encoding the P picture, the B picture, and the B picture in order is repeated until the frame FRMv (4) -FRMv (15). For example, the frame FRMv (4) of the P picture is encoded after the frame FRMv (3). Next to the frame FRMv (4), the frames FRMv (5) and FRMv (6) of the B picture are sequentially encoded.

  In the GOP having the IBBP structure, when the MP4 file is reproduced, the frame FRMv of the B picture is displayed before the frame FRMv of the I picture or the P picture that has been previously encoded. For example, the frame FRMv (1) of the I picture is displayed next to the frame FRMv (3) of the B picture. The P picture frame FRMv (4) is displayed next to the B picture frame FRMv (6).

  Therefore, in the GOP having the IBBP structure, the number of frames CRMv of the chunk CNKv is set to a multiple of 3. For example, the chunk determination unit 20 sets the number of the frame CRMv of the chunk CNKv to a minimum value of 2 or more (“3” in FIG. 6) among multiples of 3. Thereby, for example, the frames FRMv (1), FRMv (2), and FRMv (3) are managed as the chunk CNKv (1). Also, the frames FRMv (4), FRMv (5), and FRMv (6) are managed as chunk CNKv (2). The frames FRMv (13), FRMv (14), and FRMv (15) are managed as a chunk CNKv (5).

  As described above, the chunk determination unit 20 sets the number of frames of the chunk CNKv to a minimum value of 2 or more among the number of frames that can change the order of the frame FRMv in the display order when reproducing the MP4 file in the chunk CNKv. Set. The setting of the number of frames of chunk CNKv is not limited to this example. For example, one GOP includes the chunk CNKv of the frame FRMv (1) -FRMv (6), the chunk CNKv of the frame FRMv (7) -FRMv (12), and the chunk CNKv of the frame FRMv (13) -FRMv (15). It may be managed by.

  The audio data chunk CNKa is set to correspond to the video data chunk CNKv. For example, the chunk determination unit 20 sets the number of the frame FRMa of the chunk CNKa to the same number as the number of the frames FRMv of the chunk CNKv.

  FIG. 7 shows another example of the chunk CNK when the occupation amount of the memory 30 is equal to or larger than the threshold value. FIG. 7 shows an example of the chunk CNKv when the GOP has an IPPP structure and the number of GOP frames is an odd number. The meaning of the frame FRMv indicated by the bold line in the figure is the same as that in FIG. The meanings of the numbers in parentheses of the frame FRMv are the same as those in FIG.

  In the example of FIG. 7, the GOP has 15 frames FRM. In the GOP having the IPPP structure, first, a frame FRMv (1) of an I picture is encoded. Next, the frames FRMv (2) -FRMv (15) of the P picture are sequentially encoded. In addition, in the GOP having the IPPP structure, the display order when reproducing the MP4 file is the same as the encoding processing order.

  Accordingly, in the GOP having the IPPP structure, for example, the chunk determination unit 20 sets the number of frames FRMv of the chunk CNKv to 3. Thereby, the plurality of chunks CNKv included in one GOP are set to the same number of frames. For example, the frames FRMv (1), FRMv (2), and FRMv (3) are managed as chunk CNKv (1). Also, the frames FRMv (4), FRMv (5), and FRMv (6) are managed as chunk CNKv (2). The frames FRMv (13), FRMv (14), and FRMv (15) are managed as a chunk CNKv (5).

  Note that the number of the frame FRMv of the chunk CNKv may be set to a value of 2 or more other than 3. For example, the chunk determination unit 20 may set the number of frames FRMv of the chunk CNKv to 5. In this embodiment, an increase in the size of the MP4 file is suppressed by setting the number of the frame FRMv of the chunk CNKv to 2 or more.

  For example, the message MES is generated for each chunk CNK as described with reference to FIG. For this reason, when the number of frames CRMv of chunk CNKv is set to 1, the number of message MESs increases and the total size of message MESs increases. The message MES is stored in the movie box (moov) B20 shown in FIG. For this reason, the size of the movie box (moov) B20 increases, and the size of the MP4 file increases. In this embodiment, since the number of frames FRMv of chunk CNKv is set to 2 or more, it is possible to suppress an increase in the size of the MP4 file.

  The audio data chunk CNKa is set to correspond to the video data chunk CNKv. For example, the chunk determination unit 20 sets the number of the frame FRMa of the chunk CNKa to the same number as the number of the frames FRMv of the chunk CNKv.

  FIG. 8 shows another example of the chunk CNK when the occupation amount of the memory 30 is equal to or larger than the threshold value. FIG. 8 shows an example of the chunk CNKv when the GOP has an IPPP structure and the number of GOP frames is an even number. The meaning of the frame FRMv indicated by the bold line in the figure is the same as that in FIG. The meanings of the numbers in parentheses of the frame FRMv are the same as those in FIG.

  In the example of FIG. 8, the GOP has 16 frames FRM. In the GOP having the IPPP structure, first, a frame FRMv (1) of an I picture is encoded. Next, the frames FRMv (2) -FRMv (16) of the P picture are sequentially encoded. In addition, in the GOP having the IPPP structure, the display order when reproducing the MP4 file is the same as the encoding processing order.

  Accordingly, in the GOP having the IPPP structure, for example, the chunk determination unit 20 sets the number of frames FRMv of the chunk CNKv to 2. Thereby, the plurality of chunks CNKv included in one GOP are set to the same number of frames. For example, the frames FRMv (1) and FRMv (2) are managed as chunk CNKv (1). The frames FRMv (3) and FRMv (4) are managed as chunk CNKv (2). Frames FRMv (5) and FRMv (6) are managed as chunk CNKv (3). Frames FRMv (15) and FRMv (16) are managed as chunk CNKv (8).

  Note that the number of frames CRMv of the chunk CNKv may be set to a value of 2 or more other than 2. For example, the chunk determination unit 20 may set the number of frames FRMv of the chunk CNKv to 4 or may set the number of frames FRMv of the chunk CNKv to 8. Thus, in this embodiment, since the number of chunk CNKv frames FRMv is set to 2 or more, an increase in the size of the MP4 file can be suppressed.

  The audio data chunk CNKa is set to correspond to the video data chunk CNKv. For example, the chunk determination unit 20 sets the number of the frame FRMa of the chunk CNKa to the same number as the number of the frames FRMv of the chunk CNKv.

  FIG. 9 shows another example of the chunk CNK when the occupation amount of the memory 30 is equal to or larger than the threshold value. FIG. 9 shows an example of the chunk CNKv when the GOP has an IBP structure. The meaning of the frame FRMv indicated by the bold line in the figure is the same as that in FIG. The meanings of the numbers in parentheses of the frame FRMv are the same as those in FIG.

  In the example of FIG. 9, the GOP has 16 frames FRM. In the GOP having the IBP structure, first, a frame FRMv (1) of an I picture is encoded. Next, the frame FRMv (2) of the B picture is encoded. Then, the process of encoding the P picture and the B picture in order is repeated until the frame FRMv (3) -FRMv (16). For example, the frame FRMv (3) of the P picture is encoded after the frame FRMv (2). Next to the frame FRMv (3), the frame FRMv (4) of the B picture is encoded.

  In the GOP having the IBP structure, when the MP4 file is reproduced, the frame FRMv of the B picture is displayed before the frame FRMv of the I picture or P picture that has been previously encoded. For example, the frame FRMv (1) of the I picture is displayed next to the frame FRMv (2) of the B picture. The P picture frame FRMv (3) is displayed next to the B picture frame FRMv (4).

  Therefore, in the GOP having the IBP structure, the number of frames FRMv of the chunk CNKv is set to a multiple of 2. For example, the chunk determination unit 20 sets the number of the frame CRMv of the chunk CNKv to a minimum value of 2 or more (“2” in FIG. 9) among multiples of 2. Thereby, for example, the frames FRMv (1) and FRMv (2) are managed as the chunk CNKv (1). The frames FRMv (3) and FRMv (4) are managed as chunk CNKv (2). Frames FRMv (5) and FRMv (6) are managed as chunk CNKv (3). Frames FRMv (15) and FRMv (16) are managed as chunk CNKv (8).

  Note that the number of frames FRMv of the chunk CNKv may be set to a multiple of 2 other than 2. For example, the chunk determination unit 20 may set the number of frames FRMv of the chunk CNKv to 4 or may set the number of frames FRMv of the chunk CNKv to 8. Thus, in this embodiment, since the number of chunk CNKv frames FRMv is set to 2 or more, an increase in the size of the MP4 file can be suppressed.

  The audio data chunk CNKa is set to correspond to the video data chunk CNKv. For example, the chunk determination unit 20 sets the number of the frame FRMa of the chunk CNKa to the same number as the number of the frames FRMv of the chunk CNKv.

  As described with reference to FIG. 6 to FIG. 9, when the number of frames FRM of the chunk CNK is smaller than the number of GOP frames, the chunk determination unit 20 determines the chunk based on the GOP structure and the GOP size (frame number). Sets the number of CNK frames.

  The operation of the video / audio data processing apparatus 10 is not limited to this example. For example, the chunk determination unit 20 may change the number of frames of the chunk CNK step by step according to the free space in the video area 32 and the audio area 34. For example, at least one of the first threshold value of the video area 32 and the second threshold value of the audio area 34 may have a plurality of values. Then, the chunk determination unit 20 may change the number of frames of the chunk CNK step by step according to the occupation amount compared with a plurality of values.

  For example, the chunk determination unit 20 sets a plurality of values for the threshold value of the video area 32, and changes the number of frames of the chunk CNK step by step according to the comparison result between the occupied amount of the video area 32 and the plurality of values. May be. Alternatively, the chunk determination unit 20 sets a plurality of values for the threshold value of the voice area 34, and changes the number of frames of the chunk CNK stepwise in accordance with the comparison result between the occupation amount of the voice area 34 and the plurality of values. May be.

  As described above, in this embodiment, the video / audio data processing device 10 makes the number of chunk frames smaller than the number of GOP frames when the free space of at least one of the video area 32 and the audio area 34 is small. For example, the number of frames of the chunk CNK is such that the relationship between the display order when reproducing the MP4 file and the encoding processing order is closed in the chunk CNK, and the I picture or IDR picture becomes the first frame FRM of the chunk CNK. It is set to a number that satisfies both of the conditions. As a result, in this embodiment, the search efficiency of the random access point RAP and the search efficiency of the target frame FRM whose order is changed can be improved. As a result, for example, an apparatus that plays an MP4 file can efficiently search for a random access point RAP. Further, for example, in an apparatus that plays back an MP4 file, when changing the order of the frames FRM to the display order, it is possible to efficiently search for the frame FRM whose order is changed.

  FIG. 10 shows an example of the video / audio data processing device 12 in another embodiment. The same elements as those described in the above-described embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted. The video / audio data processing device 12 is mounted on, for example, a system that generates an MP4 file.

  The video / audio data processing device 12 includes, for example, a system unit 22, a memory 30, a data encoding unit 40, and a multiplexing unit 50. The memory 30 includes, for example, a video area 32, an audio area 34, a setting area 36, and an attached information area 38. The video area 32 and the audio area 34 are the same as those in the above-described embodiment. In the setting area 36, for example, a GOP structure designated by an external host or the like is stored. The attached information area 38 stores header information and the like related to the encoded video data VDA and audio data ADA. The attached information area 38 may be divided into video data VDA and audio data ADA.

  For example, the data encoding unit 40 sequentially receives the video data VDA and the audio data ADA, and sequentially encodes the received video data VDA and audio data ADA by an encoding method compliant with MPEG-4 or the like. For example, the data encoding unit 40 includes an encoding control unit 42, a video encoding unit 44, and an audio encoding unit 46.

  The encoding control unit 42 acquires the GOP structure specified by the external host or the like from the setting area 36 of the memory 30. Then, the encoding control unit 42 reflects the acquired GOP structure and the like in the encoding processing of the video encoding unit 44 and the audio encoding unit 46. For example, the encoding control unit 42 stores header information related to encoding of the video data VDA in the attached information area 38 of the memory 30 in synchronization with the encoding of one video frame by the video encoding unit 44. The header information related to the encoding of the video data VDA is, for example, the GOP structure, the picture type, and the size of one video frame. Further, for example, when encoding of one video frame is completed, the encoding control unit 42 notifies the system control unit 26 of the system unit 22 that encoding of one video frame is completed.

  Also, the encoding control unit 42 stores, for example, header information related to encoding of the audio data ADA in the attached information area 38 of the memory 30 in synchronization with the encoding of one audio frame by the audio encoding unit 46. Furthermore, for example, when encoding of one audio frame is completed, the encoding control unit 42 notifies the system control unit 26 of the system unit 22 that encoding of one audio frame is completed.

  For example, the video encoding unit 44 sequentially receives the video data VDA, sequentially encodes the received video data VDA by an encoding method compliant with MPEG-4 or the like, and generates encoded video data VED (encoded video data VDA). ) Is generated. For example, the video encoding unit 44 encodes the video data VDA based on the GOP structure and the like acquired by the encoding control unit 42 from the setting area 36 of the memory 30. Then, the video encoding unit 44 stores the encoded video data VED in the video area 32 of the memory 30.

  For example, the audio encoding unit 46 sequentially receives the audio data ADA, sequentially encodes the received audio data ADA using an encoding method compliant with MPEG-4 or the like, and generates encoded audio data AED (encoded audio data ADA). ) Is generated. Then, the speech encoding unit 46 stores the encoded speech data AED in the speech area 34 of the memory 30.

  For example, the system unit 22 controls the multiplexing unit 50 and generates a message MES. For example, the system unit 22 includes a chunk determination unit 24, a system control unit 26, and a message generation unit 28. When the system control unit 26 receives from the encoding control unit 42 a notification of completion of encoding of one video frame, the system control unit 26 acquires header information related to encoding of the video data VDA from the attached information area 38 of the memory 30. That is, in response to the completion of encoding of one video frame, the system control unit 26 acquires the GOP structure, picture type, size of one video frame, and the like. Then, for example, the system control unit 26 notifies the chunk determination unit 24 and the message generation unit 28 of header information (GOP structure, picture type, size of one video frame, etc.) related to the encoding of the video data VDA.

  Further, when the system control unit 26 receives a notification of completion of encoding of one audio frame from the encoding control unit 42, the system control unit 26 acquires header information related to encoding of the audio data ADA from the attached information area 38 of the memory 30. Then, for example, the system control unit 26 notifies the chunk determination unit 24 and the message generation unit 28 of header information related to encoding of the audio data ADA.

  The chunk determination unit 24 monitors the occupation amount of the video area 32 and the occupation amount of the audio area 34. Then, when the free capacity of at least one of the video area 32 and the audio area 34 is small, the chunk determination unit 24, as described with reference to FIGS. 6 to 9, based on the GOP structure, picture type, and the like, Set the number.

  For example, the chunk determination unit 24 sets the number of chunk frames based on the header information regarding the encoding of the video data VDA and the header information regarding the encoding of the audio data ADA received from the system control unit 26. Then, the chunk determination unit 24 notifies, for example, information on the number of frames of the set chunk (hereinafter also referred to as chunk setting notification) to the multiplexing control unit 52 of the multiplexing unit 50 via the system control unit 26.

  For example, the message generation unit 28 generates a message MES corresponding to a chunk based on information received from the system control unit 26 (GOP structure, picture type, size of one video frame, and the like). Then, the message generation unit 28 outputs the generated message MES to the outside (for example, the external device 100 illustrated in FIG. 3) in synchronization with the output of the corresponding chunk.

  The multiplexing unit 50 multiplexes the encoded video data VED and the encoded audio data AED stored in the video area 32 and the audio area 34, respectively, and outputs them to the outside (for example, the external device 100 shown in FIG. 3). . For example, the multiplexing unit 50 includes a multiplexing control unit 52, an input unit 54, a multiplexing buffer 56, an encryption unit 58, and an output buffer 60.

  For example, the multiplexing control unit 52 controls the input unit 54 and the like based on information from the system unit 22. For example, the multiplexing control unit 52 activates the input unit 54 in response to the chunk setting notification received from the system control unit 26. For example, the input unit 54 alternately performs a process of reading the encoded video data VED for one chunk from the video area 32 and a process of reading the encoded audio data AED for one chunk from the audio area 34. The encoded video data VED and the encoded audio data AED read from the video area 32 and the audio area 34 are sequentially output to the multiplexing buffer 56.

  The multiplexing buffer 56 sequentially outputs the encoded video data VED and the encoded audio data AED received from the input unit 54 to the encryption unit 58. Thus, the encryption unit 58 alternately receives one chunk of encoded video data VED and one chunk of encoded audio data AED. The encryption unit 58 sequentially encrypts the encoded video data VED and the encoded audio data AED received from the multiplexing buffer 56. The encryption method and the like are specified in advance by the system unit 22, for example. For example, the encryption unit 58 receives information on the encryption method and the like from the system control unit 26 of the system unit 22 via the multiplexing control unit 52.

  The encoded video data VED and encoded audio data AED encrypted by the encryption unit 58 are sequentially output to the output buffer 60. That is, the output buffer 60 alternately receives the encrypted video data VED for one chunk and the encoded audio data AED for one chunk. The output buffer 60 alternately outputs encrypted video data VED for one chunk and encrypted audio data AED for one chunk.

  That is, the output buffer 60 sends the stream data SDA obtained by multiplexing the encoded video data VED for one chunk and the encoded audio data AED for one chunk to the outside (for example, the external device 100 shown in FIG. 3). Output. For example, the external device that receives the stream data SDA and the message MES stores the stream data SDA and the message MES in an MP4 file, and generates an MP4 file.

  In this way, the video / audio data processing device 12 functions as an encoder conforming to MPEG-4 or the like. The configuration of the video / audio data processing device 12 is not limited to this example. For example, the video / audio data processing device 12 may have a function of decoding encoded video data and audio data. In other words, the video / audio data processing device 12 may be formed to function as a codec or a transcoder.

  FIG. 11 shows an example of the operation of the video / audio data processing device 12 shown in FIG. The operation of FIG. 11 may be realized only by hardware, or may be realized by controlling a hard wafer by software.

  In process S <b> 100, the data encoding unit 40 encodes input data (video data VDA and audio data ADA) and writes the encoded input data into the memory 30. For example, the video encoding unit 44 of the data encoding unit 40 writes the encoded video data VED generated by encoding the video data VDA into the video area 32 of the memory 30. Further, for example, the voice encoding unit 46 of the data encoding unit 40 writes the encoded voice data AED generated by encoding the voice data ADA in the voice area 34 of the memory 30.

  In the process S110, the chunk confirmation unit 24 determines whether the occupation amounts of the video area 32 and the audio area 34 are equal to or greater than a threshold value. For example, the chunk determination unit 24 calculates the occupation amount of the video area 32 using the size information of one video frame received from the system control unit 26 (encoded frame size information). Note that the chunk determination unit 24 may calculate the occupation amount of the video area 32 and the audio area 34 based on the write address and the read address for the video area 32 and the audio area 34.

  Then, the chunk determination unit 24 compares the calculated occupation amount with a threshold value of the video area 32 (hereinafter also referred to as a first threshold value). The first threshold is set in advance based on the size of the video area 32, for example. Note that the comparison between the occupation amount of the audio area 34 and the threshold value (hereinafter also referred to as a second threshold value) is performed in the same manner as the comparison between the occupation amount of the video area 32 and the first threshold value. The second threshold is set in advance based on the size of the audio area 34, for example.

  When at least one of the first condition in which the occupation amount of the video area 32 is equal to or greater than the first threshold and the second condition in which the occupation amount of the audio area 34 is equal to or greater than the second threshold is satisfied (Yes in step S110), the chunk determination unit 24 The operation proceeds to step S130. That is, when the free space in the video area 32 and the audio area 34 is small, the operation of the chunk determination unit 24 proceeds to processing S130. When neither the first condition nor the second condition is satisfied (No in process S110), the operation of the chunk confirmation unit 24 proceeds to process S120. That is, when the free space in the video area 32 and the audio area 34 is large, the operation of the chunk determination unit 24 proceeds to processing S120.

  In the process S120, the chunk determination unit 24 sets the number of chunk frames to the same number as the number of frames of 1 GOP. For example, the information on the number of chunk frames (chunk setting notification) set in step S120 is notified to the multiplexing unit 50. As described above, when the free space in the video area 32 and the audio area 34 is large, the number of frames of the chunk is set to the same number as the number of frames of 1 GOP. On the other hand, when the free space in the video area 32 and the audio area 34 is small, the number of chunk frames is set to a number smaller than the number of frames in 1 GOP in steps S130 and S140.

  In the process S130, the chunk determination unit 24, for example, based on information received from the system control unit 26 (GOP structure, picture type, size of one video frame, etc.), the number of frames per chunk (“j”) To decide. In the process S140, the chunk confirmation unit 24 sets the number of frames of the chunk to the number of frames (“j”) determined in the process S130. For example, the chunk determination unit 24 notifies the multiplexing unit 50 of information on the number of chunk frames determined in step S130 (chunk setting notification).

  In the process S150, for example, the multiplexing unit 50 sequentially outputs encoded data (encoded video data VED or encoded audio data AED) based on the chunk set in the processes S120 and S140. For example, the system control unit 26 determines whether or not output of encoded data for one chunk has been completed. When the output of the encoded data for one chunk is completed (Yes in process S150), in process S180, the message generator 28 outputs a message MES corresponding to the output chunk. On the other hand, when the output of the encoded data for one chunk has not been completed (No in process S150), the operation of the video / audio data processing device 12 proceeds to process S160.

  In the process S160, for example, the system control unit 26 determines whether any of the video area 32 and the audio area 34 of the memory 30 has overflowed. The determination of overflow may be performed by a module other than the system control unit 26 (for example, the chunk determination unit 24). When either the video area 32 or the audio area 34 of the memory 30 overflows (Yes in process S160), the data in the overflowed area (video area 32 or audio area 34) is deleted in process S170. On the other hand, when the memory 30 has not overflowed (No in process S160), the operation of the video / audio data processing device 12 returns to process S150.

  As described above, the video / audio data processing device 12 makes the number of chunk frames smaller than the number of GOP frames when the free capacity of at least one of the video area 32 and the audio area 34 is small. The condition for the number of frames in the chunk is the same as that in the above-described embodiment. The operation of the video / audio data processing device 12 is not limited to this example. For example, the chunk determination unit 24 may change the number of chunk frames step by step according to the free space in the video area 32 and the audio area 34.

  As described above, also in this embodiment, the same effect as that of the above-described embodiment can be obtained.

  From the above detailed description, features and advantages of the embodiments will become apparent. This is intended to cover the features and advantages of the embodiments described above without departing from the spirit and scope of the claims. Further, any person having ordinary knowledge in the technical field should be able to easily come up with any improvements and modifications, and there is no intention to limit the scope of the embodiments having the invention to those described above. It is also possible to rely on suitable improvements and equivalents within the scope disclosed in.

  DESCRIPTION OF SYMBOLS 10, 12 ... Video / audio data processing device; 20, 24 Chunk determination part; 22 ... System part; 26 ... System control part; 28 ... Message generation part; 30 ... Memory; 32 ... Video area; 36 ... Setting area; 38 ... Attached information area; 40 ... Data encoding part; 42 ... Encoding control part; 44 ... Video encoding part; 46 ... Audio encoding part; 50 ... Multiplexing part; 54. Input unit 56 56 Multiplexing buffer 58 Encrypting unit 60 Output buffer 100 External device

Claims (5)

A first storage unit for storing encoded video data;
A second storage unit for storing encoded audio data;
The occupation amounts of the first storage unit and the second storage unit are monitored, and the first condition that the occupation amount of the first storage unit is a predetermined first threshold or more and the occupation amount of the second storage unit are A chunk determination unit that reduces the number of frames of a chunk to be smaller than the number of frames of a GOP (Group Of Picture) when at least one of the second conditions equal to or greater than a predetermined second threshold is satisfied. Audio data processing device. The chunk determination unit sets the number of frames of the chunk so that the order of the frames can be changed within the chunk to a display order when the video data is reproduced. Video encoding device. The chunk determination unit sets the number of frames of the chunk to a minimum value of 2 or more out of the number of frames in which the order of the frames can be changed in the chunk as a display order when reproducing the video data. The video / audio data processing apparatus according to claim 2. At least one of the first threshold and the second threshold has a plurality of values,
The video / audio data processing device according to claim 1, wherein the chunk determination unit changes the number of frames of the chunk in a stepwise manner in accordance with the occupation amount compared with the plurality of values. In a data multiplexing method for multiplexing encoded video data and encoded audio data,
Comparing the occupation amount of the first storage unit for storing the encoded video data with a predetermined first threshold;
Comparing the occupation amount of the second storage unit for storing the encoded audio data with a predetermined second threshold;
Number of chunk frames when the first storage unit occupies at least one of the first condition and the second storage unit occupy at least one of the second threshold and the second condition. The data multiplexing method is characterized in that the number of frames is less than the number of GOP (Group Of Picture) frames.

Images