JP2005223617A - Data generation device and method, and program and computer readable storage medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a data generation device capable of efficiently reducing the size of a data storing file while maintaining an accurate time stamp which is not visually deteriorated in inputting the multimedia data of video images, sound, etc. <P>SOLUTION: The time stamp is read from an encrypted data and a highly accurate time scale for providing second clock accuracy is calculated on the basis of the frame rate of the encrypted data and first clock accuracy relative to the time stamp. A first offset which is the logical offset of the time stamp between two frames in the encrypted data and a second offset which is the logical offset of a highly accurate time stamp between two frames in the highly accurate time scale are calculated, the time stamp of the encrypted data is converted into the highly accurate time stamp on the basis of correspondence relation between the first and second offsets and a time stamp table is generated by using the highly accurate time stamp. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to so-called multimedia data having a time axis, that is, data including audio and moving images, or a similar multimedia coding system having a data representation mechanism based on MPEG-4 and a similar coding format ( Hereinafter, the data generation apparatus is simply MPEG-4).

  In recent years, with the spread of digital still cameras, digital video cameras, mobile phones, and the like, so-called multimedia that has been mainly used on computers has become easy to use on various devices. With the spread of such multimedia, various recording formats such as MPEG-4 standardized by ISO / IEC are used.

  For example, this MPEG-4 is standardized as ISO / IEC 14496, but in general, it is also possible to compress moving images, which tend to be huge, to a smaller recording format, and to cope with resource limitations of communication and recording devices. It has become one of Therefore, in MPEG-4, variable frame rate encoding can be performed in order to perform higher-quality compression.

  On the other hand, in order to cope with such encoding, the file format for storing the encoded data has become more complicated. With a fixed frame rate, it was possible to specify the time to play back all the frames by simply recording one frame rate for each frame. The fact that we have to record all the time we should do is a straightforward example.

  The time that each frame should be reproduced is generally called a time stamp. However, if all the time stamps are recorded in a file or the like, a large storage area is actually consumed. For example, if a 16-bit wide storage area normally used in a microcomputer is used for time stamp recording, recording can be performed only at most about 1 minute in a unit of 1 millisecond minimum. This problem is greatly improved by using a 32-bit storage area, but instead, a storage area of 4 bytes per frame is required, so an image of about 30 frames per second can be recorded for 1 hour. 432KB (kilobytes) will be consumed.

  Since such a problem has been known more than before, the MP4 of the standard file format in MPEG-4 has been devised to record the time difference between frames, that is, the time stamp difference. The difference in time stamps is theoretically a constant value, especially at a fixed frame rate, and there is a possibility that the storage area can be greatly reduced by recording the number of consecutive time stamp differences. is there. Such a method is called continuous length compression (run-length compression) from the viewpoint of data compression, and is excellent as a simple and light load method for time-series compression.

  In order to examine the effect of such a method, a simple example may be considered. For example, if the frame rate is 10 frames per second (referred to as 10 fps or 10 Hz) and the time stamp is obtained in units of 1 millisecond, the time stamp of each frame is in order from the top, The difference between time stamps is always 100, such as 0, 100, 200, 300. If the data is 10 seconds, there are 100 frames as a whole. Therefore, the number of differences is 99, and it can be sufficiently expressed by recording that there are 99 time stamp differences of 100 milliseconds.

  As an example, if 4 bytes of storage is applied to all data, the storage required to store all the time stamps is 400 bytes, whereas in the case of run length compression, it is only 8 bytes. Absent.

  As described above, the method of recording the time stamp difference by continuous length compression reliably records all time stamp differences in the case of the variable frame rate, while the storage area in the case of the fixed frame rate. Can be significantly compressed.

  By the way, in some recording formats including the standard file format MP4 in MPEG-4, the minimum unit of time stamp is expressed by the concept of time scale.

  The time scale is expressed by the number of divisions when a predetermined time unit is divided by an arbitrary number to set the divided time length as the minimum unit of time. Normally, this arbitrary time unit is 1 second. For example, a time scale of 1000 means that 1 millisecond obtained by dividing 1 second into 1000 is the minimum unit.

  In the following description, unless otherwise specified, it is assumed that this predetermined arbitrary time unit which is a reference in the time scale is 1 second for the sake of simplicity. In addition, a difference between time stamps may be expressed as a time stamp offset.

  When this time scale concept is used, time stamps and time stamp differences are expressed by this time scale. If the time scale is 1000 and the time stamp difference is 100, the time stamp difference with a minimum unit of 1 millisecond means 100 milliseconds. On the other hand, if the time scale is 500, which is half, 2 milliseconds obtained by dividing 1 second into 500 is the minimum unit. Therefore, if the time stamp difference is 100, the time stamp difference having a minimum unit of 2 milliseconds means 200 milliseconds.

Such a device makes it possible to use an arbitrary unit of time for the description of time stamps and the like, and it enables more flexible and highly accurate recording.
The simplest example is when the time scale is 10 when the frame rate is 10 fps. That is, when there are 10 frames of data per second, the difference in time stamps can be set to 1 by recording in units of time obtained by dividing 1 second into 10 units. This means that ideally a very accurate time stamp can be recorded.

  Although the time stamp recording format corresponds to the variable frame rate described above, particularly in the case of the variable frame rate, the problem remains that the time stamp recording becomes large depending on the encoded data. ing. Since the difference between time stamps (time stamp offset) varies depending on the encoded data, if the difference between time stamps (time stamp offset) does not happen to be the same in succession, the effect of continuous length compression will be effective. This is because it does not appear.

In order to cope with such a problem, for example, Japanese Patent Laid-Open No. 2002-238050 proposes a method of inserting an empty frame as if a certain time stamp has occurred. In other words, this method is a data generation apparatus having a conversion mechanism from a variable frame rate to a fixed frame rate, in which encoded data at a variable frame rate is expressed by a logical time stamp at a fixed frame rate. is there.
A time stamp that does not actually have a frame and is generated by converting to a fixed frame rate can be used as a significant time stamp by storing the encoded data as encoded data having information indicating non-update. Alternatively, by controlling to record as a frame of size 0, it can behave as if it is a fixed frame rate.

  Also, according to Japanese Patent Laid-Open No. 2003-143550, as a countermeasure for a case where an accurate time stamp cannot be given even at a fixed frame rate, depending on the accuracy of an operating system (OS) or a clock (clock) device. There is a method of replacing a time stamp difference between adjacent frames with a logical time stamp when the difference is within a certain error range.

As described above, according to the conventional technology, it is possible to perform recording with storing a time stamp in a smaller storage area while dealing with an error of a variable frame rate and a clock device by various devices. There is an advantage.
JP 2002-238050 A JP 2003-143550 A

  However, even if the fixed frame rate is still generally used, the difference between the time stamps (timestamp offset) is not always constant depending on the accuracy of the actual time stamp and the logical time stamp. In such a case, there is a problem that the effect of continuous length compression is not sufficiently exhibited.

  More specifically, for example, when each frame is generated at a frame rate set in advance as 15 fps, the time stamp of the frame may not necessarily depend on the accuracy of the operating system (OS) or clock (clock) device that generates the frame. it's not correct. For example, if the time scale in units of time processed by an operating system (OS) or clock device (referred to as system time scale for convenience) is 1000, ie 1 millisecond, 1 Since 15 frames are processed per second, a time stamp occurs approximately every 66.7 milliseconds, which is 1000 milliseconds divided by 15. This approximately 66.7 milliseconds requires a higher accuracy than the system time scale accuracy. Regardless of whether the time stamp is generated at the time of capturing the frame data or is calculated at the time of encoding the frame data, if this about 66.7 milliseconds is used as a reference, Obviously, errors will occur.

  Such an error can be solved by setting the time scale to 15 which is equal to the frame rate according to this example, as described in the prior art. Is inconvenient when processed on the system time scale.

As described above, according to the conventional method, in the time stamp storage method using continuous length compression such as MP4, the time stamp storage table is enlarged depending on the accuracy of the time stamp, and the storage efficiency is lowered. There is a problem.
Also, especially in the process of sequentially writing frames to a file in real time, if the time stamp storage table stored in the temporary storage memory is enlarged, the memory usage will also be reduced in devices with restrictions that have only a small memory. The problem of enlargement also occurs. As a result, the unexpected enlargement of the temporary storage memory may lead to unexpected memory pressure on the processing equipment itself, and may cause a serious processing abnormality.

  The present invention has been made to solve such a problem. When inputting multimedia data such as video and audio, the data storage file size is maintained while maintaining an accurate time stamp without visual deterioration. It is an object of the present invention to provide a data generation device that can efficiently reduce the size of the data.

  In order to solve the above problems, a data generation apparatus according to the present invention includes a storage unit that stores encoded data, a time stamp reading unit that reads a time stamp from the encoded data, and a frame rate of the encoded data. Time scale calculating means for calculating a high-accuracy time scale for providing a second clock accuracy based on the first clock accuracy related to the time stamp, and the time between two frames in the encoded data Offset calculating means for calculating a first offset that is a theoretical offset of a stamp, and a second offset that is a theoretical offset of a high-accuracy time stamp between two frames in the high-accuracy time scale; 2 based on the correspondence relationship of the offset of 2, the time stamp of the encoded data is A time stamp converter for converting the precision timestamp, characterized in that it comprises a time stamp table generating means for generating a time stamp table, and using the high precision time stamp.

  Further features of the present invention will become apparent from the embodiments described below and the accompanying drawings.

  According to the present invention, it is possible to generate a high-accuracy time stamp that is logically free of errors, and the storage efficiency of the time stamp storage table can be improved while maintaining an accurate time stamp that is not visually deteriorated. The file size of the stored file can be effectively reduced.

  As a result, even in a restricted device having only a small memory, it is possible to reduce the possibility of inducing a serious processing abnormality due to memory compression.

<First Embodiment>
Hereinafter, a first embodiment according to the present invention will be described with reference to FIGS.
First, an example of a multimedia data multiplexing apparatus for recording moving images / sounds suitable for the embodiment will be described. FIG. 1 is an example of a block diagram showing an outline of such a multimedia data multiplexing apparatus.

The multimedia data multiplexing apparatus shown in the block diagram includes an imaging apparatus 101, an A / D conversion apparatus 102, an imaging processing apparatus (encoder) 103, a clock (clock) apparatus 104, a CPU 105, a data conversion apparatus 106, a temporary storage apparatus ( Memory) 107, memory / bus controller device 108, recording medium I / F device 109, and recording medium 110.
According to this multimedia data multiplexing device, it is generally obtained by analog / digital conversion of an image photographed by the photographing device 101 such as a photographing lens, a shutter, and an imaging device by the A / D conversion device 102. By appropriately processing a digital image, it can be recorded as multimedia data in which multimedia data is multiplexed.

  The image data digitized by the A / D conversion device 102 is subjected to, for example, MPEG-4 video encoding processing in the image pickup processing device (encoder) 103 under appropriate control from the CPU 105, and becomes data according to the present invention. The data is sequentially transferred to the conversion device 106. The time control at this time is determined by a timing signal generated by the clock device 104, and a so-called time stamp is given to the multimedia data.

  At this time, the frame rate is the number of frames processed per second by the imaging processor (encoder) 103.

  The temporary storage device (memory) 107 controlled by the memory / bus controller device 108 is used not only for temporarily storing a part of the encoded data, but also for the time stamp and the processing used in the processing of the present invention. It is also used for temporary storage of tables. Similarly, when the result of the multiplexing process of multimedia data is written as a data file by performing an encoding process, the recording medium I / F device 109 controlled by the memory / bus controller device 108 is also used. Through the recording medium 110.

  The essence of the present invention resides in the data converter 106 in this block diagram.

  The block diagram of the multimedia data multiplexing apparatus shown in FIG. 1 is described as an example having a photographing apparatus having a lens such as a digital camera, but a stationary recording apparatus such as a video recorder, for example. It may be something like this. Even in the case of a digital camera, in this block diagram, related parts such as an operation unit and a mechanical drive unit that are not necessary for explanation are omitted.

  Next, detailed description will be given using such a multimedia data multiplexing apparatus, particularly using a case where a moving image is recorded.

  In general, when a moving image is generated at a so-called fixed frame rate at which the frame rate of the moving image does not change dynamically, it is assumed that the frame rate is set in advance and the recording apparatus can know this. Good.

Here, for convenience of explanation, it is assumed that the time scale processed by the operating system (OS) or clock (clock) device in which the recording apparatus operates is 1000. Such a time scale on the apparatus is referred to herein as a system time scale.
Note that, as described in the description of the prior art, an arbitrary predetermined time unit serving as a reference on the time scale is assumed to be a general one second.

  In the embodiment described here, first, a high-accuracy system time scale different from the actual minimum time unit is obtained by calculating the least common multiple of the frame rate and the system time scale. This calculated high accuracy system time scale may in some cases give smaller time units than the actual system time scale. Details of these will be explained in turn.

  Any method may be used for the calculation of the least common multiple. For example, as a generally well-known method, there is a method of calculating the least common multiple after obtaining the greatest common divisor by the Euclidean algorithm. Since the Euclidean algorithm is a widely known algorithm, a detailed description thereof is omitted here. However, simply speaking, a remainder of two numbers is obtained and the result is recursively calculated to be 0. The greatest common divisor can be obtained by repeating the above. The least common multiple is the result of dividing the product of two by this greatest common divisor.

  A description will be given with a more specific example. For example, if the frame rate is 15 frames / second (fps), the system time scale is 1000, so the least common multiple is 3000. This 3000 is the calculated high-accuracy system time scale, and is hereinafter referred to as the high-accuracy system time scale.

  In practice, the high precision system time scale is not necessarily the least common multiple, and the present invention operates effectively even with an integral multiple thereof. This point is clear from the essence of the present invention as described later.

  Also, if the least common multiple is the same as the frame rate or system time scale, the frame rate or system time scale can be used as it is. This will be described later.

  Next, a theoretical time stamp offset value (difference) between time stamps of each frame of the moving image is calculated from this high-accuracy system time scale. That is, it is a value expressing the time difference between frames using a time scale.

  If calculating directly from the system time scale and the frame rate, the system time scale is 1000 and the frame rate is 15, so the theoretical stamp offset is calculated from 1000 ÷ 15 to about 66.7. . The system time scale value of 1000 means that the minimum unit of time is 1 millisecond, and therefore corresponds to about 66.7 milliseconds.

  On the other hand, since the high-accuracy system time scale is calculated as 3000 from the least common multiple, the timestamp offset value (difference) in this case is calculated as 3000 ÷ 15 to 200.

It should be noted that the calculated number of 200 corresponds to about 66.7 milliseconds when converted to real time because the high-accuracy system time scale is 3000.
This time stamp offset is schematically shown in FIG.

  A rectangular frame 201 illustrated three-dimensionally in the center of FIG. 2 schematically shows each frame (sample) of a moving image. Each frame is arranged in order at a frame rate 202 of 15 fps. Above each frame 201, the original system time scale 203 is shown with a time axis, and below the high accuracy system time scale 204 is shown with a time axis.

  As explained above, the original timestamp offset value (difference) 205 is the time between frames represented by the original system time scale 203 and is approximately 66.7. The theoretical time stamp offset value (difference) 206 is a time between frames represented by the high-precision system time scale 204 and is 200.

  It can be seen that the original time stamp offset value (difference) 205 and the theoretical time stamp offset value (difference) 206 are different in value, but are logically equivalent.

In the calculation so far, a high precision system time scale value of 3000 and a theoretical time stamp offset value (difference) of 200 units were obtained.
The accuracy ratio of the high-accuracy system time scale to the actual system time scale is three times 3000/1000.

  Next, the error rate is calculated. The time processed by the operating system (OS) or clock device includes an error of up to 1 millisecond according to the system time scale.

  Such an error is, for example, a fraction less than the smallest unit of time that occurs when an encoder module that generates each frame of a moving image (sometimes called a sample) tries to operate at an absolute frame rate. Generated by truncating or rounding.

  There are various other factors, but at least the minimum unit of error is equal to or greater than the minimum unit of the system time scale. In the high precision system time scale, 3 units (3 x minimum unit) It is clear that this is the case.

  Normally, the time stamp given to each frame (sample) is given in the smallest time unit according to the system time scale, so that even if the encoder gives it with reference to this, the output result is logical. Even if given, the time stamp is considered to contain a certain error. For example, if the time is rounded down with a precision less than the minimum unit, the time stamp is set to 0 milliseconds, 66 milliseconds, 132 milliseconds, 199 milliseconds, 266 milliseconds, and so on.

  Apparently, the offset value (difference) of the actually acquired time stamp is not a constant value such as 66 milliseconds, 66 milliseconds, 67 milliseconds, and 67 milliseconds. If the offset value (difference) obtained in this way is used for recording moving image data, it is obvious that the header area for storing the offset value naturally increases.

  Similarly, when the time stamp offset is converted to the high precision system time scale using the high precision system time scale factor, these are in turn 198, 198, 201, 201 and still contain errors. Yes.

  In the present invention, if the offset value (difference) of the time stamp given in this high-accuracy system time scale is within the previously calculated error (3 units), the logical time calculated in the same way is used. 200 units that are stamp offset values (difference) are always used.

  At first glance, this seems to be equivalent for both the high-accuracy system time scale and the original system time scale, but in fact, the original system time scale has a theoretical time stamp. This offset value (difference) requires accuracy below the minimum time and cannot be used.

  As described above, by using the present invention, it is possible to always set a time stamp offset value (difference) constant.

  In calculating the time stamp offset value (difference) converted from the actual time stamp to the high precision system time scale, the time stamp is converted to the high precision system time scale before the offset value ( It is mathematically clear that the result is equivalent even if the difference is calculated.

  In addition, although the algorithms are sequentially described here, there is no difference in the basic concept of the algorithm even if the calculation order and value are stored in order to minimize the rounding error in calculation.

  Here, an example will be described in which the offset of the time stamp as a result of the actual processing is applied to the so-called MP4 format, which is the MPEG-4 standard file format.

  As described in the prior art, according to the MPEG-4 standard provided by ISO / IEC 14496, the time stamp such as the moving image data actually exemplified here is called a track for storing any moving image data. Each unit is stored as a file with the following definition.

This data structure consists of “entry-count” that indicates the number of elements in the offset table of the stored timestamp, and the number of frames (samples) indicated by the elements of the table that are prepared for that number of elements and the time It consists of stamp offset value (difference) “sample-delta”.
In order to clarify this data structure, FIG. 3 schematically shows the storage format of the time stamp as an example of so-called MP4 which is a standard of MPEG-4.

  In the case described above, the time stamp offset value (difference) given by the original system time scale 203, i.e., 66 milliseconds, 66 milliseconds, 67 milliseconds, 67 milliseconds, is used. The data structure to be stored is like a table 301 based on the original system time scale shown in FIG. The value of the element number 302 is 3, and in the first element, the frame number 303 is 2 and the offset value (difference) 304 is 66. Similarly, 3 and 67 are set in the second element. In the third element, 1 and 66 are set.

  On the other hand, according to the high-precision system time scale 204, the number of table elements 306 is 1 as in the table 305, the number of frames 307 is n, and the offset value (difference) 308 is 200. Can be reduced. For example, when the frame rate is 15 fps, n = 14.

  In this example, there is little difference in absolute size because the number of target frames is small, but due to a time stamp error that will occur periodically, the use area increases as the number of frames increases. It is clear that the effect of compression increases.

  This compression effect not only affects the file size, but is also effective in compressing the memory size to be used. This is because the program cannot write the table itself to the file until all data has been output.

  Note that if the high-accuracy system time scale is not stored on the MP4 file, the actual time stamp cannot be derived later. However, as a place to store such a value, “Media Header Atom” for each track is used. Is defined.

  The side that reads the data once stored in the MP4 file first obtains the time scale from this “Media Header Atom”, and sequentially converts the offset value (difference) of the time stamp from the time scale to the real time and takes it out. The correct value can be extracted. Since such data extraction is standardized by ISO / IEC 14496 and is not the essence of the present invention, the description thereof is omitted.

  Now, when the least common multiple is the same as the frame rate or system time scale, we have not explained so far, but as described above, set the frame rate or system time scale to the high accuracy, system time scale. May be.

  An example of this is when the system time scale is 1 millisecond and the frame rate is 10 fps. In such a case, the least common multiple happens to coincide with the system time scale and the like, and is not particularly inconsistent with the present invention.

  However, if the least common multiple is the same as the frame rate or system time scale, and if it is clear that the time stamp is provided without error, then the algorithm described here Regardless, the high-accuracy system time scale may be the frame rate value and the time stamp offset value (difference) may be 1.

  As an example of this, there exists a system in which a time stamp is always logically given by a calculated value.

  Next, additional explanation will be given for other error factors. In the examples so far, the error of the system time scale is assumed to be 1 millisecond. However, in an actual apparatus, an error element not directly related to the essence of the present invention, such as an encoder processing error, is included. there is a possibility.

  If there is such an error, in addition to the error of the system time scale, an error parameter caused by such an external may be taken into consideration.

  Furthermore, if there is a time stamp value that occurs exceptionally even at a fixed frame rate, it can be stored as an accurate time stamp by converting it to a high-accuracy system time scale. Is possible. Such a calculation can be dealt with by using the value as it is if the calculated offset value (difference) of the time stamp is not within the error range.

  Furthermore, in the present invention, if the recording apparatus of the present invention is necessary, the time stamp next to the time stamp in which the exception has occurred is offset from the time stamp immediately preceding the time stamp in which the exception occurred (difference). If the offset value to be calculated is a value obtained by subtracting the offset value in which an exception has occurred from the offset value as the theoretical value, the error can be further reduced.

  An actual example including such an exception is shown in FIG. FIG. 4 is a diagram showing, in a tabular form, a time stamp output by a certain encoder application and a time stamp based on a high-precision system time scale.

  In this figure, the actual time stamp 401 is arranged in time series on the left end. The actual time stamp 401 is not necessarily coincident with the actually obtained time stamp 402 shown in the second column, and is logical. Since this is a time that is more detailed than the minimum unit of time indicated by the system time scale, it is not physically obtained in reality. Since it is the minimum unit of time provided by the OS or the like, in reality, it is treated as a value where truncation has occurred, for example, as with the obtained time stamp 402.

  The third column is a time stamp offset value (difference) 403 as used in the MPEG-4 standard given by ISO / IEC 14496, and represents the difference between the time stamps of the obtained time stamp 402. . That is, it is equivalent to the offset value (difference) 304 in FIG.

  The fourth column shows a high-accuracy time stamp offset 404 converted on the high-accuracy system time scale. As described above, since this value is replaced with the theoretical value of the offset value (difference) of the time stamp within a certain error range, most of the value is 200.

  The two columns on the right are a high-accuracy time stamp 405 obtained by integrating the high-accuracy time stamp offset 404 and a reproduction time stamp 406 derived from this value. The time stamp 406 at the time of reproduction is calculated at the time of reading, and is rounded off to the second decimal place as an example when it is assumed that the system time scale of the application for reading is similarly 10000. It is a reference value.

  The part highlighted in italics in the figure is the part where the exception occurs. In this part, the obtained time stamp 402 does not match 15 fps which is the value of the frame rate 202. As is clear from the value of the high-accuracy time stamp offset 404, the error range is exceeded, so the theoretical value 200 of the time stamp offset value (difference) is not set.

  However, the time stamp immediately below shows the correct value again. In such a case, a comparison is made with the time stamp one above, and this is within the error range of the time stamp offset value (difference), so the time stamp offset value (difference) theory 65 is set as a value obtained by subtracting the offset value of the portion where the exception has occurred from the value 200.

As a result, the high-accuracy time stamp 405 of the entire figure faithfully reproduces the actual time stamp 401 for all values including exceptions.
Finally, the algorithm according to the present invention described so far will be described with reference to a flowchart.

  FIG. 5 and FIG. 6 describe the algorithm described so far with flowcharts. Actually, as explained earlier, the calculation order can be changed within the range of the algorithm concept in consideration of the rounding error and speed in the calculation. Listed in order.

  FIG. 5 shows a part for preparing various parameters to be performed as a preceding stage when actually recording a moving image / audio. The recording portion of the frame (sample) including the subsequent time stamp is shown in FIG. 6 by the page connector “A”.

  Step S501: First, initial information necessary for this algorithm, the system time scale 203 and the frame rate 202 are acquired. As explained above, for example, the system time scale is 1000 and the frame rate is 15. In this flowchart, for the sake of readability, the system time scale is indicated by STS and the frame rate is indicated by FPS.

Step S502: The least common multiple 501 is calculated from the acquired system time scale 203 and frame rate 202. For the calculation of the least common multiple, Euclidean mutual division or the like can be used. As a result, it may be a value obtained by multiplying the least common multiple by an integer. Again, for the sake of readability, the least common multiple is indicated by the symbol LCM.
Step S503: It is known in advance that the obtained least common multiple 501 matches the system time scale 203 or the frame rate 202, and the actually acquired frame (sample) is accompanied by an accurate time stamp according to the frame rate 202. If yes (Yes), the process proceeds to step S504. Otherwise, the process proceeds to step S506.

  Step S504: If the condition is Yes in step S503, the high-accuracy system time scale is set to the frame rate 202.

  Step S505: Subsequently, the time stamp offset value (difference) is set to 1.

  Step S506: If the condition is No in Step S503, the high precision system time scale is set to the calculated least common multiple 501.

  Step S507: Subsequently, the time stamp offset value (difference) is calculated as a theoretical value. This value is obtained by dividing the time difference between frames using the high-precision system time scale, and is obtained by dividing the high-precision system time scale set in step S506 above by the frame rate 202. Result.

  Step S508: Further, an error due to the high accuracy system time scale is calculated. Since the error is at least one unit in the original system time scale 203, the error is obtained as a quotient obtained by dividing the high accuracy system time scale 204 by the original system time scale 203. If there is an external setting error rate 502 known in advance, this is added.

  Step S509: Here, the high-precision system time scale 204, the time stamp offset value (difference) 206, and the error 503 are stored as parameters calculated by the present algorithm. If the error is not calculated according to the condition in step S503, 0 may be stored as the error 503. This value is not used if it is not necessary in the subsequent processing.

  Next, FIG. 6 will be described. FIG. 6 is a flowchart for explaining a recording portion of a frame (sample) including a time stamp at the subsequent stage following FIG. Here, while sequentially acquiring each frame (sample) data, the table 603 is output in accordance with the algorithm of the present invention in a time stamp storage format similar to the MPEG-4 standard provided by ISO / IEC 14496. . A table 603 is a representation on a memory corresponding to the table 305 shown in FIG. Of course, the expression “on memory” is used here for the sake of convenience, but it may actually exist on an external storage device such as an intermediate file.

  Step S601: First, the time stamp 601 and frame data 602 of each frame are acquired. Although the frame data 602 is not directly related to the essence of the present invention, it is shown as a form of a general application for reference.

  Step S602: It is determined whether it is the end of the frame. If there is no input in step S601, the process proceeds to an end process in step S610.

  Step S603: Conversely, if the first frame, that is, the frame is acquired for the first time, the process proceeds to Step S604. Otherwise, the process proceeds to step S605.

  Step S604: If it is the first frame, there is no target for calculating the offset of the time stamp, so the obtained time stamp 601 is stored in the temporary storage area, and the process returns to the acquisition of the frame in step S601 again.

  Step S605: If the previous time stamp has already been acquired, the time stamp offset value is calculated by subtracting the previous time stamp from the current time stamp. At this time, the offset value is converted in accordance with the high-accuracy system time scale calculated in the flowchart of FIG. Here, in order to simplify the flowchart, such details are omitted, but the processing of this part is as described in detail previously.

  Step S606: Calculate the difference between the high-accuracy time stamp offset value converted to the high-precision system time scale calculated in step S605 and the time stamp offset value (difference) 206 calculated in the flowchart of FIG. Then, it is determined whether or not this exceeds the range of the error 503 calculated in the flowchart of FIG. If it is larger than the error, the process proceeds to step S607, and if not, the process proceeds to step S608.

  Step S607: Recalculate the offset value. In order to simplify the flowchart, this process is described as a predefined process. This process is the exception process described with reference to FIG. Here, the description will be omitted because it is redundant.

  Step S608: If within the error range, the calculated time stamp offset value (difference) is replaced with the time stamp offset value (difference) 206 calculated in the flowchart of FIG.

  Step S609: At the end of the processing of each frame, the resulting time stamp offset value (difference) is stored in the table 603. Needless to say, if the time stamp offset value (difference) is the same value as the last element of the table 603, the process is only to increase the number of frames of the last element by one.

  Step S610: Here, the end process is shown symbolically, but specifically, for example, the data of the table 603 is stored in the final file.

  As described above, when the present invention is applied to an encoder output result storage application including a time stamp, it is clear that the time stamp storage area can be reduced without losing the accuracy of the time stamp. is there.

<Second Embodiment>
In the embodiments described so far, it has been assumed that the given frame rate is an integer. However, for example, the case where the frame rate is 7.5 frames / second is not impossible. Even in such a case, the present invention can be easily applied by a slight pretreatment.

  For such a frame rate that is not an integer, the frame rate is converted into an integer as preprocessing. The term “integerization” as used herein refers to processing for converting the result obtained by multiplying the frame rate by an integer to an integer. For example, the frame rate of 7.5 frames / second can be converted to 15 by multiplying it by two.

  There are several methods for converting to an integer, but the simplest method is realized by multiplying the number of decimal places by 10 times. When the decimal part is divisible by 2 or 5, the number of decimals × 10 of the decimal part may be similarly divided by 2 or 5 when the decimal part is divisible. In this example, since the decimal part 5 of 7.5 is divisible by 5, it can be seen that it is sufficient to double the number of digits of the decimal part × 10 ÷ 5.

  This calculated integerized frame rate can be used in the algorithm in the first embodiment. In this example, since 15 is calculated as a result, processing equivalent to that in the first embodiment is performed. However, unlike the case of 15 fps, the time stamp output from the encoder is 0, 133, 266, 400, 533 (milliseconds).

  As a result, since it is the same as the case of 15 fps, if the system time scale is 1000, the least common multiple is 3000, which is used as the high-definition system time scale.

  It should be noted that the result of calculating the theoretical time stamp offset is different from the case of 15 fps. The high-definition system timescale is 3000, but the frame rate is 7.5 instead of 15, so the calculated result is 3000 / 7.5 to 400.

  Such a difference is not an algorithm difference, but a difference in the calculated result. That is, when the frame rate includes a fractional part, the present invention can be easily applied by adding a pre-process for calculating an integerized frame rate by multiplying an arbitrary integer that is an integer.

<Third Embodiment>
In the case where the frame rate is considered to be fixed, the present invention may be applicable even if the operation is a variable frame rate operation. The variable frame rate operation referred to here is a case where a plurality of predetermined frame rates are dynamically used even though they are not completely fixed frame rates.

  As an example of this, for example, there is a case where the frame rate is normally 10 fps, but the scene is changed to 15 fps due to a large difference between frames in a scene with fast motion. Alternatively, when it is desired to keep the bit rate of a stream such as a moving image constant due to communication restrictions, on the contrary, in a scene with fast movement, the compression rate is lowered to maintain the image quality, and a frame rate of 15 fps is usually set. Changing to 10 fps is also used.

  When such a frame rate setting is known in advance, the present invention is sufficiently applicable by calculating the frame rate 202 shown in the flowchart of FIG. 5 as a plurality of values.

  In this case, the calculation of the least common multiple in step S <b> 502 may be performed at a plurality of frame rates 202 and the system time scale 203. If the least common multiple is calculated at two frame rates of 10 fps and 15 fps and a system time scale value of 1000, a result of 3000 is obtained. At this time, the theoretical values 206 of the time stamp offsets are 300 and 200, respectively. That is, there are a plurality of theoretical values 206 of time stamp offsets.

  It is well known that the method of obtaining the least common multiple from three or more values can also apply the Euclidean mutual division method. Therefore, exactly the same processing can be applied except that the input frame rate 202 and the output time stamp offset theoretical value 206 become plural.

  After that, in the flowchart of FIG. 6, it is only necessary to change the theoretical value of the offset of the applied time stamp while determining at which frame rate the frame to which the time stamp 601 is applied is output.

  Since the frame rate is changed in the middle, the compression effect of the time stamp storage area is smaller than when using a single time stamp, but if the number of frame rate changes is sufficiently small, the effect is still It is sufficiently significant.

<Fourth Embodiment>
Here, other embodiments that could not be described so far will be described.
The present invention is also effective when a plurality of different frame rates such as moving images and sounds are used.

  If only a single time scale can be stored in the same file, the least common multiple is calculated from one frame rate and system time scale, and if a high precision system time scale is calculated, another frame is calculated. There is a possibility to sacrifice the accuracy of the rate. For example, if the frame rate of a moving image is 10 fps and the system time scale is 1000, the resulting high accuracy system time scale is 1000 according to the present algorithm. However, if the frame rate of another moving image is 15 fps, if this is processed with a high-accuracy system time scale value of 1000, it is apparent that the effects of both time stamp accuracy and storage area cannot be achieved. .

  Even in such a case, the present invention is effective by applying the present algorithm in the same manner as in the case of the plurality of frame rates described above.

  As described above, according to the present invention, it is possible to reduce the time stamp storage area without losing the accuracy of the time stamp, and a relatively storage area such as a digital camera can be used. It is possible to record more accurate time stamps while dealing with various problems that occur in devices with large restrictions.

  As described above, according to the present invention, it is possible to generate a highly accurate time stamp that does not logically generate an error, while maintaining an accurate time stamp that is not visually deteriorated. The storage efficiency of the storage table can be improved, and the file size of the storage file can be efficiently reduced.

  As a result, even in a restricted device having only a small memory, it is possible to reduce the possibility of inducing a serious processing abnormality due to memory compression.

<Others>
In the processing of the embodiment, a storage medium recording software program codes embodying each function is provided to a system or apparatus, and a computer (or CPU or MPU) of the system or apparatus is stored in the storage medium. It can also be realized by reading and executing the program code. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. As a storage medium for supplying such program code, for example, floppy (registered trademark) disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD-R, magnetic tape, nonvolatile memory card, ROM Etc. can be used.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an OS (Operating System) running on the computer based on the instruction of the program code. Includes the case where the functions of the above-described embodiments are realized by performing part or all of the actual processing.

  Furthermore, after the program code read from the storage medium is written in the memory provided in the function expansion board inserted into the computer or the function expansion unit connected to the computer, the function is determined based on the instruction of the program code. This includes a case where a CPU or the like provided in an expansion board or function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

It is a block diagram which shows the outline of a multimedia data multiplexing apparatus. It is the figure which showed the time stamp offset schematically. It is a schematic diagram which shows MP4 which is a MPEG-4 standard for the time stamp storage format. FIG. 5 is a diagram showing examples of actual values according to the present invention including exceptions. It is a flowchart (1) describing the algorithm by this invention. It is a flowchart (2) describing the algorithm by this invention.

Claims (23)

Storage means for storing encoded data;
Time stamp reading means for reading a time stamp from the encoded data;
A time scale calculating means for calculating a high accuracy time scale for providing a second clock accuracy based on a frame rate of the encoded data and a first clock accuracy related to the time stamp;
A first offset that is a theoretical offset of the time stamp between two frames in the encoded data, and a second offset that is a theoretical offset of a high precision time stamp between two frames in the high precision time scale. Offset calculating means for calculating,
A time stamp conversion means for converting a time stamp of the encoded data into the high-accuracy time stamp based on the correspondence relationship between the first and second offsets;
A time stamp table generating means for generating a time stamp table using the high precision time stamp;
A data generation device comprising:   The high-accuracy time scale calculation means obtains the high-accuracy time scale by calculating a common multiple of a frame rate of the encoded data and a basic time scale for providing the first clock accuracy. The data generation device according to claim 1.   3. The data according to claim 2, wherein the high-accuracy time scale calculation means calculates a least common multiple of a frame rate of the encoded data and a basic time scale for providing the first clock accuracy. Generator.   4. The offset calculation unit according to claim 1, wherein the offset calculation unit sets a quotient obtained by dividing the high-accuracy time scale by the frame rate as the second offset. 5. Data generator.   The time stamp conversion means obtains the high precision time stamp by multiplying the time stamp by a quotient obtained by dividing the high precision time scale by the first clock precision. The data generation device according to any one of 1 to 4. The time stamp table generation means further relates to the high precision time stamp obtained by the time stamp conversion means, wherein a third offset which is an actual offset of the high precision time stamp between two consecutive frames is a predetermined error. Error determining means for determining whether or not it is within the range of
The data generation apparatus according to claim 5, wherein the time stamp table is generated according to a determination result of the error determination unit.   If the error determination unit determines that the third offset is within the predetermined error range, the time stamp table generation unit uses the second offset calculated by the calculation unit. The data generation apparatus according to claim 5, wherein a stamp table is generated.   The time stamp table generating means is an actual offset of a high-precision time stamp between two corresponding frames if the error determining means determines that the third offset is outside the predetermined error range. 7. The data generation apparatus according to claim 6, wherein an offset of 4 is calculated, and the time stamp table is generated using the fourth offset.   The time stamp conversion means calculates a third offset which is an actual offset of a time stamp between two consecutive frames in the encoded data, and divides the high-accuracy time scale by the first clock accuracy. 4. The fourth offset, which is an actual offset of the time stamp in the high-accuracy time scale, is calculated by multiplying the quotient obtained by the third offset with the third offset. 5. The data generation device according to claim 1. The time stamp table generation means further comprises error determination means for determining whether or not the fourth offset is within a predetermined error range,
The data generation apparatus according to claim 9, wherein the time stamp table is generated according to a determination result of the error determination unit.   The time stamp table generating unit generates the time stamp table using the second offset when the error determining unit determines that the fourth offset is within the predetermined error range. The data generation device according to claim 10, wherein   The time stamp table generating means is an actual offset of a high-precision time stamp between two corresponding frames if the error determining means determines that the fourth offset is outside the predetermined error range. The data generation apparatus according to claim 11, wherein an offset of 5 is calculated, and the time stamp table is generated using the fifth offset.   The error determination means obtains the first offset and the actual offset of the time stamp between two consecutive frames in the encoded data by dividing the high-accuracy time scale by the first clock accuracy. The data generation apparatus according to claim 6, wherein it is determined whether or not the quotient is smaller than a given quotient.   The error determination means performs determination using a value obtained by adding a predetermined error value in addition to a quotient obtained by dividing the high-accuracy time scale by the first clock accuracy. The data generation device according to claim 6, wherein the data generation device is a data generation device.   When the error determination means determines that the actual offset of the time stamp between two frames is outside the range of the predetermined error, the actual offset of the previous time stamp is further determined to be the predetermined error. When it is determined whether or not it is within the range, and when it is determined that it is within the range, the time stamp table generating means uses the actual offset determined to be out of the range and the second offset, and The data generation apparatus according to claim 6, wherein a time stamp table is generated.   The data generation device according to any one of claims 1 to 15, wherein, when the frame rate is not an integer, the time scale calculation unit performs integer multiplication by multiplying the frame rate by an integer. .   The time scale calculation means, when there are a plurality of the frame rates, calculates the high accuracy time scale from the plurality of frame rates and the clock accuracy associated with the time stamp. The data generation device according to any one of the above.   The time scale calculation means is used for the purpose of storing a plurality of data in the same file, and when only a single time scale data can be stored in the same file, each of the plurality of data The data generation apparatus according to claim 17, wherein the high-accuracy time scale is calculated from a frame rate and a clock accuracy related to the time stamp. Data receiving means for receiving multimedia data;
The data generation apparatus according to any one of claims 1 to 18, further comprising encoded data generation means for encoding the multimedia data to generate the encoded data.   The data generation apparatus according to claim 19, wherein the encoded data is stored in any one of MP4, 3GPP, and MotionJPEG2000. A storage step for storing encoded data;
A time stamp reading step of reading a time stamp from the encoded data;
A time scale calculating step for calculating a high accuracy time scale for providing a second clock accuracy based on a frame rate of the encoded data and a first clock accuracy related to the time stamp;
A first offset that is a theoretical offset of the time stamp between two frames in the encoded data, and a second offset that is a theoretical offset of a high precision time stamp between two frames in the high precision time scale. An offset calculation step to calculate,
A time stamp conversion step of converting a time stamp of the encoded data into the high-accuracy time stamp based on the correspondence between the first and second offsets;
A time stamp table generating step for generating a time stamp table using the high-accuracy time stamp;
A data generation method comprising: Code for executing a storage step for storing encoded data;
A code for executing a time stamp reading step of reading a time stamp from the encoded data;
Code for executing a time scale calculation step for calculating a high accuracy time scale for providing a second clock accuracy based on a frame rate of the encoded data and a first clock accuracy related to the time stamp When,
A first offset that is a theoretical offset of the time stamp between two frames in the encoded data, and a second offset that is a theoretical offset of a high precision time stamp between two frames in the high precision time scale. Code to perform the offset calculation step to calculate,
A code for executing a time stamp conversion step for converting a time stamp of the encoded data into the high-accuracy time stamp based on the correspondence between the first and second offsets;
A code for executing a time stamp table generating step for generating a time stamp table using the high-precision time stamp;
A program comprising: A computer-readable storage medium storing the program according to claim 22.

Images