JP3401505B2 - Time encoding method for VOP - Google Patents

Description

Detailed Description of the Invention

[0001]

FIELD OF THE INVENTION The present invention is useful for encoding digital audiovisual material that requires multiple independently encoded audiovisual objects to be synchronized for presentation. The invention is particularly useful when the temporal sampling of audiovisual material is not the same.

[0002]

2. Description of the Related Art In MPEG1 and MPEG2,
The input video consists of image frames sampled at regular time intervals. This represents the finest temporal resolution of the input video. FIG. 1 shows a video sequence with a constant frame rate in which image frames are sampled at regular intervals. In a coded representation of a video sequence using the MPEG1 standard and the MPEG2 standard, the display order of decoded frames is represented by a temporal reference. This parameter is described in the picture header of the bitstream syntax. The value of this parameter is incremented by one for each decoded frame when checking the display order.

H. In the H.263 standard, frames may be skipped and thus variable frame rate video sequences may be decoded. However, the sampling of the frame remains unchanged. Thus, the time reference methods used in MPEG1 and MPEG2 are still appropriate and need only be modified to increment by 1 (the number of non-transferred pictures at the input frame rate) instead of incrementing by 1. is there.

Currently, research and development is being done on encoding video as independent objects in multiple video object images. It represents a new concept in the decoding and synchronization of each video object. Each of these individual video objects may originate from multiple sources,
It is also expected that they may have completely different frame rates. Some of the objects may have a near continuous temporal sampling rate. When these video target images are combined and displayed, they become a composite image. Therefore, some kind of synchronization is required for this synthesis. The display frame rate may be different from the frame rate of any video target image. FIG. 2 shows an example of two video target images having different frame rates. Even if a common frame rate between the two video objects can be found, the frame rate will not necessarily be the same as the output frame rate of the compositing processor.

The problem in the video domain will now be described, but similar principles of the invention can be extended to the audio domain and also to a combination of the two.

It is clear from the above that the current state of the art does not meet the requirements for synchronizing video objects. Also, the current situation in the art does not provide a common reference time when different video objects have different frame rates that are not multiples of each other.

The first problem is how to provide a common local time base mechanism for each video object. This time base is a very precise time granularity
In addition to being able to provide temporal granularity, it also has to be able to deal with the fact that there can be very long intervals between two consecutive instances of a video object.

A second problem is how to provide a mechanism for synchronizing video objects having different frame rates.

[0009]

The above problem can be solved by introducing a common time resolution used for all local time references. To provide a wide range of temporal granularity, the local time base is divided into two distinct parts. The first part comprises a temporal resolution with a fine granularity that provides a short time base. The second part contains a temporal resolution with a coarse granularity that provides a long time base. A short time base is included in each video object and provides a reference time at the time of the video object. This short time base is then synchronized to the long time base common to all video objects. This long time base is used to synchronize all the various video objects to a common time base provided by the master clock.

[0010]

According to a first aspect of the present invention, a method of encoding a local time base of a visual sequence in compressed data, the method comprising obtaining a time point of the visual sequence by temporal sampling. A step of determining a local time base of the time point to be encoded in the compressed data, and a modulo time base representing generation of a series of reference times given at equal time intervals in the local time base, Encoding the local time reference with two parts consisting of time reference increments applied within the uniform time interval; inserting a modulo time reference into the compressed data each time the time interval elapses. A method comprising: inserting a time base increment into the compressed data at said point in time of the visual sequence. That.

According to a second aspect of the invention, there is provided a method of encoding a local time base of a visual sequence in compressed data, the method comprising: obtaining a time point of the visual sequence by temporal sampling; Determining a local time base of said time point to be encoded into, a first compression method without reference to any future time point, and a second compression method with reference to a future playback time point. One of two methods, encoding the time point, and two parts, a modulo time base and a time base increment representing the occurrence of a series of evenly spaced reference times in the local time base. Thus, encoding the local time base, and incrementing the time base relative to the time compressed using the first compression method, Encoding as an absolute value based on a reference time located at, and a time base increment for a time point compressed using the second compression method at a time point previously compressed using the first compression method. Encoding as a related value of the local time base of the, inserting a modulo time base into the compressed data each time the specific interval has elapsed, and adding a time base increment to the compressed data of the time point of the visual sequence. And a step of inserting.

According to a third aspect of the present invention, a plurality of compressed bitstreams having encoded local time base information contained therein are multiplexed, and the time base offset is encoded for multiplexing. Synchronizing the local time bases of individual compressed bitstreams to a common time base by inserting them into a multiplexed bitstream, and multiplexing until all compressed bitstreams have reached the modulo time base. Inspecting each compressed bitstream for the next compression time to be placed in the bitstream and inserting a common modulo time base into the multiplexed bitstream and skipping the modulo time base of the compressed bitstream The last two steps until all compressed bitstreams have been processed. The method of claim 1 or claim 2, further comprising: repeating the steps.

According to a fourth aspect of the present invention, a plurality of compressed bitstreams having encoded local time base information contained therein are multiplexed and demultiplexed,
Decoding the timebase offsets of the individual compressed bitstreams for demultiplexing, and checking the multiplexed bitstream for the next compression time until the modulo timebase is detected in the multiplexed bitstream. Then placing the instants in the appropriate compressed bitstreams, inserting the modulo time base into each compressed bitstream, and repeating the last two steps until all multiplexed bitstreams have been processed. The method of claim 1 or claim 2, further comprising:

According to a fifth aspect of the present invention, there is provided a method of decoding a local time base of a visual sequence from a time base of compressed data encoded by the method of claim 1, the time base offset Taking into account the initialization of the reference time base, incrementing the reference time base by a specific interval for each decoded modulo time base, and decoding the time base increment at the time of compression. And determining the decoding time reference at the time point by adding the value of the decoded time reference increment to a reference time reference.

According to a sixth aspect of the present invention there is provided a method of decoding a local time base of a visual sequence from a time base of compressed data encoded by the method of claim 2 wherein the time base offset Taking into account the initialization of the reference time base, incrementing the reference time base by a specific interval for each decoded modulo time base, and decoding the time base increment at the time of compression. And if the first compression method was used to encode the time instant, then the time base increment was determined to be absolute, and if the second compression method was used to encode the instant time. The step of determining that the time base increment is relative, and, if the time base increment is determined to be absolute, adding the value of the decoded time base increment to the reference time base. Therefore, the step of determining the decoding time base at the time point and the step of determining that the time base increment is relative, the value of the decoded time base increment is encoded using the first compression method. Determining the decoding time reference at the time point by adding to the decoding time reference at the previous time point.

According to a seventh aspect of the present invention, there is provided a method for multiplexing a plurality of compressed bitstreams according to claim 3, wherein each compressed bitstream itself is a multiplexed bitstream.

According to an eighth aspect of the present invention, the local time base, the time base increment, and the time base offset are in units of milliseconds, and the particular interval has a duration of 1000 milliseconds. Alternatively, it is a method for encoding the time reference according to any one of claims 2 to 4.

[0018]

DETAILED DESCRIPTION OF THE INVENTION The present invention operates by using two time display formats. The first time display format is
It is a short time reference added to the video target image. Less than,
This time reference is called a VOP (Video Object Plane) time increment. This VOP time increment serves as the timing used for the video object in association with a long time reference that is added to the group of video objects that are decoded and combined together. This long time base is called the modulo time base. And these VOPs
The time increment and the modulo time base are used in conjunction to determine the actual time base for use in compositing the video object image into the final composite sequence for display.

To facilitate editing the bitstream and synthesizing different video objects from different sources into a new group of video objects, the local time of each video object can be changed from a common time reference. A third component is needed that provides a constant offset value to the reference. In the following, this offset will be referred to as the VOP time offset. This prevents different target images from having to be synchronized with a granularity equal to the modulo time base interval. This component should be constant for each video object in the group to which the video objects that are multiplexed together belong.

First, the modulo time base will be described.

The modulo time base represents the coarse resolution of the local time base. It does not have a value like VOP Time Increment. In fact, it ’s a VOP
It is a more important synchronization mechanism for synchronizing the time increment to the local time base of the video object. It is placed as a marker in the encoded bitstream,
It indicates that the VOP time increment of the video object that follows must be reset, and that the local time base reference must be incremented in units of one or more modulo time base intervals. 3, FIG. 4, FIG. 5 to FIG. 10, and FIG.
At 1, the modulo time base is represented in the bitstream header by a consecutive "1" inserted before the VOP time increment, followed by a "0". The number of consecutive "1" s is zero or more. The number of "1" s inserted in the bitstream depends on the number of units of the modulo time base that have passed since the last I-VOP or P-VOP. In the encoder and decoder, the modulo time base counter is incremented by one each time a "1" is detected. The modulo time base counter is of finite length, so in a real system the modulo time base is reset to 0 when its maximum value is exceeded. In a typical video sequence, video objects form groups of VOPs. Therefore,
The modulo time base is normally reset at the start of this VOP group.

Next, the VOP time increment will be described.

The VOP time increment must be in units that can make use of the shortest temporal sampling of the video object. It may be a negative time reference used for the target picture. Therefore, it represents the finest temporal resolution granularity required or the finest temporal resolution granularity available.

The VOP time increment may then be represented by a finite length number that is greater than or equal to the ratio of the global time base interval / local time base resolution. FIG. 3 shows V used for I and P-video objects.
An example of OP time increment and modulo time base reference is shown. Absolute time base is used. The VOP time increment is reset each time the modulo time base is detected. FIG. 4 shows another example using I, P, and B-video objects. The operation is the same except that the modulo time base is similarly repeated in the B-video object. If the modulo time base is not repeated in the same way in the B-video object, ambiguity occurs due to the difference in decoding and presentation order. This is detailed below.

Since the VOP time increment corresponds to the presentation time reference, a potential problem occurs when the encoding order is different from the presentation order. This is caused by the B-video target image. MPEG-1
And as with the MPEG-2 B-picture
-Video object pictures are coded after their reference video object pictures, even if their presentation order is before the reference I-video object picture and the reference P-video object picture. Since the VOP time increment is finite and relative based on the modulo time base, the VOP time increment is reset when the modulo time base is detected. However, the encoding order for the B-video target picture remains delayed. 5-8 show possible ambiguities. It is not possible to determine when the VOP time increment should be reset. In fact, Figure 5
Given the sequence of events to be encoded as shown in Figure 6, it is not possible to know which timing position in Figure 6, Figure 7 and Figure 8 it is trying to represent. This problem arises because it uses a single modulo time base that is shared by all different types of video objects that have a mixture of different coding and presentation orders. There is nothing that can be done about the encoding order. This is because this reference information is needed by the B-video target picture. Also, it is not desirable for each of the different prediction forms to have its own modulo time base.

Next, the VOP time offset will be described.

In addition to the above, the modulo time base is shared among all video objects. This means that the synchronization between different video objects has a granularity equal to the modulo time base interval. This is particularly unacceptable when video target images from different groups are combined to form a new group of video target images. FIG. 11 shows an example of two different video target images encoded by two different local time bases offset from each other. Thus, when these video target images are multiplexed, the video target images will also be out of sync. A finer granularity is achieved by having a VOP time offset for each individual video object. This means that only this value will change if the video object is manipulated and multiplexed. Not only does the VOP time increment need not be changed, but different video objects can be multiplexed without the use of timing differences with coarse granularity.
FIG. 11 illustrates the use of this time base offset.

The preferred embodiment of the present invention is a method of encoding the time reference used for each individual video object picture bitstream, and a method of multiplexing different video object pictures to a common time reference. It includes a method of demultiplexing a multiplexed bitstream into components, and a method of reproducing a time reference from a component bitstream.

Next, time-based coding will be described.

A flow chart of an embodiment for encoding the time base is shown in FIG. In the encoder, in step 1, the local time base is first initialized to the local start time. The process moves to step 2, where the encoder determines the current value of the local time base. In step 3, the obtained local time base is compared with a pre-encoded modulo time base to check whether the interval exceeds the modulo time base interval. If so, control passes to step 4, where the required number of modulo time bases are inserted into the bitstream. If the interval is not exceeded, no special treatment is needed. The process then proceeds to step 5, where the VOP time increment is inserted into the bitstream. Next, in step 6, the target image is encoded and inserted into the bitstream. Then, the encoder checks in step 7 to determine if there are more target pictures to be encoded. If there is a target picture to be encoded, the process returns to step 2 where the local time reference is obtained. If there is no target image to be encoded, the process ends.

The following equations are used to determine the absolute and relative VOP time increments for the I / P- and B-video objects, respectively. t _GTBn = n × t _GTBI + t _GTB0 (n = 0, 1, 2, 3, ...) (1) t _AVTI = t _{ETBI / P} −t _GTBn (2) t _RVTI = t _ETBB −t _{ETBI / P} ( 3) where t _GTBn is the encoder time base represented by the nth coded modulo time base.

T _GTBI is a predetermined modulo time base interval.

T _GTB0 is the start time of the encoder time base.

T _AVTI is the absolute VOP time increment for the I or P-video subject.

T _{ETBI / P} is the encoder time base at the start of encoding the I or P-video object picture.

T _RVTI is the relative VOP time increment for the B-video subject.

T _ETBB is the encoder time base at the start of encoding the B-video target picture. Next, multiplexing of a plurality of video target images will be described.

When multiple video target images are multiplexed into one, the multiplexer examines the bitstreams of the multiplexed video target images to determine not only synchronization but also the order of multiplexing. The operations included in this are shown in FIG. In step 11, the VOP time offset for each video object picture to be multiplexed is inserted into the bitstream. Next, in step 12, all bitstreams of the video objects to be multiplexed are examined to determine if all the video objects are their respective modulo time bases. If so, the process proceeds to step 13, where the common modulo time base is inserted into the multiplexed bitstream. If not, the process proceeds to step 14, where the next coded video object is inserted into the multiplexed bitstream. In step 15, the bitstream of the video object to be multiplexed is checked again for further video objects to be multiplexed. If so, control again passes to step 12. If not, this process ends.

Next, demultiplexing of a bit stream including a plurality of video target images will be described.

The demultiplexing of a bitstream containing multiple video objects is shown in FIG. This process is step 2
Starting from 1, the VOP time offset is decoded and sent to the decoder for use in synchronization. And
In step 22, the multiplexed bitstream is examined to determine if a modulo time base has been detected. If the modulo time base is detected,
The process proceeds to step 23, where the modulo time base is inserted into all video object bitstreams. If the modulo time base is not detected, the process proceeds to step 24, where the next video object picture is examined and inserted into the appropriate video object picture bitstream. Finally, the multiplexed bitstream is examined again to determine if there are more video objects to demultiplex. If so, the process returns to step 22.
Proceed to. If not, the process ends.

Next, time-based reproduction will be described.

An example of reproducing the time reference is shown in FIG. When regenerating the local time base, the process begins at step 31 where the local time base is initialized taking into account the VOP time offset decoded by the demultiplexer. The process then proceeds to step 32, where the bitstream is examined to determine if the modulo time base has been decoded. If the modulo time base has been decoded, the process proceeds to step 33, where the local time base is incremented by the modulo time base increment. Then, the process proceeds to step 37. If the modulo time base has not been decoded, the process proceeds to step 34, where the video object is examined to determine if it is a B-video object. If it is a B-video target image, the process proceeds to step 35, where the decoding time reference of the B-video target image is calculated based on equation (6). Then, the process proceeds to step 37. If the result of step 34 is B
If not a video object, the process proceeds to step 36, where the decoding time reference is calculated according to equation (5). Then, the process proceeds to step 37. In step 37, the bitstream is examined to determine if there are more video objects to decode. If so, the process proceeds to step 32 again. If not, the process ends.

To determine the presentation time stamp of the video subject, the following equation is used: t _GTBn = n × t _GTBI + t _GTB0 (n = 0, 1, 2, 3, ...) (4) t _{DTBI / P} = t _AVTI + t _GTBn (5) t _DTBB = t _RVTI + t _{DTBI / P} (6) Where t _GTBn is the decoding time base represented by the nth decoded modulo time base.

T _GTBI is a predetermined modulo time base interval.

T _GTB0 is the start time based on the decoding time.

T _{DTBI / P} is the decoding time reference at the starting point of decoding the I or P-video target picture.

T _AVTI is the decoded absolute VOP time increment for the I or P-video object picture.

T _DTBB is a decoding time reference at the start point of decoding the B-video target picture.

T _RVTI is the decoded relative VOP time increment for the B-video target picture.

Next, an embodiment of the bit stream encoder will be described.

FIG. 16 shows the modulo time base and VOP.
It is a block block diagram explaining the Example of the bitstream encoder for encoding a time increment. The example shown in FIG. 4 is used for this description. Because bi-directional prediction is used, the coding order differs from the presentation order shown in FIG. The encoding order is B-
I-VOP followed by P-VOP before VOP
It starts from. This is explained in the following three paragraphs.

The process is an initialization step 41.
, Where the bitstream encoder begins by initializing the local time base register to the initial value of the time code. The same time code value is encoded in the bitstream. At the beginning of the encoding of the next I-VOP, the time code comparator step 42.
Compares the presentation time of the I-VOP with the local time base register. The result is sent to step 43 which is the modulo time base encoder. The modulo time base encoder inserts into the bitstream as many "1's" as necessary, equal to the number of modulo time base increments that have elapsed.
Then, this is followed by the symbol "0" to indicate the end of the modulo time base code. The local time base register is updated with the current modulo time base. The process then proceeds to step 44, which is a VOP time base incremental encoder, where the remaining portion of the presentation time code of the I-VOP is encoded.

This process is repeated for the next encoded video object that is a P-VOP. A time code comparator, step 42, compares the presentation time of the P-VOP with a local time base register. The result is sent to step 43 which is the modulo time base encoder. The modulo time base encoder inserts the required number of "1" equal to the number of modulo time base increments that have elapsed. Then, this is followed by the symbol "0" to indicate the end of the modulo time base code. The B-VOP time base register is set to the value of the local time base register and the local time base register is updated to the current modulo time base. The process then proceeds to step 44, which is a VOP time base incremental encoder, where the remaining portion of the presentation time code of the P-VOP is encoded.

The process is then repeated for the next encoded video object that is a B-VOP. Step 42, a time code comparator, compares the B-VOP presentation time with the B-VOP time reference register.
Step 43, the result of which is the modulo time base encoder.
Sent to. The modulo time base encoder inserts the required number of "1" equal to the number of modulo time base increments that have elapsed. Then, this is followed by the symbol "0" to indicate the end of the modulo time base code. Both the B-VOP time base register and the local time base register are B
-No change after processing of VOP. The process then proceeds to step 44 which is a VOP time base increment encoder,
Therefore, the remaining part of the presentation time code of the B-VOP is encoded.

The local time base register stores the next VOP.
It is reset at the next I-VOP, which marks the beginning of the group.

Next, an embodiment of the bitstream decoder will be described.

FIG. 17 is a block diagram illustrating an embodiment of a decoder used for the modulo time base and the VOP time increment to reproduce the presentation time stamp. As in the encoder embodiment, the example shown in FIG. 4 is used. The decoding order is the same as the encoding order, and the I-VOP is preceded by the B-VOP.
And the subsequent P-VOP are decoded. This is explained in the following paragraphs.

The process is an initializer, step 51.
Beginning with, the local time base register is set to the value of the time code decoded from the bitstream. The process then proceeds to the modulo time base decoder, step 52, where the modulo time base increment is decoded. The total number of modulo time base increments decoded is given by the number of "1" s decoded before the symbol "0". Next, the VOP time base increment is VO
It is decoded in step 53 which is a P time base incremental decoder. In step 54, which is a time base calculator, the presentation time of the I-VOP is played. The sum of the decoded modulo time base increments is added to the local time base register. The VOP time base increment is then added to the local time base register,
The presentation time of the I-VOP is obtained. The process then proceeds to the video target image decoder, step 55, where the video target image is decoded.

For the P-VOP, the process is repeated at step 52, which is the modulo time base decoder, where the modulo time base increment is decoded. The total number of modulo time base increments decoded is given by the number of "1" s decoded before the symbol "0". The VOP time base increment is then decoded in step 53 which is a VOP time base increment decoder. In step 54, which is a time base calculator, the presentation time of the P-VOP is played. The B-VOP modulo time base register is set to the value of the local time base register. The sum of the decoded modulo time base increments is added to the local time base register. And
The VOP time base increment is added to the local time base register to obtain the presentation time of the P-VOP. The process proceeds to a video target image decoder, where the video target image is decoded.

For the B-VOP, the process is repeated at step 52, which is the modulo time base decoder, where the modulo time base increment is decoded. The total number of modulo time base increments decoded is given by the number of "1" s decoded before the symbol "0". The VOP time base increment is then decoded in step 53 which is a VOP time base increment decoder. In step 54, which is a time base calculator, the presentation time of the B-VOP is played. The total value of the decoded modulo time base increment and the VOP time base increment are B−
The presentation time of B-VOP is obtained by adding to the VOP time reference register. Both the B-VOP time base register and the local time base register remain unchanged. Then, the process proceeds to the video target image decoder, where the video target image is decoded.

The local time base register stores the next VOP.
It is reset at the next I-VOP, which marks the beginning of the group.

Next, a concrete example will be described.

Referring to FIG. 18, an example of the steps for encoding the compressed data into bitstream data is shown. As shown in the upper row of FIG. 18, the compressed video data VOP is displayed in the order of I1, B1, B.
2, P1, B3, P2 are arranged in a line, and GOP
A (Group of Pictures) header is inserted at the beginning of the VOP group. As displayed, the local time at which the display is performed is determined for each VOP using the local time clock. For example,
The first VOP (I1-VOP) is displayed at 1: 23: 45: 350 milliseconds (1: 23: 45: 350) counted from the very start point of the video data, and the second VOP (I1-VOP) is displayed.
OP (B1-VOP) is displayed at 1: 23: 45: 750, and the third VOP (B2-VOP) is 1: 2.
Displayed at 23: 46: 150, and so on.

In order to encode the VOP, it is necessary to insert the display time data into each VOP.
If the time data is inserted in the complete form including hours, minutes, seconds, and milliseconds, a considerable data area is required in the header part of each VOP. The purpose of the invention is to reduce such data areas,
It is also to simplify the time data to be inserted in each VOP.

Each of the VOPs shown in the top row of FIG. 18 stores display time data of milliseconds in the VOP time increment area. Also, the VO in the top row
Each P temporarily stores display time data including hours, minutes, and seconds. GOP header is the first VOP
The display data composed of hours, minutes and seconds used for (I1-VOP) is stored.

As shown in the second row of FIG.
The VOP is delayed for a predetermined time using a buffer (not shown). According to the bidirectional prediction method, the order of the VOPs changes when the VOPs are generated from the buffer.
Should be located after the P-VOP referenced by that B-VOP. Therefore, VOP is I1, P1, B
They are arranged in a line in the order of 1, B2, P2, B3.

As shown in the third row of FIG.
At time T1, i.e. when the GOP header is just encoded, when stored in the GOP header,
The minute and second data are stored in the local time base register as they are. In the example shown in FIG. 18, the local time base register stores 1:23:45. Then, before the time T2, the bit stream data corresponding to the GOP header is obtained, and the hour, minute, and second data are inserted as shown in the lower part of FIG.

Then, at time T2, the first VOP
(I1-VOP) is captured. The time code comparator
The time stored in the local time base register (hour, minute,
Seconds) is compared to the time (hours, minutes, seconds) temporarily stored in the first VOP (I1-VOP). According to this example,
The comparison result is the same. Therefore, the comparator produces a "0" indicating that the first VOP (I1-VOP) occurred in the same second as the second held in the local time base register. The "0" generated by the comparator is directly added to the modulo time base area of the first VOP (I1-VOP). At the same time, the first VOP
The hour, minute, and second data temporarily stored in (I1-VOP) is removed. Therefore, before time T3, the bitstream data corresponding to the first VOP (I1-VOP) is obtained, "0" is inserted in the modulo time base area, and "350" is inserted in the VOP time increment area. To be done.

Next, at time T3, the second VOP
(P1-VOP) is captured. The time code comparator
The time stored in the local time base register (hour, minute,
Seconds) is compared to the time (hours, minutes, seconds) temporarily stored in the second VOP (P1-VOP). According to this example,
The result of the comparison is that the time temporarily stored in the second VOP (P1-VOP) is one second larger than the time stored in the local time base register. Therefore, the comparator produces a "10" indicating that the second VOP (P1-VOP) has occurred in the second following the second held in the local time base register. If the second
If the VOP (P1-VOP) of 1 occurs in the second following the second held in the local time base register, the comparator generates "110".

After time T3, the B-VOP time reference register is set to a time equal to the time held in the local time reference register immediately before time T3. In this example, the B-VOP time base register is
1:23:45 is set. Further, after the time T3, the local time base register is set to the second VO
It is incremented to a time equal to the time temporarily stored in P (P1-VOP). Therefore, in this example, the local time base register is 1:23:
Incremented to 46.

"10" obtained as a result of generation by the comparator is the second VOP (P1-VOP) as it is.
Is assigned to the modulo time base area of. At the same time,
The hour, minute and second data temporarily stored in the second VOP (P1-VOP) is removed. Therefore, time T
Before 4, the bitstream data corresponding to the second VOP (P1-VOP) is obtained, "10" is inserted in the modulo time base area, and "550" is the VOP.
It is inserted in the time increment area.

Then, at time T4, the third VOP
(B1-VOP) is captured. The time code comparator
The time stored in the B-VOP time base register (hour,
Minutes, seconds) with the time (hours, minutes, seconds) temporarily stored in the third VOP (B1-VOP). According to this example, the comparison result is the same. Therefore, the comparator is
A "0" is generated to indicate that the third VOP (B1-VOP) has occurred in the same second as the second held in the B-VOP time base register. The resultant "0" generated by the comparator is directly used as the third VOP (B
1-VOP) is added to the modulo time base area. At the same time, the hour, minute, and second data temporarily stored in the first VOP (I1-VOP) is removed. Therefore, before time T5, the third VOP (B1-
VOP) corresponding bitstream data is obtained,
"0" is inserted in the modulo time base area and "750"
Is inserted in the VOP time increment area.

Then, at time T5, the fourth VOP
(B2-VOP) is captured. The time code comparator
The time stored in the B-VOP time base register (hour,
Minutes, seconds) with the time (hours, minutes, seconds) temporarily stored in the fourth VOP (B2-VOP). According to this example, the result of the comparison is that the time temporarily stored in the fourth VOP (B2-VOP) is one second larger than the time stored in the B-VOP time base register. Therefore, in the comparator, the fourth VOP (B2-VOP) is B
-One after the second held in the VOP time base register
Generate "10" to represent what happened in seconds.

While processing a B-type VOP,
Regardless of what result the comparator produces, neither the local time base register nor the B-VOP time base register is incremented.

The value "10" obtained as a result of the comparator is directly used as the fourth VOP (B2-VOP).
Is assigned to the modulo time base area of. At the same time,
The hour, minute, and second data temporarily stored in the fourth VOP (B2-VOP) is removed. Therefore, time T
Before 6, bit stream data corresponding to the fourth VOP (B2-VOP) is obtained, "10" is inserted in the modulo time base area, and "150" is VOP.
It is inserted in the time increment area.

Then, at time T6, the fifth VOP
(P2-VOP) is captured. The time code comparator
The time stored in the local time base register (hour, minute,
Seconds) is compared to the time (hours, minutes, seconds) temporarily stored in the fifth VOP (P2-VOP). According to this example,
The result of the comparison is that the time temporarily stored in the fifth VOP (P2-VOP) is one second larger than the time stored in the local time base register. Therefore, the comparator produces a "10" indicating that the fifth VOP (P2-VOP) occurred in the second following the second held in the local time base register.

After time T6, the B-VOP time base register is incremented to a time equal to the time held in the local time base register immediately before time T6. In this example, the B-VOP time base register is incremented 1:23:46.
Further, after time T6, the local time base register is incremented to a time equal to the time temporarily stored in the fifth VOP (P2-VOP).
Therefore, in this example, the local time base register is incremented at 1:23:47.

"10" obtained as a result by the comparator is the fifth VOP (P2-VOP) as it is.
Is assigned to the modulo time base area of. At the same time,
The hour, minute and second data temporarily stored in the fifth VOP (P2-VOP) is removed. Therefore, time T
Before 7, bit stream data corresponding to the fifth VOP (P2-VOP) is obtained, "10" is inserted in the modulo time base area, and "350" is VOP.
It is inserted in the time increment area.

After that, similar processing is executed, and bit stream data for subsequent VOPs is formed.

In order to decode this bit stream data, processing reverse to the above processing is executed. First of all, the time (hours, minutes, seconds) held in the GOP header
Is read. The read time is stored in the local time base register.

When an I-type or P-type VOP, that is, a VOP other than the B-type is received, the data stored in the modulo time base area is read. If the read data is "0", that is, if there is no 1 before 0, the local time base register is not changed. Also, the B-VOP time base register is not changed. If the read data is "10", the time stored in the local time base register is 1
Incremented by seconds. If the read data is "110", the time stored in the local time base register is incremented by 2 seconds. Thus, the number of seconds to be incremented is determined by the number of 1's inserted before the 0's. When the read data is “10” or “110”, the B-VOP time base register, which is a memory, copies the time held by the local time base register immediately before the increment. Then, the time (hours, minutes, seconds) held in the local time base register is combined with the time (milliseconds) held in the VOP time increment area to determine the time when the I-type or P-type VOP should be generated. Will be confirmed.

When a B type VOP is received, the data stored in the modulo time base area is read. If the read data is "0", the time (hour, minute, second) held in the B-VOP time base register is VO.
Combined with the time (milliseconds) held in the P time increment area, the time at which the B type VOP should be generated is determined. If the read data is "10", B
The time (hour, minute,
Second) is added by 1 second, and the time obtained by the addition is combined with the time (millisecond) held in the VOP time increment area to determine the time when the B type VOP should be generated. If the read data is “110”, 2 seconds is added to the time (hour, minute, second) held in the B-VOP time reference register, and the time obtained by this addition is the VOP time increment area. Combined with the time (milliseconds) held by the B.sub.2, the time at which a B-type VOP should be generated is determined.

An advantage of the present invention is that it is possible to multiplex video objects encoded by different encoders. Further, the present invention facilitates manipulating compressed data from different sources based on the target image to generate a new bitstream. The present invention is
A method of synchronizing audiovisual target images is provided.

Although the present invention has been described thus far, the above description can be modified in various forms. Such modifications do not depart from the spirit and scope of the invention, and as will be apparent to one of ordinary skill in the art, all such modifications are intended to be within the scope of the following claims. is there.

[Brief description of drawings]

1 is a diagram illustrating temporal sampling according to the related art in which frames of a video sequence are sampled at regular intervals.

FIG. 2 is a diagram for explaining the concept of video target images and their relationship to each other. The sampling of the video object image may be irregular, and the sampling period may change abruptly.

FIG. 3 is a diagram illustrating the present invention in which a reference time of a video target image is represented by a modulo time base and a VOP time increment. In this description, only I-VOP and P-VOP are used.

FIG. 4 is a diagram illustrating the present invention in which a reference time of a video target image is represented by a modulo time base and a VOP time increment. In this description, I-VOP, P-VOP, and B-VOP are used.

FIG. 5 is a diagram illustrating an example of ambiguity that may occur when the presentation order and the encoding order are different for a B-video target image.

FIG. 6 is a diagram illustrating an example of ambiguity that may occur when a presentation order and a coding order are different for a B-video target image.

FIG. 7 is a diagram illustrating an example of ambiguity that may occur when the presentation order and the encoding order are different for a B-video target image.

FIG. 8 is a diagram illustrating an example of ambiguity that may occur when a presentation order and a coding order are different for a B-video target image.

FIG. 9 is a diagram illustrating resolving ambiguity by using an absolute time base and a relative time base.

FIG. 10 is a diagram illustrating resolving ambiguity by using an absolute time base and a relative time base.

FIG. 11 is a diagram illustrating a combination of two VOPs and synchronizing them to a common time reference by using a VOP time offset.

FIG. 12 is a flowchart illustrating time-based encoding.

FIG. 13 is a flowchart illustrating multiplexing of a plurality of video target images.

FIG. 14 is a flowchart illustrating demultiplexing of a plurality of video target images.

FIG. 15 is a flowchart illustrating reproduction of a presentation time stamp.

FIG. 16 is a block diagram illustrating an operation of a bitstream encoder for encoding a time reference.

FIG. 17 is a block diagram illustrating an operation of a bitstream decoder for decoding a time base.

FIG. 18 is a time chart explaining formation of bit stream data.

─────────────────────────────────────────────────── ─── Continuation of front page (72) Inventor Shen Mei Shen Singapore 534415 Singapore, Thai Sen Avenue, Block 1022, 04-3530, Thai Sen Industrial Estate, Panasonic Singapore Research (72) Inventor Chak Joo Lee Singapore 534415 Singapore, Thai Sen Avenue, Block 1022, 04-3530, Thai Sen Industrial Estate, Panasonic Singapore Research Institute Co., Ltd. (58) Fields surveyed (Int.Cl. ⁷ , DB name) H04N ^7/ 24-7/68

Claims (5)

(57) [Claims] 1. A method for encoding the time of a VOP included in compressed data, wherein the local time base register sets a time of one second unit among the time of the VOP encoded immediately before the VOP to be encoded. The modulo time base increment is used to encode the increment of the time of the encoding target VOP with respect to the time of the local time reference register as a modulo time reference increment, and the increment of less than 1 second of the encoding target VOP is VOP.
A VOP time encoding method, characterized by encoding as a time increment and expressing the VOP time by a combination of the modulo time base increment and the VOP time increment. 2. The VOP time coding method according to claim 1, wherein the VOP is an I-VOP (intra-coded VOP) or a P-VOP (prediction-coded VOP). . 3. Times of I-VOPs (intra-coded VOPs), P-VOPs (prediction-coded VOPs) and B-VOPs (VOPs coded by bidirectional prediction) included in the compressed data. The local time base register stores the time in 1-second units of the I-VOP or P-VOP coded immediately before the I-VOP or P-VOP to be coded. The B-VOP time base register stores the time of one second unit among the times of the I-VOP or P-VOP whose display order of the B-VOP to be coded is immediately before, and the local time. I-VOP is the increment of the time of the I-VOP or P-VOP to be encoded with respect to the time of the reference register in units of one second.
Or encoded as a modulo time base increment of P-VOP, encoding an increment of less than 1 second of the time of the I-VOP or P-VOP to be encoded as a VOP time increment of I-VOP or P-VOP, The time of the I-VOP or P-VOP to be encoded is set to the I-VOP.
Or P-VOP modulo time base increment and the I-VOP or
Expressed in combination with the VOP time increment of P-VOP, store the encoded modulo time base increment of the I-VOP or P-VOP in the local time base register, for the time of the B-VOP time base register The increment of the second of the time of the B-VOP to be encoded is encoded as a modulo time base increment, and the increment of less than 1 second of the B-VOP to be encoded is the BV.
The VOP is encoded as a VOP time increment of OP, and the time of the B-VOP to be encoded is represented by a combination of a modulo time base increment of the B-VOP and a VOP time increment of the B-VOP. Time encoding method. 4. The time of the local time base register is stored in the B-BOP before the modulo time base increment of the encoded I-VOP or P-VOP is stored in the local time base register.
4. The VOP time encoding method according to claim 3, wherein the time is stored in a VOP time reference register. 5. The modulo time base increment comprises a sequence of 0 or 1 or more “1” s, and the number of “1s” is the increment of 1 second unit. 4. The VOP time encoding method according to any one of 4).