WO2005101414A1

WO2005101414A1 - Creating a seamless connection of multimedia sections without requiring recoding

Info

Publication number: WO2005101414A1
Application number: PCT/IB2005/051175
Authority: WO
Inventors: Serhiy Tkachenko; Arie Koene; Nikolco Gidalov
Original assignee: Koninklijke Philips Electronics N.V.
Priority date: 2004-04-15
Filing date: 2005-04-11
Publication date: 2005-10-27
Also published as: TW200605684A

Abstract

The invention concerns a method, devices and a computer program code (10) for interconnecting multimedia stream sections. The method comprises detecting userselected switching from a first to a second section at an exit time (USEX) of the first section and an entry time (USEN) of the second section, determining exit (AEX) and entry locations (AEN) having sequence start markers based on the exit time and entry time, selecting, in the first section, all media packets (V2, A2) associated with the marker and possible media packets (Al) after the exit location associated with previous sequences, selecting, in the second section, stream control information (SC) before the entry location, removing nonselected packets after the exit location in the first section and before the entry location in the second section, placing a sequence end marker (E) in the first section and adding filling packets to the sections (TS' l, TS'2) for meeting coding requirements.

Description

Creating a seamless connection of multimedia sections without requiring re coding

TECHNICAL FIELD The present invention is generally related to manipulating multimedia data streams and more particularly to a method, devices and a computer program product for interconnecting two multimedia stream sections.

BACKGROUND OF THE INVENTION In recent years there have been developed a number of standards for recording multimedia data on data carriers, examples of such standards being DVD (Digital Versatile Disc) and Blu-ray disc video. In for instance Blu-ray it is possible to allow a user to seamlessly connect together two streams or clips provided on a Blu-ray disc. The user does this by manipulating a so-called playlist, which list points at streams or clips that are to be played. One way of providing a seamless connection between one clip and another clip, which can take place when part of a playlist is deleted, i.e. one stream is to be ended prematurely and another stream is to be started after the original start position, is by truncating the two streams and interconnect them seamlessly. One way of doing this is described in US-2003/0103604, which describes the seamless connection of two clips at selected locations within the two clips. The two sections are connected together by truncating a last part of a first clip and truncating a first part of the second clip. However, according to said document, the last media information provided in the first clip and the first media information in the second clip are recoded. Such recoding is fairly complex, requires quite heavy processing and is therefore expensive by nature as well as slow. This type of seamless connection is therefore normally suitable for professional editing of multimedia streams. However, it would be interesting to provide this type of functionality also within the consumer market, in which case there is a need for a less complex way of providing seamless connection between streams in multimedia editing situations. SUMMARY OF THE INVENTION The present invention is therefore directed towards solving the above- mentioned problem of providing seamless connection between truncated multimedia streams in a more simple way in for instance consumer equipment. One object of the present invention is thus to provide a method of creating seamless connection between truncated multimedia streams in a more simple way that does not require recoding of the multimedia streams. According to a first aspect of the present invention, this object is achieved by a method of interconnecting two multimedia stream sections comprising the steps of: detecting user-selected switching from a first to a second section at a certain exit time of the first section and a certain entry time of the second section, determining an exit location based on the user-selected exit time in the first section, which exit location has a sequence start marker, selecting all media packets in the first section associated with the sequence start marker and possible media packets provided after the exit location associated with previous sequences in the first section, removing non-selected media packets after the exit location in the first section, creating and placing a sequence end marker in the first section, determining an entry location based on the user-selected entry time in the second section, which entry location has a sequence start marker, selecting stream control information provided in the second section before the entry location, removing non-selected packets before the entry location in the second section, and adding filling packets to the two sections in a number that meets coding requirements, such that the two sections can be seamlessly interconnected at the entry and exit locations. Another object of the present invention is to provide a device for creating seamless connection between truncated multimedia streams in a more simple way that does not require recoding of the multimedia streams. According to a second aspect of the present invention, this object is achieved by a device for connecting together two multimedia stream sections provided on a storage medium and comprising: a reading and a writing unit for reading and writing multimedia data on the storage medium, and a control unit for manipulating at least one multimedia stream provided on the storage medium, wherein the control unit is arranged to: detect user-selected switching from a first to a second section at a certain exit time of the first section and a certain entry time of the second section, determine an exit location based on the user selected exit time in the first section, which exit location has a sequence start marker, select all media packets in the first section associated with the sequence start marker and possible media packets provided after the exit location associated with previous sequences in the first section, remove non-selected media packets after the exit location in the first section, create and place a sequence end marker in the first section, determine an entry location based on the user-selected entry time in the second section, which entry location has a sequence start marker, select stream control information provided in the second section before the entry location, remove non-selected packets before the entry location in the second section, and add filling packets to the two sections in a number that meets coding requirements, such that the two sections can be seamlessly interconnected at the entry and exit locations. Another object is to provide a multimedia data presentation device that provides seamless connection between truncated multimedia streams in a more simple way that does not require recoding of the multimedia streams. This object is achieved according to a third aspect of the present invention by a multimedia data presentation device comprising a device for connecting together two multimedia stream sections according to the second aspect of the invention. Another object of the present invention is to provide a computer program product that provides seamless connection between truncated multimedia streams in a more simple way that does not require recoding of multimedia streams. According to a fourth aspect of the present invention, this object is achieved by a computer program product to be used on a computer for connecting together two multimedia stream sections and comprising a computer program code for making the computer execute, when said code is loaded into the computer, the following functions: detect user-selected switching from a first to a second section at a certain exit time of the first section and a certain entry time of the second section, determine an exit location based on the user-selected exit time in the first section, which exit location has a sequence start marker, select all media packets in the first section associated with the sequence start marker and possible media packets provided after the exit location associated with previous sequences in the first section, remove non-selected media packets after the exit location in the first section, create and place a sequence end marker in the first section, determine an entry location based on the user-selected entry time in the second section, which entry location has a sequence start marker, select stream control information provided in the second section before the entry location, remove non-selected packets before the entry location in the second section, and add filling packets to the two sections in a number that meets coding requirements, such that the two sections can be seamlessly interconnected at the entry and exit locations. Claim 2 is directed towards providing the exit and entry locations as the locations that are closest to the user-selected exit and entry times in order to minimize the offset from the desired exit and entry times. According to claim 3, the selected stream control information is the closest information in previous sections of the second section. In this way the most relevant such information is used and unnecessary searching is avoided. According to claim 4, the media packets include video packets, wherein the sequence end marker is provided after the last video packet associated with the sequence start marker. This is done in order to clearly define the end of the video information originating from the first section, which might also be necessary in order to meet requirements of the recording standard used. According to claims 5 and 6, null packets are inserted in the first section between non-selected packets and selected audio packets originating from the first section and/or stream control packets of the second section that might be necessary in order to meet requirements of the recording standard used. Claim 7 is directed towards using overhead data in the form of entry point maps in locating sequence start markers, which makes the locating process fast and efficient. Claim 8 is directed towards changing the entry point maps of the two sections in order to reflect the changes made, which might be necessary in order to meet requirements of the recording standard used. Claims 9 and 10 are directed towards interconnecting sections provided in the same stream and in different streams, respectively. With the present invention the object of interconnecting multimedia streams is achieved, without the need to recode said multimedia streams, but by only retaining relevant information together with the possible addition of null packets and/or stream control packets. Because of this, the connection is enabled at small cost and without the complexity associated with this recoding. The connection is furthermore fast and can be established without any perceivable delays to the user. The invention furthermore does not need an encoder for its implementation, which means that it can be provided in an overall simpler and cheaper device. The basic idea of the invention is to detect a user-entered exit time from a first multimedia stream section to an entry time to a second multimedia stream section, determine an exit location based on the user-selected exit time and an entry location based on the user- selected entry time, which entry and exit locations each have a sequence start marker, select all media packets associated with the sequence start marker and possible media packets provided after the exit location associated with previous sequences in the first section, remove non-selected media packets after the exit location in the first section, create and place a sequence end marker in the first section, select stream control information provided in the second section before the entry location, remove non-selected packets before the entry location in the second section, and add filling packets to the two sections in a number that meets coding requirements. In this way the two sections can be seamlessly interconnected at the entry and exit locations. The above-mentioned and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS The present invention will be further described in relation to the accompanying drawings, in which: Fig. 1 shows a block schematic of a multimedia presentation device comprising a device for interconnecting two multimedia stream sections according to the present invention, Fig. 2 schematically shows two transport streams that are to be connected, Fig. 3 schematically shows the inserting of user-selected entry and exit points in the two streams as well as the provision of actual entry and exit locations, Fig. 4 schematically shows the two streams after being modified for enabling seamless connection between the streams in a method according to the present invention, and Fig. 5 schematically shows a computer program product in the form of a CD ROM disc having a computer program code for performing the method according to the present invention.

DETAILED DESCRIPTION OF THE INVENTION The present invention is generally directed towards seamlessly interconnecting sections of multimedia streams without recoding of the multimedia content. In the following, a system for providing this facility will be described in relation to MPEG coded multimedia data in the form of MPEG transport streams that are stored on an optical disc according to the Blu-ray standard. It should however be realized that the invention is not limited to Blu-ray or even MPEG, but can be used just as well on other standards having suitable similar properties. The invention is furthermore not limited to such multimedia streams being provided on an optical disc, but includes storage of such multimedia streams on any suitable storage medium, like hard discs or memory sticks. Fig. 1 shows a block schematic of a multimedia presentation device 10 in the form of a simplified Blu-ray player. The player includes an optical disc drive 14 arranged to receive a Blu-ray disc 18 on which is stored a number of multimedia streams coded according to the Blu-ray standard. The disc drive 14 is connected to a device 12 for interconnecting two multimedia stream sections in the form of a Blu-ray coder/decoder, which device 12 is indicated by a dashed box. A user interface unit 16 is connected to a control unit 26 of the device 12 in order to allow a user to make selections of multimedia streams or clips to be played and also to connect one stream with another in a seamless fashion. The device 12 therefore comprises a reading unit 20 connected to the disc drive 14 for reading information on the disc 18. The reading unit 20 is furthermore connected to a stream separating unit 22, which removes additional data that is encoded with a transport stream of MPEG multimedia data. The stream separating unit 22 is connected to the control unit 26, to a buffer 24 and to a multimedia decoding unit 32, which multimedia decoding unit 32 provides decoded multimedia signals, which in this embodiment are decoded MPEG video and audio signals for a television set. In order to write data onto the disc 18, a writing unit 30 is connected to the disc drive 14, which writing unit 30 is further connected to a stream combining unit 28, said stream combining unit 28 being connected to both the control unit 26 and the buffer 24. In Fig. 1 the device 12 is provided for only playing multimedia data in order to better describe the invention. It should however be realised that it could just as well be provided with actual multimedia recording facilities, in which case there would be required a multimedia coding unit. The multimedia decoding unit 32 is a standard MPEG decoder, which separates the audio and video stream from each other and decodes the coded video and audio information. It should also be realised that there might be several buffers provided for temporary storage of different multimedia streams Fig. 2 schematically outlines two transport streams TS1 and TS2 or two video clips, which are retrieved from the disc 18. The streams include a number of multimedia packets grouped into frames. In the first stream TS1 there is a first frame Vi of video packets next to another frame of video packets N . Spaced from this frame N there are a number of audio packets Ai, which are associated with the video frame Ni, and yet further forward there are a number of audio packets A₂, which are associated with the video frame N₂. The video frame N₂ has a first data packet that has a section start marker or section start code, which indicates the start of a section having a GOP (Group of Pictures) I-frame. This means that this packet has all video information necessary for showing a certain scene. The packet furthermore has a timestamp and is associated with an entry point table EPi provided as overhead information in relation to the first transport stream TS1 and thus outside this transport stream, and including such information as the number of packets in the sequence. This entry point table EP ι directly points at the sequence start code, which makes it easy to locate. The second transport stream TS2 includes a number of stream control packets SC, which can be PAT (Program Association Table), PMT (Program Map Table) or PCR (Program Clock Reference) packets according to the Blu-ray standard which are separated from a frame of video packets N₃, wherein the first packet of this frame also has a data packet that has a section start code, a timestamp and is associated with an entry point table EP₂ provided as overhead information in relation to and outside the second transport stream TS2. This frame V₃ thus also has a GOP I-frame. At a distance from the video frame N₃ there is provided a number of audio packets A₃ associated with this video frame N₃. The reason for providing audio packets distanced from the video packets is that the processing of audio and video in the decoder 32 takes up different amounts of time. The end result is however that these should be provided to the user so as to be synchronised with each other. An embodiment of the present invention will now be described with reference being made to Figs. 1, 2, 3 and 4, where Fig. 3 shows a schematic view of the transport streams TSl and TS2 when a user has entered entry and exit points into the two transport streams, and Fig. 4 shows the two transport streams TS'l and TS'2 after being modified according to the principles of the present invention. A user wanting to play a piece of multimedia can select this by editing a playlist provided by the control unit 26. For the sake of explaining the present invention, it is assumed that the user has selected the multimedia stream TSl, which is stored on the disc 18, for playing. In this case the reading unit 20 reads the data from the disc, the stream separating unit 22 separates overhead data such as the entry point map EPi from the transport stream TSl and provides this information to the control unit 26. The rest of the data, i.e. the transport stream TSl, is forwarded to the decoder 30, which decodes the stream in a normal and well-known fashion. As mentioned before, the user can set up a playlist, where he for instance wants to play the first transport stream TSl first and then the second transport stream TS2. This playlist is entered using the interface 16. One option that exists is that the user can select to connect the transport streams TSl and TS2 seamlessly, i.e. connect the first transport stream TSl with the second transport stream TS2 somewhere in the middle or vice versa. In the present case this is done such that the transport streams are permanently changed and passages provided in the first stream TSl after the exit position and passages in the second stream TS2 before the entry position are removed, thus making the connection permanent. When the user makes such a selection, also called delete part of playlist, he selects an exit time from the first stream and an entry time into the second stream. These selections are made via the interface 16 and thus detected by the control unit 26. As a result of these selections, the control unit 26 furthermore enters markers or points in the form of timestamps into the streams TSl and TS2, which are indicated by a user-selected out time or exit time USEX in the first stream TSl and a user-selected in time or entry time USEΝ in the second stream TS2 in Fig. 3. The relevant area of the first stream TSl around this user- selected exit time USEX and the relevant area of the second stream TS2 around this user- selected entry time USEN are then moved to the buffer 24. The control unit 26 then looks forwards and backwards in the first stream TSl around the user-selected exit time for a section start code. This is done using the corresponding entry point tables associated with the stream, which the control unit 26 has received from the stream separating unit 22. The control unit 26 then determines an actual exit location AEX of the first stream TSl, which points to the first video packet of the frame V₂. It thus connects the user-selected exit point USEX with this section start code. The location selected is the one that is closest to the user-selected exit time in either direction, i.e. backwards or forwards, and thus the closest frame that has a GOP I-frame. Because of the Blu-ray standard this change is never more than 0.5 seconds, which is exact enough for most consumer applications. The control unit 26 also selects a number of multimedia packets to be used in the connection. It selects a number of video packets of the frame V₂ associated with this section start code as well as all the audio packets A₂ associated with this section start code. The frame V₂ contains the last video packet of the first stream TSl that is to be presented. It also selects possible media packets associated with previous section start codes provided between the section start code and the last audio packet of A₂. In the preferred embodiment the type of packets selected here are audio packets. These selected packets have timestamps that are equal to or smaller than the timestamp of frame V₂. Here the packet Ai is thus also selected, which is associated with the video frame Ni of a previous section in TSl. In the same manner the control unit 26 looks at the second stream TS2, in which the control unit 26 has entered an entry point corresponding to the user-selected entry time USEΝ. It searches for a section start code in both directions backwards and forwards from the user-selected entry point USEΝ by looking at entry point tables and then identifies the entry point table EP₂ which points at a section start code associated with a video frame V₃, that includes all necessary video information for a scene, i.e. a GOP I-frame. Also this actual entry point AEΝ is the point having a sequence start code that is the closest to the user selected entry point USEΝ. The control unit 26 then makes this point the actual entry point AEΝ and associates the user-selected entry point USEΝ with this actual entry location.

Thereafter the control unit 26 searches the second transport stream TS2 backwards in order to find the latest streaming control packets SC, i.e. the last such packets before the actual entry location AEΝ, which it selects. These packets are such packets as PCR packets, PAT packets and PMT packets. Thereafter the control unit 26 goes on to modify the two streams TSl and TS2 in the way shown in Fig. 4. The modified first transport stream TS'l remains unchanged up to and including the packets V₂. Thereafter the control unit 26 creates and places an end of section marker or code E after the last video packet of frame V₂. After this end of section code E, it places the audio packets Ai associated with previous video frames of TSl. Thereafter it creates one or more null packets N corresponding to the amount of packets provided between Ai and A₂ in the stream TSl followed by the audio packets A₂. The control unit 26 then adds a number of null packets N in order to pad the audio packets to an aligned unit boundary and to satisfy the requirements of the recording standard used. All other packets of the first stream after the actual exit location AEX are removed by the control unit 26 and thus a modified first transport stream TS'l is created. In the modified second stream TS'2, the control unit 26 keeps the stream control packets SC of the second stream TS2 and provides these just before the selected packets of the first frame of video data V₃ of the second stream TS2. All other packets before the actual entry location AEN are removed and thus a modified second transport stream TS'2 is created. In case the stream control packets SC include PCR packets with video data, this video data is replaced with padding and adaptation flags (MPEGSYS) in order to satisfy Blu-ray standard restrictions. Also in this case it is furthermore possible to provide a number of null packets before the first frame of video data V₃ in order to satisfy the MPEG buffering model. Now that the two transport streams TS' 1 and TS'2 have been modified, also the meta or overhead data associated with the streams have to be modified to reflect the changed situation. The first and second entry point tables EPi and EP₂ are modified into tables EP'i and EP'₂ by deleting the appropriate number of EP entries. The actual SPN (Source Packet Number) value is corrected in order to match the actual clip situation. All other stream information modifications are made according to the requirements of the Blu- ray standard. Thereafter the control unit 26 ensures that the two modified streams TS'l and TS'2, or at least the relevant parts thereof, i.e. the changed parts, are forwarded from the buffer 24 to the stream combining unit 28, to which the modified entry point tables EPi and EP₂ are also supplied, and are combined there and supplied to the writing unit 30 for writing and storage on the disc 18 via the disc drive 14. In this way the user can seamlessly connect the actual entry and exit locations of the two streams by deleting part of the playlist. The described method is advantageous since it does not require any recoding of the streamed data for the streams, but only retaining the relevant information packets together with the addition of null packets or other stream control packets. Because of this, the connection is enabled at low cost and without the above-mentioned complexity. It is furthermore fast and can be performed without any perceivable delays to the user. When the user wants to connect from the first to the second stream, the streams are modified quickly and thus a seamless connection is obtained. The connecting positions are probably slightly offset from the ones selected by the user, which is hardly noticeable in a consumer environment. The invention furthermore does not need an encoder for its implementation, which means that it can be provided in a simpler and cheaper device, like for instance in a Blu-ray player. The connection in the example above was described in relation to sections of two separate streams. It should be realized that the connection can just as well be provided between two different sections of the same stream. The locations of connections could also be chosen virtually anywhere inside the transport streams provided there are section start codes close at hand. In case there is no overhead information at hand, i.e. no entry point tables, the control unit would need to search the actual transport streams to find the section start codes. If there is no such overhead information there will also be no stream separating unit or stream combining unit in the device for interconnecting multimedia stream sections. The control unit is preferably provided in the form of a processor with an associated program code for performing the method according to the invention. This program code can also be provided on a data carrier, such as a data carrier 34 in the form of a CD Rom disc as shown in Fig. 5. The program code can furthermore be provided on a server and downloaded into the multimedia presentation device. The multimedia presentation device furthermore need not be a Blu-ray disc player, but can for instance also be a computer like a PC (Personal Computer). It should be emphasized that the term "comprises/comprising" when used in this specification is taken to specify the presence of stated features, integers, steps or components, but does not preclude the presence or addition of one or more other features, integers, steps, components or groups thereof. It should furthermore be realized that reference signs appearing in the claims should in no way be construed as limiting the scope of the present invention.

Claims

CLAIMS:

1. Method of connecting together two multimedia stream sections comprising the steps of: detecting user-selected switching from a first to a second section at a certain exit time (USEX) of the first section (TSl) and a certain entry time (USEN) of the second section (TS2), determining an exit location (AEX) based on the user-selected exit time in the first section, which exit location has a sequence start marker, selecting all media packets (V₂, A₂) in the first section associated with the sequence start marker and possible media packets (Ai) provided after the exit location associated with previous sequences in the first section, removing non-selected media packets after the exit location in the first section, creating and placing a sequence end marker (E) in the first section, determining an entry location (AEN) based on the user-selected entry time in the second section, which entry location has a sequence start marker, selecting stream control information (SC) provided in the second section before the entry location, removing non-selected packets before the entry location in the second section, and adding filling packets to the two sections in a number that meets coding requirements, such that the two sections (TS'l, TS'2) can be seamlessly interconnected at the entry and exit locations.

2. Method according to claim 1, wherein the actual entry and exit locations are the locations being closest to the user-selected entry and exit times.

3. Method according to claim 1, wherein the selected stream control information is the closest stream control information provided in previous sequences of the second section.

4. Method according to claim 1, wherein the media packets comprise video packets (Ni, N₂, N₃) and the sequence end marker is placed after the last video packet (N₂) in the first section associated with the sequence start marker of the exit location.

5. Method according to claim 1, wherein the media packets comprise audio packets (Ai, A₂, A₃), said method further comprising the step of inserting null packets and/or stream control packets in positions originally occupied by non-selected media packets in the first section between said exit location and the last audio packet associated with the sequence start marker.

6. Method according to claim 1, further comprising the step of adding null packets to the stream control packets in the second section before the sequence start code.

7. Method according to claim 1, wherein at least said exit location of the first section and the entry location of the second section each comprise an entry point map (EPi,

EP ), which are used for identifying exit and entry locations.

8. Method according to claim 7, further comprising the step of changing the entry point map (EP'i) of the first section and the entry point map (EP'₂) of the second section in order to reflect the changes made to the sections.

9. Method according to claim 1, wherein the sections are provided in the same multimedia stream.

10. Method according to claim 1, wherein the sections are provided in two different multimedia streams.

11. Device (12) for interconnecting two multimedia stream sections provided on a storage medium (18) and comprising: a reading (20) and a writing (30) unit for reading and writing multimedia data on the storage medium (18), and a control unit (26) for manipulating at least one multimedia stream provided on the storage medium, wherein the control unit is arranged to: detect user-selected switching from a first to a second section at a certain exit time (USEX) of the first section (TSl) and a certain entry time (USEN) of the second section (TS2), determine an exit location (AEX) based on the user-selected exit time in the first section, which exit location has a sequence start marker, select all media packets (V₂, A₂) in the first section associated with the sequence start marker and possible media packets (Ai) provided after the exit location associated with previous sequences in the first section, remove non-selected media packets after the exit location in the first section, create and place a sequence end marker (E) in the first section, determine an entry location (AEN) based on the user-selected entry time in the second section, which entry location has a sequence start marker, select stream control information (SC) provided in the second section before the entry location, remove non-selected packets before the entry location in the second section, and add filling packets to the two sections in a number that meets coding requirements, such that the two sections (TS'l, TS'2) can be seamlessly interconnected at the entry and exit locations.

12. Device according to claim 11, further comprising a stream separating unit (22) and a stream combining unit (28) for providing a stream of additional overhead data related to a multimedia stream.

13. Multimedia data presentation device (10) comprising a device (12) for interconnecting two multimedia stream sections according to claim 11.

14. Computer program product (34) to be used on a computer for interconnecting two multimedia stream sections and comprising a computer program code for making the computer execute, when said code is loaded into the computer, the following functions: detect user-selected switching from a first to a second section at a certain exit time (USEX) of the first section (TSl) and a certain entry time (USEN) of the second section (TS2), determine an exit location (AEX) based on the user selected exit time in the first section, which exit location has a sequence start marker, select all media packets (V₂, A ) in the first section associated with the sequence start marker and possible media packets (Ai) provided after the exit location associated with previous sequences in the first section, remove non-selected media packets after the exit location in the first section, create and place a sequence end marker (E) in the first section, determine an entry location (AEN) based on the user selected entry time in the second section, which entry location has a sequence start marker, select stream control information (SC) provided in the second section before the entry location, remove non-selected packets before the entry location in the second section, and add filling packets to the two sections in a number that meets coding requirements, such that the two sections (TS'l, TS'2) can be seamlessly interconnected at the entry and exit locations.