WO2017055876A1

WO2017055876A1 - Video and/or audio data processing system

Info

Publication number: WO2017055876A1
Application number: PCT/GB2016/053063
Authority: WO
Inventors: Patrick Christian
Original assignee: Culloma Technologies Limited
Priority date: 2015-10-02
Filing date: 2016-10-03
Publication date: 2017-04-06
Also published as: GB201517411D0

Abstract

The invention relates to apparatus and a method for the adaptation of a received video and/or audio datastream. One or more portions of the datastream which are not appropriate for viewing at one or more predetermined viewing locations, such as a particular geographical location is identified and markers inserted and passed to further processing means at which the portions of data identified by the markers can be removed and replaced with data which is selected as being more appropriate with respect to the subsequent viewer location for that video stream. In one embodiment the identified portions are adverts for goods or services which may be appropriate for broadcast in the country in which the service or channel is originally provided, but are not appropriate for broadcast in a different country in which the said service or channel is subsequently provided.

Description

Video and/or Audio Data Processing System

The invention which is the subject of this application relates to the field of generation of video, audio and/or auxiliary information from digital data which is transmitted from a head end by a broadcaster to a plurality of end users.

The transmission of television services, comprising video and associated audio, has always been considered as a continuous stream in that the video images are carried as a sequence of frames which are sent at a uniform rate, without break, from the beginning to the end of the transmission. The delivery of audio is even more uniform, as, even when encoded digitally, the sound is represented as a continuous sequence of bytes with only a start and end of the transmission. When the video is encoded digitally, which is now commonly the case, the concept of frames is used as part of the compression and encoding process. As a result of this, the majority of the frames contain data which is not the actual image of that frame itself, but rather the differences between the image of that frame and at least one of its immediate neighbouring frames.

This continuous stream approach typically also includes advertising in which a number of adverts are typically provided sequentially at the end/start of programmes and/ or at time intervals during the programme.

The adverts which are shown may be selected to be provided on a relatively wide scale basis such that the same adverts are shown to viewers in the same country or region or on an international basis. Alternatively all or certain adverts may be selected to be shown on a regional basis so as to provide the adverts to a more focus sed group of viewers. What is most typically the case is that the adverts which are provided are provided with reference to the identified geographical viewer group for the television or internet channel. For example, a television or internet channel which is regarded as an Egyptian television or internet channel will show adverts which are directed towards products or services which are available in Egypt. While this approach may be appropriate for the majority of the viewers of that channel, it is increasingly the case that viewers in other, geographically remote, regions may be provided with the ability to view the channel as a result of accessing and/ or subscribing to an in internet based service which relays the channel to them to view, regardless of the geographical location of the viewer. While the provision of the service is of great advantage to the remote viewers, it means that the advertising that the viewers are presented with in these geographically remote locations is of little relevance to them and, even if the viewer was interested in a particular advertised product or service the same may not be available or accessible to them.

In addition to the frustration which this can cause to the viewer and the annoyance which is created by having to watch these adverts, especially when they are of no relevance, the showing of these non-relevant adverts to the viewer represents a loss in potential advertising revenue earnings which the internet service provider could make if they could sell advertising space to companies to show geographically relevant adverts to the viewers.

It should also be appreciated that although a significant issue does relate to advertising, the same problems can also relate to other types of programming, such as, for example, the provision of news programmes which are relevant to the locality at which the television service was initially directed but is of no relevance to viewers at a geographically remote location.

An aim of the present invention is to provide a method using video and/or audio recognition techniques to accurately identify television advertisements or other portions of video and/or audio data such that these may be amended as appropriate to insert geographically or subject-relevant material in place of the original material which would have been shown at that time.

In a first aspect of the invention there is provided apparatus for the transmission of content in the form of video and/or audio digital data which represents a programme service or channel, said apparatus including receiving means to receive one or more video and/or audio data streams wherein identification means are provided to identify one or more portions of the data and insert markers to identify the said portions and then forward the said data including said markers for onward provision to one or more viewers and said portions are identified with reference to one or more detectable parameters which are present in the received data..

In one embodiment said parameters, in one embodiment, indicative of one or more adverts which are provided as part of the received channel or service.

In one embodiment the said portion is identified for the video data and corresponding audio data and markers are inserted for both portions.

Preferably the identification means and marker insertion means are provided within a server which receives, adapts and transmits the adapted video and/or audio data streams.

Typically the apparatus includes data insert means downstream which selectively substitute video and/or audio data in those portions which are indicated by the inserted markers. In one embodiment the data insert means makes reference to the subsequent viewer destination to which the data for the service or channel is to be transmitted. Upon making this identification video and/ or audio data is retrieved from a database and inserted as a substitute to said identified portion and the data is then transmitted to the viewer destination.

In one embodiment the subsequent viewer destination is identified geographically.

In one embodiment the said database includes portions of video and/ or audio data which are identified with respect to their relevance to specific geographical regions and upon a match between the subsequent viewer destination and the specific geographical region in the database, the appropriate portion of data is selected from the database.

In one embodiment the apparatus is scalable so as to allow the identification and insertion of markers for portions of data on a number of streams of video and/or data for a plurality of programmes or services which are received simultaneously.

In one embodiment the identification means and marker insertion means are provided within a server which receives, adapts and transmits the adapted video and/ or audio data streams.

In one embodiment the capacity of the server is dependent upon any or any combination of the number of channels, number of portions, such as adverts, identified per hour and/ or the lengths of the identified portions in time.

In one embodiment the server is able to simultaneously process data for in the region of 10 to 20 channels. In a further aspect of the invention there is provided a method for the adaptation of received video and/or audio data, said adaptation occurring intermediate the location from which the video data is broadcast and the subsequent viewer location for the video data and wherein said adaptation includes the insertion of markers to identify at least one portion of said video data.

In one embodiment the adaptation and insertion are made with respect to at least one predetermined parameter.

In one embodiment the video represents a television programme service or channel.

Typically the method includes the steps of receiving video and/or audio data streams for the said service or channel, and identifying one or more portions of the data which match certain predetermined criteria.

Typically the method includes the steps of inserting markers to identify the said portions and then forwarding the adapted data stream including said markers for onward provision and/or further adaptation by the removal and/or insertion of portions of data. The markers may also be carried separately from the data.

In one embodiment the subsequent adaptation is the insertion of an alternative portion of data such as the insertion of alternative adverts.

In one embodiment the data is received in an encoded format and is required to be at least partially decoded in order to identify the portions. In one embodiment the method allows the identification of portions of data in a data stream in the form of adverts and inserting respective markers in the data stream to identify the portions and subsequently the markers are used to identify that part of the data stream in which replacement adverts which are of greater relevance to the subsequent viewer locations are inserted.

In one embodiment the markers identify the start and end of said portions. In one embodiment the insertion of the replacement adverts occurs downstream of the stage of inserting the markers.

In one embodiment the said portions of data are recognised using video recognition based on the use of advert 'signatures'.

An advert signature is a collection of numbers which is used as part of a mathematical operation applied to the video data, typically in real time, as the data stream is received.

Typically a numerical result is generated which indicates how well the video data and the signature match.

Typically this process is performed on every frame of the received video data and compared with every signature held in a memory and if ever the result of the comparison is above a predetermined threshold value, a 'match' or 'hit' is flagged.

Typically when a match or hit is identified the insertion means insert a marker, typically an SCTE35 marker, into the data transport stream at the appropriate location. Alternative mechanisms may also be used. In one embodiment, in addition to insetting a marker the identification of the "hit" or "marker" is also used to generate other actions, including, for example, sending an email alert.

Typically, for every channel or service the method includes at least monitoring the input video data stream to identify any signature matches and, when identified, inserting markers which are in line with the Society of Cable and Telecommunications Engineers 35 Standard (SCTE35) into the datastream to identify the relevant portion of data and/ or recording video sequences to identify candidates for new advert signatures.

Typically in addition the method includes receiving video records or 'snippets' from those which have been recorded by the method, or from external sources and attempting to generate new advert signatures.

Typically the recording of 'snippets' is triggered by video matching a signature that has previously been created. Thus, for example, an entire commercial break of several adverts may be recorded just by the detection of a single advert within that break.

Typically, signatures for adverts that are no longer in use are deleted over time.

Typically the method includes the steps of monitoring advert recognition activity and consolidating 'as run' reports for onward delivery to the customer.

Typically the reference and operating data in the form of signatures, reports, and the like are stored in a hierarchical database which is mirrored on a second database so that the same are synchronised. Typically the input video stream is encoded in either MPEG2 or H.264 formats and carried as MPEG-TS in multicast IPv4 over Ethernet

Typically a presentation time stamp (PTS) is present in the transport stream for every input video frame and preferably the input video stream is substantially error free and has a substantially constant frame rate with substantially no jitter.

In certain geographical regions no markers are required to be inserted in video streams to identify adverts and even in those regions where markers are inserted at the time of broadcast the number of advert markers which are inserted are significantly less than is normally required. These problems are overcome by the apparatus and method of the current invention.

In one embodiment the method includes the steps of selectively adapting video data at a location intermediate the head end from which the data is broadcast and the subsequent viewer locations, said adaptation including the insertion of one or more markers to identify a portion of the said data.

In one embodiment the said adaptation further includes the step of replacing the said identified portion of data with another portion of data and transmitting the adapted video data to the subsequent viewer locations.

In one embodiment the video data is represented by frames, and said frames are divided into Groups of Pictures (GOP) and wherein a GOP and audio therefore are represented as a group, and the start of the group is determined by the detection of a pre-determined parameter with respect to the transmitted content or the content which is to be transmitted, such that upon the detection of the same the current group is closed and a new group is started. In one embodiment the identification of the portion is performed in accordance with the system and method as identified in the co-pending European patent application EP2543188 the contents of which are included herein.

Specific embodiments of the invention are now described with reference to the accompanying drawings wherein;

Figure 1 illustrates in a schematic manner the apparatus components and method in accordance with one embodiment of the invention; and

Figure 2 illustrates the manner in which different versions of the television service can be generated in accordance with one embodiment of the invention.

The invention relates to the transmission of programme content in the form of video and/or audio data. MPEG2, H.264 and other encoding and compression mechanisms for video transmission which minimize the amount of bandwidth required to carry a video sequence by fully encoding a single frame (often called an I-frame, anchor frame or reference frame) and then encoding the subsequent frames (P- or B-frames) as a sequence of frames which only include data for the differences or deltas from that reference frame (the I frame) and/or other neighbouring frames.

Data for the programme content is typically broadcast from one or more broadcast locations (the head end) and is transmitted to a plurality of subsequent viewer locations. The invention is particularly directed towards the distribution system where the broadcast data is received at an intermediate location in a first format such as via satellite, cable, terrestrial or internet and then transmitted onwards from the intermediate location to the subsequent viewer locations, most typically, although not necessarily exclusively, via internet transmission means. The invention provides for the adaptation of the said data at the intermediate location, most typically to allow the data to be adapted to make the service provided more relevant to the respective subsequent viewer locations.

An example of this system is provided in Figure 1. In Figure 1 there is illustrated by arrow 2 the stream of data which is received from one or more headends. At the intermediate location there is provided a server 4 and the video data is passed through a video recognition step based on the use of advert signatures 6. If the recognition step identifies a match between a signature held in its database 8, which typically will have signature representing known adverts, with a portion of the video data which has been received this will trigger the insertion of SCTE35 'markers' into the transport stream for the video data to indicate the relevant portion of data so that the output video data 10 will include the inserted markers therein.

In addition in the analysis of the input video data 2, identified data snippets will be recorded 12 and these recorded snippets are used as new portions of data for which a new signature 16 is required to be generated and the new signatures are added to the database 18 via path 24 The database 18 and database 8 are synchronised such that both will hold the same signatures at the same time. In addition as the stored signatures get older so the same may be removed from the databases to make space for new signatures to reflect the changes in adverts which are displayed on the incoming video stream 2.

The system also has an as-run generator to maintain an as-run file 20 for the video for every TV channel that is active. The file is typically in comma-separated- variable (CSV) format and has a record of every ad recognized on that channel that day. Typically a new file is started for each channel every day. Typically a secure link between the client 26 and host 28 remains available at all times.

The host 28 can also be used to generate reports 30, alerts 32 and real time monitoring 34 of the performance of the system.

The adapted video data which is output with markers 10, can then be passed to further processing means 36 at which the portions of data identified by the markers can be removed and replaced with data which is selected with respect to the subsequent viewer location for that video stream. For example, an identified portion in the data stream may relate to an advert for goods which are only available in a country from which the television channel originates, but the goods are not available in the country of the subsequent viewer location and to which the vide o and/ or audio data is to be onwardly transmitted. As such the adaptation of the video data in accordance with the invention allow the original advert location to be identified and replaced by an advert which is locally relevant to the subsequent viewer location prior to the data for the service being passed on to the subsequent viewer location.

Thus, as illustrated in Figure 2 in accordance with the invention the input video data stream 2 from the headend 38 is adapted at the client 26 to include markers to identify portions 40 of data and then subsequently those portions of data can be selectively replaced at 36 with an alternative first portion of data for a first subsequent viewer location 42, an alternative second portion of data for a second subsequent viewer location 44, an alternative third portion of data for a third subsequent viewer location 46 and so that many different versions of the input data stream may be generated.

Claims

1. Apparatus for the transmission of content in the form of video and/or audio digital data which represents a programme service or channel, said apparatus including receiving means to receive one or more video and/or audio data streams wherein identification means are provided to identify one or more portions of the data and insert markers to identify the said portions and then forward the said data including said markers for onward provision to one or more viewers and wherein said portions are identified with reference to one or more detectable parameters which are present in the received data..

2. Apparatus according to claim 1 wherein said parameters are indicative of one or more adverts which are provided as part of the received programme service or channel.

3. Apparatus according to claim 1 wherein a said portion is identified for the video data and a portion is identified for corresponding audio data and markers are inserted for both portions.

4. Apparatus according to claim 1 wherein the identification means and marker insertion means are provided within a server which receives, adapts and transmits the adapted video and/or audio data streams.

5. Apparatus according to claim 1 wherein the apparatus includes data insert means downstream of the marker insertion means which selectively substitute video and/ or audio data in those portions which are indicated by the inserted markers.

6. Apparatus according to claim 5 wherein the data insert means refer to the subsequent viewer destination to which the data for the service or channel is to be transmitted and video and/or audio data is retrieved from a database and inserted as a substitute to said identified portion and the data is then transmitted to the said subsequent viewer destination.

7. Apparatus according to claim 6 wherein the subsequent viewer destination is identified geographically.

8. Apparatus according to any of the preceding claims wherein the said database includes portions of video and/ or audio data which are identified with respect to their relevance to specific geographical regions and upon a match between the subsequent viewer destination and a specific geographical region in the database, an appropriate portion of data is selected from the database.

9. Apparatus according to claim 1 wherein the apparatus is scalable so as to allow the identification and insertion of markers for portions of data on a number of streams of video and/ or data for a plurality of programmes or services which are received simultaneously.

10. Apparatus according to claim 9 wherein the server is able to simultaneously process data for in the region of 10 to 20 channels.

11 Apparatus according to any of the preceding claims wherein the identification means and marker insertion means are provided within a server which receives, adapts and transmits the adapted video and/ or audio data streams.

12. Apparatus according to claim 1 wherein the capacity of the server is dependent upon any, or any combination, of the number of channels, number of portions identified in a predetermined time period and/or the lengths of the identified portions in time.

13. A method for the adaptation of received video and/or audio data, said adaptation occurring intermediate the location from which the video data is broadcast and the subsequent viewer location for the video data and wherein said adaptation includes the insertion of markers to identify at least one portion of said video data.

14 A method according to claim 13 wherein the adaptation and insertion steps are made with respect to at least one predetermined parameter.

15 A method according to claim 13 wherein the video represents a television programme service or channel.

16 A method according to claim 13 wherein the method includes the steps of receiving video and/or audio data streams for the said service or channel, and identifying one or more portions of the data which match predetermined criteria.

17 A method according to claim 13 wherein the method includes the steps of inserting markers to identify the said portions and then forwarding the adapted data stream including said markers for onward provision and/ or further adaptation by the removal and/or insertion of portions of data.

18 A method according to claim 17 wherein the markers transmitted separately to the data.

19 A method according to claim 17 wherein the subsequent adaptation is the substitution of adverts which are located in the said identified portion with an alternative portion of data.

20 A method according to claim 19 wherein the alternative portion of data is an advert which differs to the original advert in the identified portion.

21. A method according to claim 13 wherein the data is received in an encoded format and is required to be at least partially decoded in order to identify the portions.

22. A method according to claim 13 wherein the method allows the identification of portions of data in a data stream in the form of adverts and inserting respective markers in the data stream to identify the portions and subsequendy the markers are used to identify that part of the data stream in which replacement adverts which are of greater relevance to the subsequent viewer locations are inserted.

23 A method according to claim 22 wherein the insertion of the replacement adverts occurs downstream of the stage of inserting the markers.

24. A method according to any of claims 13-23 wherein the said portions of data are recognised using video recognition based on the use of advert 'signatures'.

25 A method according to claim 24 wherein the advert signature is a collection of numbers which is used as part of a mathematical operation applied to the video data as the data stream is received.

26 A method according to any of claims 24-25 wherein a numerical result is generated which indicates how well the video data and the signature match.

27 A method according to any of the claims 24-26 wherein the process is performed frame by frame of the received video data and compared with signatures held in a memory and if the result of the comparison is above a predetermined threshold value, a 'match' or 'hit' is flagged.

28. A method according to claim 27 wherein when a match or hit is identified the insertion means insert a marker into the data transport stream at the appropriate location.

29 A method according to claim 28 wherein an SCTE35 marker is inserted.

30 A method according to any of the claims 13 -29 wherein in addition to inserting a marker the identification of the "hit" or "match" is also used to generate at least one further action.

31. A method according to any of the claims 13-20 wherein for every channel or service the method includes at least monitoring the input video data stream to identify any signature matches and, when identified, inserting markers into the datastream to identify the relevant portion of data and/or recording video sequences to identify candidates for new advert signatures.

32. A method according to any of the claims 13-31 wherein the method includes receiving video records or 'snippets' from those which have been recorded by the method, or from external sources and generating new advert signatures. 33 A method according to claim 32 wherein the recording of a snippet is triggered by a hit or match.

32 A method according to claim 33 wherein a plurality of adverts for a commercial break are recorded upon the detection of a hit or match relating to a single advert.

33 A method according to any of claims 13-32 wherein the reference and operating data in the form of signatures and/ or reports are stored in a hierarchical database which is mirrored on a second database so that the same are synchronised.

34. A method according to any of the claims 13-33 wherein the video data is represented by frames, and said frames are divided into Groups of Pictures (GOP) and wherein a GOP and audio therefore are represented as a group, and the start of the group is determined by the detection of a pre-determined parameter with respect to the transmitted content or the content which is to be transmitted, such that upon the detection of the same the current group is closed and a new group is started.

35. A method according to any of claims 13-34 wherein the particular portion which is used as the substitution is selected with reference to the known geographical location of the subsequent viewer or group of viewers to which the adapted video is to be onwardly transmitted.