IES20030840A2

IES20030840A2 - Multimedia management

Info

Publication number: IES20030840A2
Application number: IE20030840A
Authority: IE
Inventors: Alan Smeaton
Original assignee: Aliope Ltd
Priority date: 2002-11-08
Filing date: 2003-11-10
Publication date: 2004-05-05
Also published as: WO2004043029A2; WO2004043029A3; AU2003278558A8; AU2003278558A1; IE20030839A1

Abstract

A multimedia management system (1) receives live stream content at an interface (5) or static content (6) and captures (7) it to a storage device (10). An analysis component (11) in parallel or in series generates meta data for a database (12), the meta data acting as an index for the storage device (10). Automatic detection of events by the analysis component (11) allows provision of real time distribution of content such as news clips to fixed and mobile networks (MS). Near real time media services are also provided with some annotations being performed by an annotation component (14) under control of a browser (13). <Figure 2>

Description

The invention relates to management of multimedia such as digitised audio-visual content.

Prior Art Discussion In recent years advances have been made in the field of distribution of multimedia in mobile networks. For example the Multimedia Messaging Service (MMS) standard for mobile networks specifies peer-to-peer multimedia messaging. United States Patent Application No. US200205/442 describes a method of transmitting/receiving a broadcast message in a mobile network.

Also, advances have been made in processing of multimedia in various aspects, such as indexing and browsing. For example, the paper “Fischlar: an on-line system for indexing and browsing broadcast television”, O'Conndr N, Marlow S, Murphy N, Smeaton A, Browne P, Deasy S, Lee H and Me Donald K. ICASSP 2001 International Conference on Acoustics, Speech, and Signal Processing. Salt Lake City, UT, 7-11 May 2001., describes indexing and browsing. Television programmes are captured in MPEG-1 format and analysed using video indexing tools. Browsing interfaces allow a user to browse a visual index to locate content of interest, and selected content can then be streamed in real time.

While these developments are of benefit, there is still a need to provide more comprehensive multimedia asset management to support content-based operations for a broader range of devices and access mechanisms.

Open to pubuc inspection UNDER SECTION 28 ANL RULE 23 IE 0 3 0 R 4 Q -2SUMMARY OF THE INVENTION According to the invention, there is provided a multimedia management system comprising: a multimedia content capture component for receiving content and for writing it to a storage device; an analysis component for analysising received content to generate meta data, and a database for storing the meta data; a server component for distributing content or meta data to a network for delivery to subscriber devices; and a controller for coordinating operation of the components of the system to provide configured services for delivery of content to subscriber devices.

In one embodiment, the analysis component generates the meta data as an index to the content in the storage device and the server uses the meta data to access the content.

In another embodiment, the analysis component extracts key frames from received video content and segments audio and video streams.

In a further embodiment, said key frames are stored to provide a storyboard of video content with images forming part of an index. -3£0308 40 In one embodiment, the analysis component operates in parallel with capture of content by the capture component for analysis in real time as a video content stream is being received.

In another embodiment, the analysis component uses a shot boundary detection technique to generate the meta data.

In a further embodiment, the analysis component automatically detects events in incoming content and generates notifications in real time.

In one embodiment, the controller receives the notifications and automatically controls components of the system to generate a content output triggered by the detected event.

In another embodiment, the notification is an alert message transmitted to a mobile station.

In a further embodiment, the system further comprises an annotation component for annotating content and directing storage of annotated content in the storage device.

In one embodiment, the annotation component operates in near real time.

In another embodiment, the controller directs output via the server of content in near real time to provide a subscriber service.

In a further embodiment, the system further comprises a browser connected to the storage device and to the database, the browser allowing supervisor access to content to control processing of the content.

|E 0 S 0 fi 4 0 -4In one embodiment, the browser is connected to the annotation component and directs operation of the annotation component to provide processed content in the near real-time service.

In another embodiment, the system further comprises a transcoding component connected to the browser and being for transcoding of content for delivery to the analysis component and to the storage device.

In a further embodiment, the controller directs routing of meta data from the database to a plurality of different devices, including local devices directly connected to the controller and remote subscriber devices.

In one embodiment, the analysis component writes some of the meta data to the storage devices to provide an addressing index.

In another embodiment, the controller uses meta data solely from the database for directing content to some subscriber devices.

In a further embodiment, the controller dynamically generates media interfaces with style sheet transformation, in which style sheets dynamically transform the media and code into user-viewable media and display code.

In one embodiment, the transformation performs independent processing of screen windows in a manner analogous to application or operating system display windows.

In another embodiment, the style sheet transformation code is separate from underlying functionality of the controller and other components involved in media output. -5In a further embodiment, the system further comprises a live streaming interface connected to the content capture component.

In one embodiment, the system further comprises a scheduling component connected to the live stream circuit for activating the live stream circuit and setting recording parameters for both the live stream circuit and the capture component.

DETAILED DESCRIPTION OF THE INVENTION Brief Description of the Drawings The invention will be more clearly understood from the following description of some embodiments thereof, given by way of example only with reference to the accompanying drawings in which :Fig. 1 is a block diagram of a multimedia management system of the invention.

Fig. 2 is a diagram showing interfacing modes; and Fig. 3 is a diagram illustrating media analysis, generation of meta-data, and presentation for content-based operations such as access, retrieval, sharing consumption.

Description of the Embodiments Referring to Fig. 1 a multimedia management system 1 of the invention comprises a controller 2 which controls in a comprehensive manner various multimedia processing components to provide a variety of services. These components are as follows.

IE 0 3 0 Λ 4 0 -65: A live stream interface for receiving broadcast digital TV channels. 6: A reader device for reading multimedia content from a static, non-live source such as DVDs. 7: A capture circuit for encoding and compressing content to a format such as MPEG.

: A content storage device for storing the encoded and compressed content files together with related meta data. The latter includes images, hyperlinks, log files for data such as start/end time of capture, programme meta-data, user account information, user preferences, copyright information, number of key frames extracted, file size, and channel meta data. Some of the meta data is generated by the capture circuit 7, and the remainder by an analysis component 11. 11: An analysis component 11 which extracts representative meta-data such as key frames, audio, text, from compressed audio-visual content. Extracted meta-data is encoded in formats and resolutions (for example JPEG, GIF, MPEG) of any size that can be displayed, selected, and interacted with. The analysis component 11 both extracts existing meta data, and generates meta data which is stored in the storage device 10. However, it also writes meta data to a database 12, set out below. Additional meta data related to the precision and accuracy of analysis techniques used against audio-visual content is generated by the analysis component 11 using the shot boundary technique to generate colour histogram, edge detection, and motion analysis data. 12: A content database for storing all meta data generated by the analysis component 11. Some of the meta data is also stored with the actual content in ΙΕ ο 3 0 8 4 ο -Ί the storage device 10, as outlined above. The database 12 stores the meta data in a manner to provide an index for the storage device 10. 13: A browser for use by local/remote staff of the organization hosting the system 1. This is used for accessing the content using indexes of the database 12. It activates interfaces for content-based operations (non-linear playback, annotation) accomplished over end-user devices. The browser 13 can also route content to a transcoding component 15. 14: An annotation component for annotation of stored content under instruction from the browser 13. 16: A scheduling component for scheduling operation of the interface 5 for receiving and encoding live streams. 17: A Web server for routing of content to infrastructure for fixed and mobile networks for onward routing to central/edge servers and end user playback devices.

The multimedia management processes performed by the system are now described. One of the problems facing operators of communication systems such as mobile network operators is that while there is a vast amount of content available, it has not been possible to manage it so that content owners and aggregators, network operators and end users get optimum use of it. It is not just a matter of coping with the limited communication bandwidths and device capabilities of many user devices, but also syndicating content which end users wish to access.

The system 1 addresses this problem by acting as a platform which takes in the multimedia content and manages it in an optimum manner so that end users can access/receive what they subscribed to. For example, security camera systems -8IE030840 generate a vast quantity of content, however, an end user is only interested in seeing suspicious activity. Thus, the analysis component 11 may automatically monitor live streams from such cameras and generate an alert for the controller 2 if a suspicious image and/or sound pattern is detected. The controller in turn notifies in real time the Web server 17 to retrieve the content meta data and push it onto either the fixed or mobile network to the end user subscriber playback devices. One suitable bearer for such alerts is GSM SMS to mobile devices.

In other situations some human input may be required such as downloading of a video sequence of a breaking news event or a score in a football match. In this case, staff monitor certain content channels and access audio-visual segments in non-linear fashion to achieve content-based operations for seamless syndication, transmission and retrieval over fixed and mobile networks. The content is cropped and rendered to be suitable for transmission to the subscriber’s mobile station MS. Such operations can typically be carried out in a few seconds, thus providing a near realtime service to the subscriber.

Fig. 2 illustrates the versatility of the system 1, providing content to a wide variety of devices to present a variety of interfaces. These include a portal interface 20, a search interface 21, a browsing interface 22, a playback interface 23, an annotation interface 24, remote user device browsing/swapping interfaces 25, and a remote user device playback interface 26. This diagram illustrates versatility of the architecture of the system 1. The controller 20 is at the centre allowing interfacing with both staff of the hosting organisation and remote end users in a versatile manner.

The controller 2 includes a database of access control data for interfacing with staff. It allows supervisor access to some of the components.

The controller 2 controls automatic deletion of content according to configured parameters. -9IE 0 3 0 Β 4 0 The analysis component 11 generates the meta data is a manner whereby it forms a content index that is represented using images of video content and meta data referenced to additional related media content. Capture and analysis of content can take place in parallel, thus enabling text, image and video distribution without the need to wait for a programme to end before managing distribution to a mass audience.

The analysis component 2 generates the meta data for some content in the form of storyboards. Live and stored media files are analysed using algorithms processing specific data streams/layers, multiplexed/encapsulated in analogue and standard media compression formats (for example MPEG formats). Each algorithm can be controlled by a dynamic threshold/parameter set to analyse different types of media information (for example video with slow-movement and short cuts compared to video with fast motion and fades). The analysis techniques are used to extract representational content from media files for indexing based on natural breaks or events in the media content. Analysis results derived from files composed of different type of media information (for example news programme with newscaster and news agencies footage) are based on a combination of algorithms in order to generate the most logical representation of programme’s content for specific playback devices.

The analysis component 11 analyses whole data streams/formats and media layers by segmenting audio, detecting faces, detecting edges, detecting black frames, and generating colour histograms, The combination of these algorithms detects video frames and video shots that are sequenced in a logical scene if media programmes follow a structure at the time of production.

Referring to Fig. 3, a media file 40 is analysed in step 45 to generate content-based meta-data 46, which is then automatically stored in the storage device 10. The metadata 46 is retrieved using a style sheet 41 that can be marked-up in representation -10ΙΕΟ 3 Ο * 4 Ο languages (for example HTML, CHTML, WML, SMIL and XML languages) useful for static/dynamic representation of media information to media devices (for example PC, PDA, and mobile phone). Dynamic representation of media content provides the basis for progressive download of media information to media devices dependent on progressive and fast retrieval of media information over narrowband networks. Thus, media data is presented or exchanged between users by communicating representational information, meta-data information, and location information rather that communicating the media data itself.

The controller 2 acts as an interface between the storage device 10 and the web server 17, the Web browser 13, and a streaming server reading commands from hyperlinks and animated media formats. Such commands are sent over the fixed and mobile network to playback devices. The Web browser 13 is automatically refreshed and mapped against media files, metafiles and meta-data found in the database 12.

Requests may be made from subscriber devices to achieve content-based operations. Each content-based operation is completed by connection (circuit switched and/or packet based) to a local media server and/or remote server depending on the availability of bandwidth that the mobile device can connect to and the location of subscriber requesting the content. Media programmes and/or media clips are subsequently and automatically sent over packet-based networks to subscriber devices in synchronised, downloadable or streamable formats that can be played using messaging software, browsers or media decoders developed specifically for the media device.

Access to interfaces and media information in the system 1 is monitored at different levels. Content-based operations requested by a subscriber can only be fulfilled after inputting a user ID. Monitored access generates information on usage of the system 1 and the media data content. Such information can be used to improve media distribution on a personalised basis. -11ΙΕΟ 3 Oft 4 θ Recording functions integrated in the scheduling component 16 provides system users and end users with recording options. Recording requests could be made to the system by clicking on hyperlinked media information such as programme title, and programme meta-data. End users can also rate media programmes in order to receive programme recommendations from the system.

Search interfaces with media highlights in an overview/detail view interface are also generated based upon the meta data. Media sequences are highlighted based on a search query and they can be browsed using a timeline bar in Detail View Media interface.

The system 1 also generates a search interface with a detail view based on text results and slide show media interface enabling system users to narrow down search results using text information (for example closed caption, annotation) tagged to indexed audio-visual content. A search interface for audio-visual information enables system users and end users to search audio-visual information indexed using different analysis techniques. Searches can be applied against different parameters populated by analysis techniques such a face detection, closed caption parsing, and shot detection. Search results are ranked and are based on representative meta-data. An overview media interface implements “overview first, zoom and filter, then details on demand”. This interface provides an overview of media files using representative meta-data, thus indicating to the end user what the audio-visual information is about.

The system 1 also generates a scroll bar media interface. System users browse media content using a scroll bar that acts as a very versatile controller. Clicking on the arrows of the scroll bar moves the page by each row of representative keyframes, clicking on the empty bar jumps the page by screenful, and dragging the square object provides immediate and continuous movement of representative keyframes thus allowing finer and more pleasant experience. ΙΕ ο 3 0 β 4 0 -12Another interface is an extension of the overview media interface. This is similar to the overview/detail browser, but allows movement into a detail view. Once in a detail view, a timeline bar is used for immediate system reaction and finer control in browsing. The timeline bar can be static or dynamic. Both types of timeline bars are useful when browsing a large amount of media information because it provides temporal orientation by showing the current point of browsing in relation to the whole media file. Not only useful as a time indicator, the bar in this browser allows random access by selecting related sections of the bar. The interface with the dynamic quick bar capabilities saves tedious and continuous clickings that the static timeline bar browser may require. One sweep of the mouse allows viewing of all of the keyframes in the video. Much finer control and continuous, immediate response from the system provides 'direct manipulation'. The browsing task is less of an 'actand-react' interaction, and is more natural and paced by the user.

A slide show media interface flips through all representative indexed meta-data in a single slot. Temporally presenting representative meta-data (rather than spatially) saves screen space considerably, and is thus useful for small end user devices. People with more dynamic interaction preference may also like this browser. It is important to provide some form of temporal orientation in this kind of temporal presentation, and in this case timeline bar is used.

A speech annotation interface is useful for typists who, in an off-line mode, have to annotate and synchronise text against audio-visual content. The speech annotation interface is based on the slide show media interface and helps annotate text at specific level of granularity into media files. Annotation against video can be searched and the system can provide audio-visual search results.

IE Ο 3 08 4 Ο -13Another annotation interface supports portal updates on remote media servers located close to end-users. The audio-visual annotation interface also enables system users to insert rights and user access information.

A media portal interface for devices with small displays provides an overview of media content available to browse and retrieve on an end user device with small display. Based on standard mark-up languages, this Interface uses media segments as retrieval units, tracking users’ access history, thus being able to recommend the most important news stories to the user, in the form of a summarised list. This way, all the user has to do is simply select a story from the list - saving awkward text input and other interaction. The idea is putting most of the processing on the system itself, rather than providing elaborate searching/browsing work to the user. Because this interface tracks user interactions, it can generate usability information useful for billing and rating mechanisms.

The server 17 also provides detailed video interfaces for devices with small display. These two interfaces make video browsing similar to video game interactions on small devices. From these interfaces, the user can control the current point of browsing, using the controller/arrows. This allows the user to use only the one hand holding the small device. Four arrow buttons are just about all that the user uses to browse representative media (other than another arrow at the bottom left, to bring back to related media information such as listing, related programmes, etc.).

Playback interfaces represent static and synchronised multimedia messaging that plays images, audio and video information in email clients, browsers, and MMS clients on MMS handsets. Receivers and viewers of MMS can interact with the media information to access related information such as related stories in video formats.

IE 0 3 Ο » 4 0 -14The system 1 provides searching capabilities across programme meta-data, meta-data parameters filled by analysis techniques, and captured and referenced closed caption data. Search results are complemented and hyperlinked to visual representation of media programmes to allow users to access media information in a random fashion thus reducing the time to play back specific media segments. Searches can be made from any web browser capable device.

To complement text-based search of media programmes, video skimming and visualisation is achieved by the system 1 to retrieve media information in a nonlinear fashion. Information rendered by browsers can be progressively downloaded on a media device for a user to interact with locally or it can be requested on-demand by selecting media information stored remotely. Based on HTML and XML mark-up languages, media browsers are designed to optimise content-based operations for media devices having unique specifications (display, resolution, media format, and/or mark-up language rendering).

The controller 2 controls playback formats, resolutions, quality and method of transmission and monitoring of the playback of a media file. Playback of a media file is supported by streaming server and streaming software, video clients, and browsers that can render proprietary and standard media formats. The management module can control processes leading to playback of media content. Processes such as buffering, progressive download, image and ad removal/insertion can be set to suit user preferences and media device specifications.

The system 1 also enables content-based interactions from file formats and streaming clients provided that the management module of the present invention can interface with the API of a streaming server.

The controller 2 generates interfaces from which a system user can automatically access, annotate, format, share and publish media segments to any file or directory. £ 0 3 Oft 4 Ο -15This method allows distribution of media clips once in multiple formats, over many networks, to multiple devices. Text annotation to media clips are created to be only accessed by authorised users. Annotation of media clips can be accomplished at different levels of granularity of media content - image, advertisement, logical media segment (shots, scenes, story) - to allow for more efficient retrieval at a later stage. Users operating this interface set terms-of-usage for each media file that is made available for content-based operations on local/remote servers. Terms-of-usage can be embedded in transmission delivery formats of each media file in order to track and confirm usage of each media file retrieved.

The outputs of media editing can be standard multimedia messaging services based on text, images, and synchronised multimedia presentations. Outputs can also be text, audio and video clips that can be rendered by email clients, MMSCs, browsers, streaming players, and streaming clients.

The above interfaces are dynamically generated by XSL transformation. A set of XSL stylesheets are used to take necessary information from the underlying XML data which is generated whenever a web request is done by a client. The generated XML data comes from various sources to collect necessary information and as a result contains all information to display for the user’s device in a request of the user's action. XSL stylesheets then transform the data into user-viewable code (HTML, JavaScript, Java Applet and SVG) to render on the web browser and other devices. For the web interfaces with sophisticated features, layers of frames have been used, each frame corresponding to each XSL stylesheet, transforming underlying XML data for that particular frame into presentable format such as HTML. Framed interface allows application-like interaction in which a clicked widget stays on with the effect visible on other parts of the screen, instead of the whole screen refreshing. For simpler interfaces such as for mobile devices, a single XSL stylesheet defines the layout and elements of the screen, thus corresponding to a single underlying XML data generated when requested by the user. -16<030«40 In this way, the XML-based architecture of the system allows easy device-depended interface implementation using XSL. This also separates the underlying functionality of the system from the interface design, ensuring more user-oriented interface consideration by the designer.

The controller 2 can “push” and/or “pull” media files that have been analysed and indexed onto servers that are closer and more accessible to authorised users of media content.

It will be appreciated that the system 1 acts as the hub around which specific applications can be supported and services provided with a short lead time. The system 1 captures, manages and distributes media information requested by authorised users across retrieval devices. The system allows easy development of services.

The following is a non-exhaustive list of services which may be provided: Distributed analysis of media information Protected editing of media information Face Detection in live and encoded video programmes Caching of referenced media information Live broadcast & indexing of media content Real-time, near real-time, on-demand representation of media information Near real-time alerts of media events Image matching in live and encoded video programmes Face matching in structured video programmes Video object detection in live and encoded video programmes Video object matching in live and encoded video programmes Motion detection in live and encoded video programmes Ig ο 3 08 4 ο -17File merging Advertisement insertion based on user lists Searching across closed and open caption Searching across image and video features Personalised media caching Media management for instant messaging Multicasting Video skimming Broadcasters transmit a continuous flow of pre-recorded and live television programmes through licensed networks (cable/satellite/terrestrial) and media device (TV/STB). Broadcast models are currently based on physical locations where a TV signal can reach viewers without necessarily identifying them and interesting them. The system 1 provides an effective mechanism to transmit beyond this traditional model and reach audiences that are mobile. This is achieved by making timesensitive broadcast events available shortly after they have occurred and in a format suitable for retrieval across personal media devices.

Archive units of content owners, producers and aggregators like parliaments and broadcasting organisations are in charge of labour intensive operations. Restoration, digitisation, cataloguing, storing, copyright clearance, copying and shipping represent of the main processes that the archive units have to manage before media assets can be available on-demand to internal and external viewers. In order to enable access to archived content, this system enables the archival process to be accomplished over a web browser capable device. It also allows retrieval of copyrighted media information based on catalogued information.

The invention is not limited to the embodiments described but may be varied in construction and detail.

Claims

1. A multimedia management system comprising: a multimedia content capture (7) component for receiving content and for writing it to a storage device (10); an analysis component (11) for analysising received content to generate meta data, and a database (12) for storing the meta data; a server component (17) for distributing content or meta data to a network (MN) for delivery to subscriber devices; and a controller (2) for coordinating operation of the components of the system to provide configured services for delivery of content to subscriber devices (MS).

2. A system as claimed in claim 1, wherein the analysis component (11) generates the meta data as an index to the content in the storage device (10) and the server (17) uses the meta data to access the content; and wherein the analysis component (11) extracts key frames from received video content and segments audio and video streams; and wherein said key frames are stored to provide a storyboard of video content with images forming part of an index.

3. A system as claimed in any preceding claim, wherein the analysis component (11) operates in parallel with capture of content by the capture component (7) for analysis in real time as a video content stream is being received; and wherein the analysis component (11) uses a shot boundary detection technique to generate the meta data; and wherein the analysis component (11) automatically detects events in incoming content and generates notifications in real time. !Ε0 3 0«

4. β -194. A system as claimed in any preceding claim, wherein the controller directs routing of meta data from the database to a plurality of different devices, including local devices directly connected to the controller and remote subscriber devices; and wherein the analysis component (11) writes some of 5. The meta data to the storage devices to provide an addressing index; and wherein the controller uses meta data solely from the database for directing content to some subscriber devices; and wherein the controller dynamically generates media interfaces with style sheet transformation, in which style sheets dynamically transform the media and code into user-viewable media 6. 10 and display code.

5. A multimedia management system substantially as described with reference to the drawings.