New! View global litigation for patent families

WO2004043029A2 - Multimedia management - Google Patents

Multimedia management

Info

Publication number
WO2004043029A2
WO2004043029A2 PCT/IE2003/000152 IE0300152W WO2004043029A2 WO 2004043029 A2 WO2004043029 A2 WO 2004043029A2 IE 0300152 W IE0300152 W IE 0300152W WO 2004043029 A2 WO2004043029 A2 WO 2004043029A2
Authority
WO
Grant status
Application
Patent type
Prior art keywords
content
media
system
data
component
Prior art date
Application number
PCT/IE2003/000152
Other languages
French (fr)
Other versions
WO2004043029A3 (en )
Inventor
Alan Smeaton
Sean Marlow
Noel Murphy
Noel O'connor
Kieran Mcdonald
Hyowon Lee
Frederic Herrera
Paul Browne
Sheila Pollard
Original Assignee
Aliope Limited
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network-specific arrangements or communication protocols supporting networked applications
    • H04L67/28Network-specific arrangements or communication protocols supporting networked applications for the provision of proxy services, e.g. intermediate processing or storage in the network
    • H04L67/2804Network-specific arrangements or communication protocols supporting networked applications for the provision of proxy services, e.g. intermediate processing or storage in the network for adding application control or application functional data, e.g. adding metadata
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRICAL DIGITAL DATA PROCESSING
    • G06F17/00Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/30Information retrieval; Database structures therefor ; File system structures therefor
    • G06F17/30781Information retrieval; Database structures therefor ; File system structures therefor of video data
    • G06F17/30784Information retrieval; Database structures therefor ; File system structures therefor of video data using features automatically derived from the video content, e.g. descriptors, fingerprints, signatures, genre
    • G06F17/30787Information retrieval; Database structures therefor ; File system structures therefor of video data using features automatically derived from the video content, e.g. descriptors, fingerprints, signatures, genre using audio features
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRICAL DIGITAL DATA PROCESSING
    • G06F17/00Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/30Information retrieval; Database structures therefor ; File system structures therefor
    • G06F17/30781Information retrieval; Database structures therefor ; File system structures therefor of video data
    • G06F17/30784Information retrieval; Database structures therefor ; File system structures therefor of video data using features automatically derived from the video content, e.g. descriptors, fingerprints, signatures, genre
    • G06F17/30799Information retrieval; Database structures therefor ; File system structures therefor of video data using features automatically derived from the video content, e.g. descriptors, fingerprints, signatures, genre using low-level visual features of the video content
    • G06F17/30802Information retrieval; Database structures therefor ; File system structures therefor of video data using features automatically derived from the video content, e.g. descriptors, fingerprints, signatures, genre using low-level visual features of the video content using colour or luminescence
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRICAL DIGITAL DATA PROCESSING
    • G06F17/00Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/30Information retrieval; Database structures therefor ; File system structures therefor
    • G06F17/30781Information retrieval; Database structures therefor ; File system structures therefor of video data
    • G06F17/30784Information retrieval; Database structures therefor ; File system structures therefor of video data using features automatically derived from the video content, e.g. descriptors, fingerprints, signatures, genre
    • G06F17/30799Information retrieval; Database structures therefor ; File system structures therefor of video data using features automatically derived from the video content, e.g. descriptors, fingerprints, signatures, genre using low-level visual features of the video content
    • G06F17/30805Information retrieval; Database structures therefor ; File system structures therefor of video data using features automatically derived from the video content, e.g. descriptors, fingerprints, signatures, genre using low-level visual features of the video content using shape
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRICAL DIGITAL DATA PROCESSING
    • G06F17/00Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/30Information retrieval; Database structures therefor ; File system structures therefor
    • G06F17/30781Information retrieval; Database structures therefor ; File system structures therefor of video data
    • G06F17/30784Information retrieval; Database structures therefor ; File system structures therefor of video data using features automatically derived from the video content, e.g. descriptors, fingerprints, signatures, genre
    • G06F17/30799Information retrieval; Database structures therefor ; File system structures therefor of video data using features automatically derived from the video content, e.g. descriptors, fingerprints, signatures, genre using low-level visual features of the video content
    • G06F17/30811Information retrieval; Database structures therefor ; File system structures therefor of video data using features automatically derived from the video content, e.g. descriptors, fingerprints, signatures, genre using low-level visual features of the video content using motion, e.g. object motion, camera motion
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRICAL DIGITAL DATA PROCESSING
    • G06F17/00Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/30Information retrieval; Database structures therefor ; File system structures therefor
    • G06F17/30781Information retrieval; Database structures therefor ; File system structures therefor of video data
    • G06F17/30846Browsing of video data
    • G06F17/30852Browsing the internal structure of a single video sequence
    • GPHYSICS
    • G11INFORMATION STORAGE
    • G11BINFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
    • G11B27/00Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
    • G11B27/10Indexing; Addressing; Timing or synchronising; Measuring tape travel
    • G11B27/102Programmed access in sequence to addressed parts of tracks of operating record carriers
    • G11B27/105Programmed access in sequence to addressed parts of tracks of operating record carriers of operating discs
    • GPHYSICS
    • G11INFORMATION STORAGE
    • G11BINFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
    • G11B27/00Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
    • G11B27/10Indexing; Addressing; Timing or synchronising; Measuring tape travel
    • G11B27/19Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier
    • G11B27/28Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier by using information signals recorded by the same method as the main recording
    • GPHYSICS
    • G11INFORMATION STORAGE
    • G11BINFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
    • G11B27/00Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
    • G11B27/10Indexing; Addressing; Timing or synchronising; Measuring tape travel
    • G11B27/19Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier
    • G11B27/28Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier by using information signals recorded by the same method as the main recording
    • G11B27/32Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier by using information signals recorded by the same method as the main recording on separate auxiliary tracks of the same or an auxiliary record carrier
    • G11B27/327Table of contents
    • G11B27/329Table of contents on a disc [VTOC]
    • GPHYSICS
    • G11INFORMATION STORAGE
    • G11BINFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
    • G11B27/00Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
    • G11B27/10Indexing; Addressing; Timing or synchronising; Measuring tape travel
    • G11B27/34Indicating arrangements
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network-specific arrangements or communication protocols supporting networked applications
    • H04L67/28Network-specific arrangements or communication protocols supporting networked applications for the provision of proxy services, e.g. intermediate processing or storage in the network
    • H04L67/2823Network-specific arrangements or communication protocols supporting networked applications for the provision of proxy services, e.g. intermediate processing or storage in the network for conversion or adaptation of application content or format

Abstract

A multimedia management system (1) receives live stream content at an interface (5) or static content (6) and captures (7) it to a storage device (10). An analysis component (11) in parallel or in series generates meta data for a database (12), the meta data acting as an index for the storage device (10). Automatic detection of events by the analysis component (11) allows provision of real time distribution of content such as news clips to fixed and mobile networks (MS). Near real time media services are also provided with some annotations being performed by an annotation component (14) under control of a browser (13).

Description

"Multimedia Management"

INTRODUCTION

Field of the Invention

The invention relates to management of multimedia such as digitised audio-visual content.

Prior Art Discussion

In recent years advances have been made in the field of distribution of multimedia in mobile networks. For example the Multimedia Messaging Service (MMS) standard for mobile networks specifies peer-to-peer multimedia messaging. United States Patent Application No. US200205/442 describes a method of transmitting/receiving a broadcast message in a mobile network.

Also, advances have been made in processing of multimedia in various aspects, such as indexing and browsing. For example, the paper "Fischlar: an on-line system for indexing and browsing broadcast television", O'Connor N, Marlow S, Murphy N, Smeaton A, Browne P, Deasy S, Lee H and Mc Donald K. ICASSP 2001 - International Conference on Acoustics, Speech, and Signal Processing. Salt Lake City, UT, 7-11 May 2001. , describes indexing and browsing. Television programmes are captured in MPEG-1 format and analysed using video indexing tools. Browsing interfaces allow a user to browse a visual index to locate content of interest, and selected content can then be streamed in real time.

While these developments are of benefit, there is still a need to provide more comprehensive multimedia asset management to support content-based operations for a broader range of devices and access mechanisms. SUMMARY OF THE INVENTION

According to the invention, there is provided a multimedia management system comprising:

a multimedia content capture component for receiving content and for writing it to a storage device;

an analysis component for analysising received content to generate meta data, and a database for storing the meta data;

a server component for distributing content or meta data to a network for delivery to subscriber devices; and

a controller for coordinating operation of the components of the system to provide configured services for delivery of content to subscriber devices.

In one embodiment, the analysis component generates the meta data as an index to the content in the storage device and the server uses the meta data to access the content.

In another embodiment, the analysis component extracts key frames from received video content and segments audio and video streams.

In a further embodiment, said key frames are stored to provide a storyboard of video content with images forming part of an index. In one embodiment, the analysis component operates in parallel with capture of content by the capture component for analysis in real time as a video content stream is being received.

In another embodiment, the analysis component uses a shot boundary detection technique to generate the meta data.

In a further embodiment, the analysis component automatically detects events in incoming content and generates notifications in real time.

In one embodiment, the controller receives the notifications and automatically controls components of the system to generate a content output triggered by the detected event.

In another embodiment, the notification is an alert message transmitted to a mobile station.

In a further embodiment, the system further comprises an annotation component for annotating content and directing storage of annotated content in the storage device.

In one embodiment, the annotation component operates in near real time.

In another embodiment, the controller directs output via the server of content in near real time to provide a subscriber service.

In a further embodiment, the system further comprises a browser connected to the storage device and to the database, the browser allowing supervisor access to content to control processing of the content. In one embodiment, the browser is connected to the annotation component and directs operation of the annotation component to provide processed content in the near real-time service.

In another embodiment, the system further comprises a transcoding component connected to the browser and being for transcoding of content for delivery to the analysis component and to the storage device.

In a further embodiment, the controller directs routing of meta data from the database to a plurality of different devices, including local devices directly connected to the controller and remote subscriber devices.

In one embodiment, the analysis component writes some of the meta data to the storage devices to provide an addressing index.

In another embodiment, the controller uses meta data solely from the database for directing content to some subscriber devices.

In a further embodiment, the controller dynamically generates media interfaces with style sheet transformation, in which style sheets dynamically transform the media and code into user-viewable media and display code.

In one embodiment, the transformation performs independent processing of screen windows in a manner analogous to application or operating system display windows.

In another embodiment, the style sheet transformation code is separate from underlying functionality of the controller and other components involved in media output. In a further embodiment, the system further comprises a live streaming interface connected to the content capture component.

In one embodiment, the system further comprises a scheduling component connected to the live stream circuit for activating the live stream circuit and setting recording parameters for both the live stream circuit and the capture component.

DETAILED DESCRIPTION OF THE INVENTION

Brief Description of the Drawings

The invention will be more clearly understood from the following description of some embodiments thereof, given by way of example only with reference to the accompanying drawings in which:-

Fig. 1 is a block diagram of a multimedia management system of the invention.

Fig. 2 is a diagram showing interfacing modes; and

Fig. 3 is a diagram illustrating media analysis, generation of meta-data, and presentation for content-based operations such as access, retrieval, sharing consumption.

Description of the Embodiments

Referring to Fig. 1 a multimedia management system 1 of the invention comprises a controller 2 which controls in a comprehensive manner various multimedia processing components to provide a variety of services. These components are as follows. : A live stream interface for receiving broadcast digital TV channels.

: A reader device for reading multimedia content from a static, non-live source such as DVDs.

: A capture circuit for encoding and compressing content to a format such as MPEG.

0: A content storage device for storing the encoded and compressed content files together with related meta data. The latter includes images, hyperlinks, log files for data such as start/ end time of capture, programme meta-data, user account information, user preferences, copyright information, number of key frames extracted, file size, and channel meta data. Some of the meta data is generated by the capture circuit 7, and the remainder by an analysis component 11.

1: An analysis component 11 which extracts representative meta-data such as key frames, audio, text, from compressed audio-visual content. Extracted meta-data is encoded in formats and resolutions (for example JPEG, GIF, MPEG) of any size that can be displayed, selected, and interacted with. The analysis component 11 both extracts existing meta data, and generates meta data which is stored in the storage device 10. However, it also writes meta data to a database 12, set out below. Additional meta data related to the precision and accuracy of analysis techniques used against audio-visual content is generated by the analysis component 11 using the shot boundary technique to generate colour histogram, edge detection, and motion analysis data.

2: A content database for storing all meta data generated by the analysis component 11. Some of the meta data is also stored with the actual content in the storage device 10, as outlined above. The database 12 stores the meta data in a manner to provide an index for the storage device 10.

13: A browser for use by local/remote staff of the organization hosting the system 1. This is used for accessing the content using indexes of the database 12. It activates interfaces for content-based operations (non-linear playback, annotation) accomplished over end-user devices. The browser 13 can also route content to a transcoding component 15.

14: An annotation component for annotation of stored content under instruction from the browser 13.

16: A scheduling component for scheduling operation of the interface 5 for receiving and encoding live streams.

17: A Web server for routing of content to infrastructure for fixed and mobile networks for onward routing to central/edge servers and end user playback devices.

The multimedia management processes performed by the system are now described. One of the problems facing operators of communication systems such as mobile network operators is that while there is a vast amount of content available, it has not been possible to manage it so that content owners and aggregators, network operators and end users get optimum use of it. It is not just a matter of coping with the limited communication bandwidths and device capabilities of many user devices, but also syndicating content which end users wish to access.

The system 1 addresses this problem by acting as a platform which takes in the multimedia content and manages it in an optimum manner so that end users can access /receive what they subscribed to. For example, security camera systems generate a vast quantity of content, however, an end user is only interested in seeing suspicious activity. Thus, the analysis component 11 may automatically monitor live streams from such cameras and generate an alert for the controller 2 if a suspicious image and/or sound pattern is detected. The controller in turn notifies in real time the Web server 17 to retrieve the content meta data and push it onto either the fixed or mobile network to the end user subscriber playback devices. One suitable bearer for such alerts is GSM SMS to mobile devices.

In other situations some human input may be required such as downloading of a video sequence of a breaking news event or a score in a football match. In this case, staff monitor certain content channels and access audio-visual segments in non-linear fashion to achieve content-based operations for seamless syndication, transmission and retrieval over fixed and mobile networks. The content is cropped and rendered to be suitable for transmission to the subscriber's mobile station MS. Such operations can typically be carried out in a few seconds, thus providing a near realtime service to the subscriber.

Fig. 2 illustrates the versatility of the system 1, providing content to a wide variety of devices to present a variety of interfaces. These include a portal interface 20, a search interface 21, a browsing interface 22, a playback interface 23, an annotation interface 24, remote user device browsing/ swapping interfaces 25, and a remote user device playback interface 26. This diagram illustrates versatility of the architecture of the system 1. The controller 20 is at the centre allowing interfacing with both staff of the hosting organisation and remote end users in a versatile manner.

The controller 2 includes a database of access control data for interfacing with staff. It allows supervisor access to some of the components.

The controller 2 controls automatic deletion of content according to configured parameters. The analysis component 11 generates the meta data is a manner whereby it forms a content index that is represented using images of video content and meta data referenced to additional related media content. Capture and analysis of content can take place in parallel, thus enabling text, image and video distribution without the need to wait for a programme to end before managing distribution to a mass audience.

The analysis component 2 generates the meta data for some content in the form of storyboards. Live and stored media files are analysed using algorithms processing specific data streams /layers, multiplexed/ encapsulated in analogue and standard media compression formats (for example MPEG formats). Each algorithm can be controlled by a dynamic threshold/parameter set to analyse different types of media information (for example video with slow-movement and short cuts compared to video with fast motion and fades). The analysis techniques are used to extract representational content from media files for indexing based on natural breaks or events in the media content. Analysis results derived from files composed of different type of media information (for example news programme with newscaster and news agencies footage) are based on a combination of algorithms in order to generate the most logical representation of programme's content for specific playback devices.

The analysis component 11 analyses whole data streams/formats and media layers by segmenting audio, detecting faces, detecting edges, detecting black frames, and generating colour histograms, The combination of these algorithms detects video frames and video shots that are sequenced in a logical scene if media programmes follow a structure at the time of production.

Referring to Fig. 3, a media file 40 is analysed in step 45 to generate content-based meta-data 46, which is then automatically stored in the storage device 10. The meta- data 46 is retrieved using a style sheet 41 that can be marked-up in representation languages (for example HTML, CHTML, WML, SMIL and XML languages) useful for static/ dynamic representation of media information to media devices (for example PC, PDA, and mobile phone). Dynamic representation of media content provides the basis for progressive download of media information to media devices dependent on progressive and fast retrieval of media information over narrowband networks. Thus, media data is presented or exchanged between users by communicating representational information, meta-data information, and location information rather that communicating the media data itself.

The controller 2 acts as an interface between the storage device 10 and the web server 17, the Web browser 13, and a streaming server reading commands from hyperlinks and animated media formats. Such commands are sent over the fixed and mobile network to playback devices. The Web browser 13 is automatically refreshed and mapped against media files, metafiles and meta-data found in the database 12.

Requests may be made from subscriber devices to achieve content-based operations. Each content-based operation is completed by connection (circuit switched and/ or packet based) to a local media server and/or remote server depending on the availability of bandwidth that the mobile device can connect to and the location of subscriber requesting the content. Media programmes and/ or media clips are subsequently and automatically sent over packet-based networks to subscriber devices in synchronised, downloadable or streamable formats that can be played using messaging software, browsers or media decoders developed specifically for the media device.

Access to interfaces and media information in the system 1 is monitored at different levels. Content-based operations requested by a subscriber can only be fulfilled after inputting a user ID. Monitored access generates information on usage of the system 1 and the media data content. Such information can be used to improve media distribution on a personalised basis. Recording functions integrated in the scheduling component 16 provides system users and end users with recording options. Recording requests could be made to the system by clicking on hyperlinked media information such as programme title, and programme meta-data. End users can also rate media programmes in order to receive programme recommendations from the system.

Search interfaces with media highlights in an overview/ detail view interface are also generated based upon the meta data. Media sequences are highlighted based on a search query and they can be browsed using a timeline bar in Detail View Media interface.

The system 1 also generates a search interface with a detail view based on text results and slide show media interface enabling system users to narrow down search results using text information (for example closed caption, annotation) tagged to indexed audio-visual content. A search interface for audio-visual information enables system users and end users to search audio-visual information indexed using different analysis techniques. Searches can be applied against different parameters populated by analysis techniques such a face detection, closed caption parsing, and shot detection. Search results are ranked and are based on representative meta-data. An overview media interface implements "overview first, zoom and filter, then details on demand". This interface provides an overview of media files using representative meta-data, thus indicating to the end user what the audio-visual information is about.

The system 1 also generates a scroll bar media interface. System users browse media content using a scroll bar that acts as a very versatile controller. Clicking on the arrows of the scroll bar moves the page by each row of representative keyframes, clicking on the empty bar jumps the page by screenful, and dragging the square object provides immediate and continuous movement of representative keyframes thus allowing finer and more pleasant experience. Another interface is an extension of the overview media interface. This is similar to the overview/ detail browser, but allows movement into a detail view. Once in a detail view, a timeline bar is used for immediate system reaction and finer control in browsing. The timeline bar can be static or dynamic. Both types of timeline bars are useful when browsing a large amount of media information because it provides temporal orientation by showing the current point of browsing in relation to the whole media file. Not only useful as a time indicator, the bar in this browser allows random access by selecting related sections of the bar. The interface with the dynamic quick bar capabilities saves tedious and continuous clickings that the static timeline bar browser may require. One sweep of the mouse allows viewing of all of the keyframes in the video. Much finer control and continuous, immediate response from the system provides 'direct manipulation'. The browsing task is less of an 'act- and-react' interaction, and is more natural and paced by the user.

A slide show media interface flips through all representative indexed meta-data in a single slot. Temporally presenting representative meta-data (rather than spatially) saves screen space considerably, and is thus useful for small end user devices. People with more dynamic interaction preference may also like this browser. It is important to provide some form of temporal orientation in this kind of temporal presentation, and in this case timeline bar is used.

A speech annotation interface is useful for typists who, in an off-line mode, have to annotate and synchronise text against audio-visual content. The speech annotation interface is based on the slide show media interface and helps annotate text at specific level of granularity into media files. Annotation against video can be searched and the system can provide audio-visual search results. Another annotation interface supports portal updates on remote media servers located close to end-users. The audio-visual annotation interface also enables system users to insert rights and user access information.

A media portal interface for devices with small displays provides an overview of media content available to browse and retrieve on an end user device with small display. Based on standard mark-up languages, this Interface uses media segments as retrieval units, tracking users' access history, thus being able to recommend the most important news stories to the user, in the form of a summarised list. This way, all the user has to do is simply select a story from the list - saving awkward text input and other interaction. The idea is putting most of the processing on the system itself, rather than providing elaborate searching/browsing work to the user. Because this interface tracks user interactions, it can generate usability information useful for billing and rating mechanisms.

The server 17 also provides detailed video interfaces for devices with small display. These two interfaces make video browsing similar to video game interactions on small devices. From these interfaces, the user can control the current point of browsing, using the controller/arrows. This allows the user to use only the one hand holding the small device. Four arrow buttons are just about all that the user uses to browse representative media (other than another arrow at the bottom left, to bring back to related media information such as listing, related programmes, etc.).

Playback interfaces represent static and synchronised multimedia messaging that plays images, audio and video information in email clients, browsers, and MMS clients on MMS handsets. Receivers and viewers of MMS can interact with the media information to access related information such as related stories in video formats. The system 1 provides searching capabilities across programme meta-data, meta-data parameters filled by analysis techniques, and captured and referenced closed caption data. Search results are complemented and hyperlinked to visual representation of media programmes to allow users to access media information in a random fashion thus reducing the time to play back specific media segments. Searches can be made from any web browser capable device.

To complement text-based search of media programmes, video skimming and visualisation is achieved by the system 1 to retrieve media information in a non- linear fashion. Information rendered by browsers can be progressively downloaded on a media device for a user to interact with locally or it can be requested on-demand by selecting media information stored remotely. Based on HTML and XML mark-up languages, media browsers are designed to optimise content-based operations for media devices having unique specifications (display, resolution, media format, and/ or mark-up language rendering) .

The controller 2 controls playback formats, resolutions, quality and method of transmission and monitoring of the playback of a media file. Playback of a media file is supported by streaming server and streaming software, video clients, and browsers that can render proprietary and standard media formats. The management module can control processes leading to playback of media content. Processes such as buffering, progressive download, image and ad removal/insertion can be set to suit user preferences and media device specifications.

The system 1 also enables content-based interactions from file formats and streaming clients provided that the management module of the present invention can interface with the API of a streaming server.

The controller 2 generates interfaces from which a system user can automatically access, annotate, format, share and publish media segments to any file or directory. This method allows distribution of media clips once in multiple formats, over many networks, to multiple devices. Text annotation to media clips are created to be only accessed by authorised users. Annotation of media clips can be accomplished at different levels of granularity of media content - image, advertisement, logical media segment (shots, scenes, story) - to allow for more efficient retrieval at a later stage. Users operating this interface set terms-of-usage for each media file that is made available for content-based operations on local/remote servers. Terms-of-usage can be embedded in transmission delivery formats of each media file in order to track and confirm usage of each media file retrieved.

The outputs of media editing can be standard multimedia messaging services based on text, images, and synchronised multimedia presentations. Outputs can also be text, audio and video clips that can be rendered by email clients, MMSCs, browsers, streaming players, and sfreaming clients.

The above interfaces are dynamically generated by XSL transformation. A set of XSL stylesheets are used to take necessary information from the underlying XML data which is generated whenever a web request is done by a client. The generated XML data comes from various sources to collect necessary information and as a result contains all information to display for the user's device in a request of the user's action. XSL stylesheets then transform the data into user-viewable code (HTML, JavaScript, Java Applet and SVG) to render on the web browser and other devices. For the web interfaces with sophisticated features, layers of frames have been used, each frame corresponding to each XSL stylesheet, transforming underlying XML data for that particular frame into presentable format such as HTML. Framed interface allows application-like interaction in which a clicked widget stays on with the effect visible on other parts of the screen, instead of the whole screen refreshing. For simpler interfaces such as for mobile devices, a single XSL stylesheet defines the layout and elements of the screen, thus corresponding to a single underlying XML data generated when requested by the user. In this way, the XML-based architecture of the system allows easy device-depended interface implementation using XSL. This also separates the underlying functionality of the system from the interface design, ensuring more user-oriented interface consideration by the designer.

The controller 2 can "push" and/ or "pull" media files that have been analysed and indexed onto servers that are closer and more accessible to authorised users of media content.

It will be appreciated that the system 1 acts as the hub around which specific applications can be supported and services provided with a short lead time. The system 1 captures, manages and distributes media information requested by authorised users across retrieval devices. The system allows easy development of services.

The following is a non-exhaustive list of services which may be provided:

Distributed analysis of media information - Protected editing of media information

Face Detection in live and encoded video programmes

Caching of referenced media information

Live broadcast & indexing of media content

Real-time, near real-time, on-demand representation of media information - Near real-time alerts of media events

Image matching in live and encoded video programmes

Face matching in structured video programmes

Video object detection in live and encoded video programmes

Video object matching in live and encoded video programmes - Motion detection in live and encoded video programmes File merging

Advertisement insertion based on user lists Searching across closed and open caption Searching across image and video features - Personalised media caching

Media management for instant messaging

Multicasting

Video skimming

Broadcasters transmit a continuous flow of pre-recorded and live television programmes through licensed networks (cable/satellite/terrestrial) and media device (TV/STB). Broadcast models are currently based on physical locations where a TV signal can reach viewers without necessarily identifying them and interesting them. The system 1 provides an effective mechanism to transmit beyond this traditional model and reach audiences that are mobile. This is achieved by making time- sensitive broadcast events available shortly after they have occurred and in a format suitable for retrieval across personal media devices.

Archive units of content owners, producers and aggregators like parliaments and broadcasting organisations are in charge of labour intensive operations. Restoration, digitisation, cataloguing, storing, copyright clearance, copying and shipping represent of the main processes that the archive units have to manage before media assets can be available on-demand to internal and external viewers. In order to enable access to archived content, this system enables the archival process to be accomplished over a web browser capable device. It also allows retrieval of copyrighted media information based on catalogued information.

The invention is not limited to the embodiments described but may be varied in construction and detail.

Claims

Claims
1. A multimedia management system comprising:
a multimedia content capture (7) component for receiving content and for writing it to a storage device (10);
an analysis component (11) for analysising received content to generate meta data, and a database (12) for storing the meta data;
a server component (17) for distributing content or meta data to a network (MN) for delivery to subscriber devices; and
a controller (2) for coordinating operation of the components of the system to provide configured services for delivery of content to subscriber devices (MS).
2. A system as claimed in claim 1, wherein the analysis component (11) generates the meta data as an index to the content in the storage device (10) and the server (17) uses the meta data to access the content.
3. A system as claimed in claim 1 or 2, wherein the analysis component (11) extracts key frames from received video content and segments audio and video streams.
4. A system as claimed in claim 3, wherein said key frames are stored to provide a storyboard of video content with images forming part of an index.
5. A system as claimed in any preceding claim, wherein the analysis component (11) operates in parallel with capture of content by the capture component (7) for analysis in real time as a video content stream is being received.
6. A system as claimed in any preceding claim, wherein the analysis component (11) uses a shot boundary detection technique to generate the meta data.
7. A system as claimed in any preceding claim, wherein the analysis component (11) automatically detects events in incoming content and generates notifications in real time.
8. A system as claimed in claim 7, wherein the controller (2) receives the notifications and automatically controls components of the system to generate a content output triggered by the detected event.
9. A system as claimed in claim 8, wherein the notification is an alert message transmitted to a mobile station.
10. A system as claimed in any preceding claim, wherein the system further comprises an annotation component (14) for annotating content and directing storage of annotated content in the storage device (10).
11. A system as claimed in claim 10, wherein the annotation component (14) operates in near real time.
12. A system as claimed in claim 11, wherein the controller directs output via the server (17) of content in near real time to provide a subscriber service.
13. A system as claimed in claim 12, further comprising a browser (13) connected to the storage device (10) and to the database (12), the browser allowing supervisor access to content to control processing of the content.
14. A system as claimed in claims 10 to 13, wherein the browser (13) is connected to the annotation component (14) and directs operation of the annotation component to provide processed content in the near real-time service.
15. A system as claimed in claim 14, further comprising a transcoding component (15) connected to the browser (13) and being for transcoding of content for delivery to the analysis component and to the storage device.
16. A system as claimed in any preceding claim, wherein the controller directs routing of meta data from the database to a plurality of different devices, including local devices directly connected to the controller and remote subscriber devices.
17. A system as claimed in claim 16, wherein the analysis component (11) writes some of the meta data to the storage devices to provide an addressing index.
18. A system as claimed in claims 16 or 17, wherein the controller uses meta data solely from the database for directing content to some subscriber devices.
19. A system as claimed in any preceding claim, wherein the controller dynamically generates media interfaces with style sheet transformation, in which style sheets dynamically transform the media and code into user- viewable media and display code.
20. A system as claimed in claim 19, wherein the transformation performs independent processing of screen windows in a manner analogous to application or operating system display windows.
21. A system as claimed in claim 19 or 20, wherein the style sheet transformation code is separate from underlying functionality of the controller and other components involved in media output.
22. A system as claimed in any preceding claim, wherein the system further comprises a live streaming interface connected to the content capture component.
23. A system as claimed in claim 22, further comprising a scheduling component connected to the live stream circuit for activating the live stream circuit and setting recording parameters for both the live stream circuit and the capture component.
24. A computer program product comprising software code for performing operations of a multimedia management system of any preceding claim when executing on a digital system.
PCT/IE2003/000152 2002-11-08 2003-11-10 Multimedia management WO2004043029A3 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
IE20020870 2002-11-08
IE2002/0870 2002-11-08

Publications (2)

Publication Number Publication Date
WO2004043029A2 true true WO2004043029A2 (en) 2004-05-21
WO2004043029A3 true WO2004043029A3 (en) 2004-08-05

Family

ID=32310088

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IE2003/000152 WO2004043029A3 (en) 2002-11-08 2003-11-10 Multimedia management

Country Status (1)

Country Link
WO (1) WO2004043029A3 (en)

Cited By (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2004093462A1 (en) * 2003-04-17 2004-10-28 Koninklijke Philips Electronics N.V. Content analysis of coded video data
FR2878116A1 (en) * 2004-11-15 2006-05-19 France Telecom Multimedia data stream indexing method for monitoring sportive event e.g. running race, involves temporally correlating detected events with temporally indexed multimedia data stream, to deliver list of detected events based on stream
WO2006077196A1 (en) * 2005-01-19 2006-07-27 France Telecom Method for generating a text-based index from a voice annotation
US20060288036A1 (en) * 2005-06-17 2006-12-21 Microsoft Corporation Device specific content indexing for optimized device operation
EP1782243A2 (en) * 2004-07-09 2007-05-09 Forte Internet Software, Inc. Methods and apparatus for identifying and facilitating a social interaction structure over a data packet network
WO2007130472A2 (en) * 2006-05-01 2007-11-15 Gotuit Media Corporation Methods and systems for providing media assets over a network
DE102007030131B3 (en) * 2007-06-29 2008-10-23 Whatmore Gmbh Information i.e. video data, synchronized provision method, for e.g. interactive application of portal, involves forming community using identifier with information related to geographical location of users
US7640560B2 (en) 1996-10-03 2009-12-29 Gotuit Media Corporation Apparatus and methods for broadcast monitoring
WO2010090622A1 (en) * 2009-02-09 2010-08-12 Vitamin D, Inc. Systems and methods for video analysis
US20100211617A1 (en) * 2009-02-17 2010-08-19 Jhilmil Jain Projects containing media data of different types
US8204955B2 (en) 2007-04-25 2012-06-19 Miovision Technologies Incorporated Method and system for analyzing multimedia content
US8316128B2 (en) 2004-01-26 2012-11-20 Forte Internet Software, Inc. Methods and system for creating and managing identity oriented networked communication
EP2811751A1 (en) * 2010-09-17 2014-12-10 Samsung Electronics Co., Ltd Method for providing media-content related information, device, server, and computer-readable storage medium for executing the method
EP2851900A1 (en) * 2013-09-18 2015-03-25 Nxp B.V. Media content real time analysis and automated semantic summarization

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2000016221A1 (en) * 1998-09-15 2000-03-23 Microsoft Corporation Interactive playlist generation using annotations
US6166735A (en) * 1997-12-03 2000-12-26 International Business Machines Corporation Video story board user interface for selective downloading and displaying of desired portions of remote-stored video data objects

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6166735A (en) * 1997-12-03 2000-12-26 International Business Machines Corporation Video story board user interface for selective downloading and displaying of desired portions of remote-stored video data objects
WO2000016221A1 (en) * 1998-09-15 2000-03-23 Microsoft Corporation Interactive playlist generation using annotations

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
A.F. SMEATON, N. MURPHY, N. E. O'CONNOR: "The Fischlàr Digital Video System: A Digital Library of Broadcast TV Programmes" INTERNET, [Online] 24 June 2001 (2001-06-24), - 28 June 2001 (2001-06-28) XP002271605 Retrieved from the Internet: URL:http://www.cdvp.dcu.ie/Papers/JCDL2001 .pdf> [retrieved on 2004-02-25] *
BARRY SMYTH, PAUL COTTER: "A Personalized Television Listings Service" INTERNET, [Online] August 2000 (2000-08), XP002271604 Retrieved from the Internet: URL:http://www-cse.uta.edu/~holder/courses /cse6362/spr03/prepubs/Smyth99.pdf> [retrieved on 2004-02-24] *
PAUL COTTER, BARRY SMYTH: "PTV: Intelligent Personalised TV Guides" INTERNET, [Online] 2000, XP002271603 Retrieved from the Internet: URL:http://www.changingworlds.com/content/ documents/PCotter.pdf> [retrieved on 2004-02-24] *

Cited By (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7640560B2 (en) 1996-10-03 2009-12-29 Gotuit Media Corporation Apparatus and methods for broadcast monitoring
WO2004093462A1 (en) * 2003-04-17 2004-10-28 Koninklijke Philips Electronics N.V. Content analysis of coded video data
US8316128B2 (en) 2004-01-26 2012-11-20 Forte Internet Software, Inc. Methods and system for creating and managing identity oriented networked communication
US8250150B2 (en) 2004-01-26 2012-08-21 Forte Internet Software, Inc. Methods and apparatus for identifying and facilitating a social interaction structure over a data packet network
EP1782243A4 (en) * 2004-07-09 2009-09-16 Forte Internet Software Inc Methods and apparatus for identifying and facilitating a social interaction structure over a data packet network
EP1782243A2 (en) * 2004-07-09 2007-05-09 Forte Internet Software, Inc. Methods and apparatus for identifying and facilitating a social interaction structure over a data packet network
FR2878116A1 (en) * 2004-11-15 2006-05-19 France Telecom Multimedia data stream indexing method for monitoring sportive event e.g. running race, involves temporally correlating detected events with temporally indexed multimedia data stream, to deliver list of detected events based on stream
WO2006077196A1 (en) * 2005-01-19 2006-07-27 France Telecom Method for generating a text-based index from a voice annotation
US20060288036A1 (en) * 2005-06-17 2006-12-21 Microsoft Corporation Device specific content indexing for optimized device operation
WO2007130472A3 (en) * 2006-05-01 2008-01-17 Gotuit Media Corp Methods and systems for providing media assets over a network
WO2007130472A2 (en) * 2006-05-01 2007-11-15 Gotuit Media Corporation Methods and systems for providing media assets over a network
US8204955B2 (en) 2007-04-25 2012-06-19 Miovision Technologies Incorporated Method and system for analyzing multimedia content
DE102007030131B3 (en) * 2007-06-29 2008-10-23 Whatmore Gmbh Information i.e. video data, synchronized provision method, for e.g. interactive application of portal, involves forming community using identifier with information related to geographical location of users
WO2010090622A1 (en) * 2009-02-09 2010-08-12 Vitamin D, Inc. Systems and methods for video analysis
US20100211617A1 (en) * 2009-02-17 2010-08-19 Jhilmil Jain Projects containing media data of different types
US9330097B2 (en) * 2009-02-17 2016-05-03 Hewlett-Packard Development Company, L.P. Projects containing media data of different types
EP2811751A1 (en) * 2010-09-17 2014-12-10 Samsung Electronics Co., Ltd Method for providing media-content related information, device, server, and computer-readable storage medium for executing the method
US9071852B2 (en) 2010-09-17 2015-06-30 Samsung Electronics Co., Ltd. Method for providing media-content related information, device, server, and computer-readable storage medium for executing the method
EP2851900A1 (en) * 2013-09-18 2015-03-25 Nxp B.V. Media content real time analysis and automated semantic summarization
US9703461B2 (en) 2013-09-18 2017-07-11 Nxp B.V. Media content creation

Also Published As

Publication number Publication date Type
WO2004043029A3 (en) 2004-08-05 application

Similar Documents

Publication Publication Date Title
US7735101B2 (en) System allowing users to embed comments at specific points in time into media presentation
US6166735A (en) Video story board user interface for selective downloading and displaying of desired portions of remote-stored video data objects
US6868415B2 (en) Information linking method, information viewer, information register, and information search equipment
US8156176B2 (en) Browser based multi-clip video editing
Ramanathan et al. Architectures for personalized multimedia
Mauthe et al. Professional content management systems: handling digital media assets
US7769819B2 (en) Video editing with timeline representations
US7110664B2 (en) Methods and apparatus for indexing and archiving encoded audio-video data
US20100145794A1 (en) Media Processing Engine and Ad-Per-View
US20030093794A1 (en) Method and system for personal information retrieval, update and presentation
US20060259589A1 (en) Browser enabled video manipulation
US7356830B1 (en) Method and apparatus for linking a video segment to another segment or information source
US20070038612A1 (en) System and method for indexing, searching, identifying, and editing multimedia files
US20020120925A1 (en) Audio and video program recording, editing and playback systems using metadata
US20050223034A1 (en) Metadata for object in video
US20070154190A1 (en) Content tracking for movie segment bookmarks
US20110060998A1 (en) System and method for managing internet media content
US20070121651A1 (en) Network-based format conversion
US7769827B2 (en) Interactive video application hosting
US20120047542A1 (en) System and method for rule based dynamic server side streaming manifest files
US20090157697A1 (en) Systems and methods for creating variable length clips from a media stream
US20080155627A1 (en) Systems and methods of searching for and presenting video and audio
US20070113184A1 (en) Method and system for providing remote digital media ingest with centralized editorial control
US20040246376A1 (en) Video content transmission device and method, video content storage device, video content reproduction device and method, meta data generation device, and video content management method
US20030074671A1 (en) Method for information retrieval based on network

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A2

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BW BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE EG ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NI NO NZ OM PG PH PL PT RO RU SC SD SE SG SK SL SY TJ TM TN TR TT TZ UA UG US UZ VC VN YU ZA ZM ZW

AL Designated countries for regional patents

Kind code of ref document: A2

Designated state(s): BW GH GM KE LS MW MZ SD SL SZ TZ UG ZM ZW AM AZ BY KG KZ MD RU TJ TM AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IT LU MC NL PT RO SE SI SK TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG

121 Ep: the epo has been informed by wipo that ep was designated in this application
122 Ep: pct application non-entry in european phase
NENP Non-entry into the national phase in:

Ref country code: JP

WWW Wipo information: withdrawn in national office

Country of ref document: JP