WO2003019416A1 - Automatic question formulation from a user selection in multimedia content - Google Patents
Automatic question formulation from a user selection in multimedia content Download PDFInfo
- Publication number
- WO2003019416A1 WO2003019416A1 PCT/IB2002/003464 IB0203464W WO03019416A1 WO 2003019416 A1 WO2003019416 A1 WO 2003019416A1 IB 0203464 W IB0203464 W IB 0203464W WO 03019416 A1 WO03019416 A1 WO 03019416A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- descriptions
- multimedia content
- node
- document
- question
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/40—Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
- G06F16/48—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/489—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using time information
Definitions
- the invention relates to electronic equipment comprising reading means for reading a multimedia content which is described in a document containing descriptions.
- the invention also relates to a system comprising such equipment.
- the invention likewise relates to a method of formulating a question intended to be transmitted to a search engine while a multimedia content is being used by a user, said multimedia content being described in a document that contains descriptions.
- the invention also relates to a program comprising program code instructions for implementing such a method when executed by a processor.
- MPEG-7 Context, Objectives and Technical Roadmap
- ISO/TEC JTC1/SC29/WG11/N2861 published by the ISO referred to as ISO/TEC JTC1/SC29/WG11/N2861, in July 1999, MPEG-7 is a standard for describing multimedia contents.
- a Multimedia content may be associated with an MPEG-7 document which describes said content, for example, to permit making searches in said multimedia content.
- Equipment according to the invention and as described in the opening paragraph is characterized in that it comprises a user command which permits a user to make a selection in said multimedia content, extraction means for extracting from said multimedia content one or more context data relating to said selection, means for recovering one or more descriptions in said document from said context data, and automatic formulation means based on recovered descriptions, of a question intended to be transmitted to a search engine.
- the invention permits a user who is reading multimedia content to launch a search relating to that which he is reading in the multimedia content, without having to formulate himself the question to be transmitted to the search engine.
- the only thing that the user has to do is to make a selection in the multimedia content. This selection is then used automatically for formulating the question by using descriptions recovered from the document that describes the multimedia content.
- the user thus: neither has to choose keywords relevant for his search, which is generally complex enough (generally various attempts with various combinations of keywords are necessary for a non-specialist user to obtain a satisfactory result), nor to seize keywords to be used for his search, which is difficult, if not impossible with equipment that has no alphabetic keyboard, for example, with a television decoder, a personal assistant, a mobile telephone ....
- the multimedia content contains a plurality of multimedia entities associated with a reading time
- the document comprises descriptions relating to one or various multimedia entities which may be recovered from a reading time and the current reading time at the moment of the selection forms context information.
- the multimedia content is formed, for example, by a video.
- the current reading time of the video is recovered. This current reading time is used for finding the descriptions of the document that relate to the passage of the video selected by the user.
- the multimedia content contains objects identified by an object identifier
- the document comprises descriptions relating to one or various objects that may be recovered from an object identifier
- the user command comprises an object selection tool and the object identifier of the selected object forms context information.
- the multimedia content is, for example, an image containing various objects that the user can select, for example, with the aid of a mouse-type selection tool, or with a stylus for a touch screen.
- the identifier of this object is recovered from the multimedia content and it is used for finding descriptions of the document that relate to the selected object.
- said document is a tree structure of father and son nodes containing one or more descriptions that are instances of one or more descriptors, a description contained in a father node being valid for a son node when no other node from the father node to the son node contains another instance description of the same descriptor, and said description recovery means comparing the context information with instances of one or more descriptors called recovery descriptors for selecting a node in the tree-like structure, and recover other descriptions which are also valid for this node.
- This embodiment is advantageous when the multimedia content is formed by a video and when the document is structured in the following fashion: the node of the first hierarchical level (root of the tree) corresponds to the complete video, the nodes of the second hierarchical level correspond to various scenes of the video, the nodes of the third hierarchical level correspond to the shots of the various scenes ....
- the descriptions which are valid for a father node are thus valid for its son nodes.
- the invention comprises searching for a start node, recovering other descriptions which are also valid for this start node, then going back in the tree step by step for recovering at each hierarchical level descriptions which are instances of descriptors for which no instance has yet been recovered.
- the start node is the node that contains the description which is an instance of the recovery descriptor and that matches with the context information.
- the invention permits to refine a question and thus to better focus the search.
- Fig. 1 is a block diagram of an example of equipment according to the invention
- Fig. 2 is a diagram of a tree-like structure of an example of a document according to the invention.
- Fig. 3 is a diagram explaining the principle of the invention
- Fig. 4 is a functional diagram of an example of a system according to the invention.
- Fig. 1 is shown a functional diagram of an example of equipment according to the invention.
- equipment according to the invention comprises: a content reader DEC-C for reading multimedia content C, a user command CDE for making a selection S from the multimedia content when the multimedia content C is being read, a document reader DEC-D which receives, from the content reader DEC-C, one or more context data Xi relating to the selection S and which uses the context data Xi for reading a document D that describes the multimedia content C so as to supply descriptions Aj relating to this or these context data Xi, a tool QUEST for automatically formulating a question to formulate a question K based on descriptions Aj read in the document D.
- a content reader DEC-C for reading multimedia content C
- a user command CDE for making a selection S from the multimedia content when the multimedia content C is being read
- a document reader DEC-D which receives, from the content reader DEC-C, one or more context data Xi relating to the selection S and which
- the multimedia content C is an MPEG-4 video
- the content reader DEC-C is an MPEG-4 decoder
- the document D is an MPEG-7 document
- the document reader DEC-D is an MPEG-7 decoder
- the multimedia content is a video
- a reading time is associated with each image in the multimedia content.
- the user command is constituted, for example, by a simple button.
- the content reader DEC-C supplies the current reading time of the video (the current reading time is the reading time associated in the multimedia content with the image that is being read at the moment of the selection). This current reading time is then used as context information to find the descriptions of the document that relate to the passage of the video that is selected by the user.
- an object identifier is associated with each object in the multimedia content.
- the user command is formed, for example, by a mouse.
- the content reader DEC-C supplies the object identifier that is associated to the selected object in the multimedia content.
- This object identifier is then used as context information to find the descriptions of the document that relate to the selected object.
- the user command is, for example, a mouse which permits the user to select an object in an image of the video.
- the current reading time and the object identifier are advantageously used as context data.
- Fig. 2 is shown an example of a tree-like structure of a document D of multimedia content C.
- this tree-like structure comprises: a first hierarchical level LI comprising a root node NO which represents the whole the multimedia content, a second hierarchical level L2 comprising three nodes Nl to N3 which represent a first, a second and a third part of the multimedia content respectively (for example, when the multimedia content is a video, each part corresponds to a different scene of the video), a third hierarchical level L3 comprising two nodes N21 and N22 which are son nodes of the node N2, and three other nodes N31, N32 and N33 which are sons of the node N3.
- the nodes N21 and N22 represent a first and a second portion of the second part of the multimedia content, respectively.
- the nodes N31, N32 and N33 represent a first, a second and a third portion of the third part of the multimedia content.
- each portion corresponds to a shot of a scene of the video.
- the nodes of the tree-like structure advantageously comprise descriptions which are instances of descriptors (a descriptor is a representation of a characteristic of all or part of the multimedia content).
- the context data must thus be such that they can be compared with the content of an instance of one of the descriptors used in the document that describes the multimedia content.
- the descriptors used for this comparison are called recovery descriptors.
- the MPEG-7 standard defines a certain number of descriptors, notably a descriptor «MediaTime» which indicates the start time and end time of a video segment, as well as semantic descriptors, for example, the descriptors «who», «what», «when», «how» ....
- the current reading time is advantageously used as context information and the content of the descriptions that are instances of the descriptor «MediaTime» is compared with the current reading time to find in the document the node corresponding to the selected segment. Then descriptions that are instances of the descriptors «who», «what», «when» and «how» are recovered for formulating the question.
- the MPEG-4 and MPEG-7 standards also define object descriptors notably an object identification descriptor.
- the objects of a multimedia content are identified in said multimedia content by a description that is an instance of this object identification descriptor.
- This description is also contained in the MPEG-7 document. It can thus be used as context information when the user selects an object. In that case the recovery descriptor is formed by the object identification descriptor.
- descriptions contained in a father node are also valid for its son nodes. For example, a description that is an instance of the descriptor «where», relating to the whole video, remains valid for all the scenes and all the video shots. However, more precise descriptions, instances of the same descriptor, may be given for son nodes. These more precise descriptions are not valid for the whole video.
- the description «France» is valid for the whole video
- the description «Paris» is valid for a scene SCENEl
- the descriptions «Montmartre» and «Palais Royal» are valid for a first and a second shot SHOT1 and SHOT2 of the scene SCENEl.
- the tree-like structure is passed through from a start node, son nodes to a father node. And for each hierarchical level, a description is only recovered if no other instance of the same descriptor has been recovered yet. If we take the previous example, when the user selects the shot SHOT1, it is the description «Montmartre» that is used for formulating the question. And when the user selects a third shot SHOT3 of the scene SCENEl, which does not contain an instance of the descriptor «where», the description «Paris» is used.
- Fig. 3 is shown a diagram summarizing the detailed course of a method according to the invention of formulating a question intended to be transmitted to a search engine.
- the user presses the selection key CDE to select a passage of a video
- the current reading time T at the moment of the selection is recovered.
- the current reading time T constitutes the context information.
- the node that comprises an instance description of the recovery descriptor «MediaTime» containing a start time Ti and an end time Tf that define a time range in which the current reading time T is included is searched for in the document D.
- the node that matches this condition is node N31.
- the branch B 1 that carries the node N31 is passed through from the node N31 to the root NO to recover the descriptions Dl, D2 and D3 which are instances of the descriptors «who», «what» and «where».
- the descriptions Dl, D2 and D3 are used for generating a question K.
- Fig. 4 is represented an example of a system according to the invention.
- Such a system comprises a remote search engine SE accommodated on a server SV. It also comprises user equipment according to the invention referred to as EQT which permits a user to read multimedia content C, to make a selection from the multimedia content during the reading so as to launch a search for the selected passage.
- the equipment EQT comprises in addition to the elements already described with reference to Fig. 1 a transceiver EX/RX for transmitting a question K to the search engine SE and receiving a response R coming from the search engine SE. It finally comprises a transmission network TR for transmitting the question K and the response R.
- equipment according to the invention comprises one or more processors and one or more program storage memories, said programs containing instructions for implementing functions that have just been described when they are executed by said processors.
- the invention is independent of the video format used. By way of example it is notably applicable to the MPEG-1, MPEG-2 and MPEG4 formats.
Abstract
Description
Claims
Priority Applications (5)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
BR0205949-5A BR0205949A (en) | 2001-08-28 | 2002-08-22 | Electronic equipment including a reading medium for reading multimedia content that is described in a document containing descriptions, method for asking a question, program including program code instructions for implementing said method, and system including equipment |
US10/487,734 US20050076055A1 (en) | 2001-08-28 | 2002-08-22 | Automatic question formulation from a user selection in multimedia content |
KR10-2004-7003001A KR20040031026A (en) | 2001-08-28 | 2002-08-22 | Automatic question formulation from a user selection in multimedia content |
JP2003523405A JP2005501343A (en) | 2001-08-28 | 2002-08-22 | Automatic question construction from user selection in multimedia content |
EP02758737A EP1423803A1 (en) | 2001-08-28 | 2002-08-22 | Automatic question formulation from a user selection in multimedia content |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
FR0111184 | 2001-08-28 | ||
FR0111184 | 2001-08-28 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2003019416A1 true WO2003019416A1 (en) | 2003-03-06 |
Family
ID=8866781
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/IB2002/003464 WO2003019416A1 (en) | 2001-08-28 | 2002-08-22 | Automatic question formulation from a user selection in multimedia content |
Country Status (7)
Country | Link |
---|---|
US (1) | US20050076055A1 (en) |
EP (1) | EP1423803A1 (en) |
JP (1) | JP2005501343A (en) |
KR (1) | KR20040031026A (en) |
CN (1) | CN1549982A (en) |
BR (1) | BR0205949A (en) |
WO (1) | WO2003019416A1 (en) |
Families Citing this family (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6735253B1 (en) | 1997-05-16 | 2004-05-11 | The Trustees Of Columbia University In The City Of New York | Methods and architecture for indexing and editing compressed video over the world wide web |
JP2002529858A (en) * | 1998-11-06 | 2002-09-10 | ザ トゥルスティーズ オブ コロンビア ユニバーシティ イン ザ シティ オブニュー ヨーク | System and method for interoperable multimedia content description |
US7143434B1 (en) * | 1998-11-06 | 2006-11-28 | Seungyup Paek | Video description system and method |
AU2002351310A1 (en) * | 2001-12-06 | 2003-06-23 | The Trustees Of Columbia University In The City Of New York | System and method for extracting text captions from video and generating video summaries |
KR20050007348A (en) * | 2002-04-26 | 2005-01-17 | 더 트러스티스 오브 콜롬비아 유니버시티 인 더 시티 오브 뉴욕 | Method and system for optimal video transcoding based on utility function descriptors |
WO2006096612A2 (en) * | 2005-03-04 | 2006-09-14 | The Trustees Of Columbia University In The City Of New York | System and method for motion estimation and mode decision for low-complexity h.264 decoder |
US7465241B2 (en) * | 2007-03-23 | 2008-12-16 | Acushnet Company | Functionalized, crosslinked, rubber nanoparticles for use in golf ball castable thermoset layers |
KR100961444B1 (en) | 2007-04-23 | 2010-06-09 | 한국전자통신연구원 | Method and apparatus for retrieving multimedia contents |
WO2009126785A2 (en) * | 2008-04-10 | 2009-10-15 | The Trustees Of Columbia University In The City Of New York | Systems and methods for image archaeology |
CN102067113A (en) * | 2008-04-24 | 2011-05-18 | 龙搜(北京)科技有限公司 | System and method for knowledge-based input in a browser |
WO2009155281A1 (en) * | 2008-06-17 | 2009-12-23 | The Trustees Of Columbia University In The City Of New York | System and method for dynamically and interactively searching media data |
US8671069B2 (en) | 2008-12-22 | 2014-03-11 | The Trustees Of Columbia University, In The City Of New York | Rapid image annotation via brain state decoding and visual pattern mining |
CN101771957B (en) * | 2008-12-26 | 2012-10-03 | 中国移动通信集团公司 | User interest point determining method and device |
KR101110202B1 (en) * | 2010-08-02 | 2012-02-16 | (주)엔써즈 | Method and system for generating database based on mutual relation between moving picture data |
KR20160014463A (en) * | 2014-07-29 | 2016-02-11 | 삼성전자주식회사 | Server, providing metheod of server, display apparatus, controlling metheod of display apparatus and informatino providing system |
Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2001003002A2 (en) * | 1999-07-02 | 2001-01-11 | Koninklijke Philips Electronics N.V. | Meta-descriptor for multimedia information |
Family Cites Families (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO1996017313A1 (en) * | 1994-11-18 | 1996-06-06 | Oracle Corporation | Method and apparatus for indexing multimedia information streams |
US5774666A (en) * | 1996-10-18 | 1998-06-30 | Silicon Graphics, Inc. | System and method for displaying uniform network resource locators embedded in time-based medium |
US6631522B1 (en) * | 1998-01-20 | 2003-10-07 | David Erdelyi | Method and system for indexing, sorting, and displaying a video database |
US6564263B1 (en) * | 1998-12-04 | 2003-05-13 | International Business Machines Corporation | Multimedia content description framework |
JP2001134589A (en) * | 1999-11-05 | 2001-05-18 | Nippon Hoso Kyokai <Nhk> | Moving picture retrieving device |
-
2002
- 2002-08-22 WO PCT/IB2002/003464 patent/WO2003019416A1/en active Application Filing
- 2002-08-22 BR BR0205949-5A patent/BR0205949A/en not_active IP Right Cessation
- 2002-08-22 CN CNA028168186A patent/CN1549982A/en active Pending
- 2002-08-22 JP JP2003523405A patent/JP2005501343A/en active Pending
- 2002-08-22 KR KR10-2004-7003001A patent/KR20040031026A/en not_active Application Discontinuation
- 2002-08-22 EP EP02758737A patent/EP1423803A1/en not_active Withdrawn
- 2002-08-22 US US10/487,734 patent/US20050076055A1/en not_active Abandoned
Patent Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2001003002A2 (en) * | 1999-07-02 | 2001-01-11 | Koninklijke Philips Electronics N.V. | Meta-descriptor for multimedia information |
Non-Patent Citations (2)
Title |
---|
"TEXT OF ISO/IEC FCD 15938-1 INFORMATION TECHNOLOGY - MULTIMEDIA CONTENT DESCRIPTION INTERFACE - PART 1 SYSTEMS", ISO/IEC JTC1/SC29/WG11 MPEG01/N4001, XX, XX, March 2001 (2001-03-01), pages 1 - 2,I-V,6-58, XP001001465 * |
HIRATA K ET AL: "CONTENT-ORIENTED INTEGRATION IN HYPERMEDIA SYSTEMS", HYPERTEXT '96. 7TH. ACM CONFERENCE ON HYPERTEXT. WASHINGTON, MAR. 16 - 20, 1996, ACM CONFERENCE ON HYPERTEXT, NEW YORK, ACM, US, vol. CONF. 7, 16 March 1996 (1996-03-16), pages 11 - 21, XP000724319, ISBN: 0-89791-778-2 * |
Also Published As
Publication number | Publication date |
---|---|
BR0205949A (en) | 2003-12-23 |
JP2005501343A (en) | 2005-01-13 |
EP1423803A1 (en) | 2004-06-02 |
US20050076055A1 (en) | 2005-04-07 |
CN1549982A (en) | 2004-11-24 |
KR20040031026A (en) | 2004-04-09 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11468109B2 (en) | Searching for segments based on an ontology | |
US20050076055A1 (en) | Automatic question formulation from a user selection in multimedia content | |
US7181757B1 (en) | Video summary description scheme and method and system of video summary description data generation for efficient overview and browsing | |
EP1125245B1 (en) | Image description system and method | |
US7653635B1 (en) | Systems and methods for interoperable multimedia content descriptions | |
US20090077034A1 (en) | Personal ordered multimedia data service method and apparatuses thereof | |
US20040172410A1 (en) | Content management system | |
KR20010086393A (en) | Method and apparatus for linking a video segment to another video segment or information source | |
CN102999498A (en) | Method and device for searching multi-media programs | |
KR101404596B1 (en) | System and method for providing video service based on image data | |
JP2013529331A (en) | Automatic image discovery and recommendation for viewing television content | |
US20060085416A1 (en) | Information reading method and information reading device | |
Ma et al. | WebTelop: Dynamic tv-content augmentation by using web pages | |
US20080016068A1 (en) | Media-personality information search system, media-personality information acquiring apparatus, media-personality information search apparatus, and method and program therefor | |
KR20030062585A (en) | Multimedia data description of content-based image retrieval | |
Dao et al. | A new spatio-temporal method for event detection and personalized retrieval of sports video | |
EP1935183A1 (en) | Method and apparatus for encoding multimedia contents and method and system for applying encoded multimedia contents | |
Cho et al. | News video retrieval using automatic indexing of korean closed-caption | |
Lalmas et al. | Searching multimedia data using MPEG-7 descriptions in a broadcast terminal | |
GB2485573A (en) | Identifying a Selected Region of Interest in Video Images, and providing Additional Information Relating to the Region of Interest | |
Masumitsu et al. | Meta-data framework for constructing individualized video digest | |
Nitta | Semantic content analysis of broadcasted sports videos with intermodal collaboration | |
Ekin et al. | Integrated semantic-syntactic video event modeling for search and retrieval | |
Shih et al. | Content-based scalable sports video retrieval system | |
Kuo et al. | An MPEG-7 content-based analysis/retrieval system and its applications |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AK | Designated states |
Kind code of ref document: A1 Designated state(s): BR CN IN JP KR US Kind code of ref document: A1 Designated state(s): BR CN IN JP KR |
|
AL | Designated countries for regional patents |
Kind code of ref document: A1 Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR IE IT LU MC NL PT SE SK TR Kind code of ref document: A1 Designated state(s): AT BE BG CH CY CZ DE DK EE ES FR GB GR IE IT LU MC NL PT SE SK TR |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application | ||
WWE | Wipo information: entry into national phase |
Ref document number: 2002758737 Country of ref document: EP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 10487734 Country of ref document: US |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2003523405 Country of ref document: JP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 20028168186 Country of ref document: CN Ref document number: 429/CHENP/2004 Country of ref document: IN Ref document number: 1020047003001 Country of ref document: KR |
|
WWP | Wipo information: published in national office |
Ref document number: 2002758737 Country of ref document: EP |