JP3915888B2 - Multimedia synchronous search method - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention is an information integration field in which videos and materials are integrated and provided, and the materials are searched by keywords while synchronizing the original videos and materials without modification, and the videos corresponding to the materials are searched. The present invention relates to a multimedia synchronous search method for extracting scenes at high speed.
[0002]
[Prior art]
Traditionally, if you're thinking about searching for presentation materials (here, documents that consist of text and figures) and videos of videos that are explained using the presentation materials, The material was a document, and the movie was a movie.
[0003]
In particular, in the case of video search, mechanisms for extracting and indexing scenes of news videos and sports program videos have already been studied, but lecture videos, papers for publications that are mainly targeted in the present invention, In lecture videos, research meeting videos, PR presentation videos, etc., scene switching and movement are less than news and sports programs, so it is difficult to cut out and index scenes. In addition, it took a large amount of money to add a valid keyword even if the scene was cut out.
[0004]
Also, from the viewpoint of the combination of video and presentation materials, the network research project WIDE (Widely Integrated Distributed Environment) SOIWG (Schoo10n the Internet Working Group) also shows videos and presentation materials simultaneously ( http://www.soi.wide.ad.jp/). However, it treats presentation materials as a kind of video, and only two videos that start and end simultaneously can be seen at the same time. That is, it does not have a search function that the present invention intends to perform.
[0005]
There is research to manage video frames interactively on the web (Rune Hjelsvold and Subu Vdaygiri and Yves Leate, Web-based Personalization and Management of Interactive Video, Proc of WWW10, pp.129-139, 2001), but we focused on personalization. However, it does not have a search mechanism like the present invention. Another related tool is an authoring tool that incorporates video.
[0006]
[Problems to be solved by the invention]
However, such an authoring tool will cost as much as creating a new material. Even if an integrated view is provided, the original material remains as it is, and it must be handled as the original material alone. For example, even if you want to update the contents of the previous presentation material, if you change it to the authoring tool format, you will not be able to change the presentation material freely. Furthermore, an index is necessary to efficiently retrieve and provide necessary information. The index needs to indicate the position (scenes such as videos) within the content, and to search for content that contains the necessary information among many contents, and if possible, these While it is desirable for the indexes to be integrated, it is not easy to provide such an integrated index at the individual authoring tool level.
[0007]
The present invention has been made in view of the problems of the prior art, and while searching for a presentation material by a keyword while synchronizing the original video and the presentation material without modification, the video corresponding to the presentation material is searched for. The purpose is to provide a multimedia synchronized search method for extracting scenes at high speed.
[0008]
[Means for Solving the Problems]
The present invention relates to a multimedia synchronized search method for retrieving presentation materials by keywords and extracting a scene of a moving image corresponding to the presentation materials at high speed while synchronizing the original movie and the presentation materials without any change. The above-mentioned object of the present invention is to search for presentation materials by a keyword input by a user, find a page of the material corresponding to the keyword, and at the same time search for and display a video scene corresponding to the page. A search method, in which an index that associates a keyword with a page of a presentation material in which the keyword is described is created in advance, and a pointer to the material page and one or a plurality of locations where the pointer is presented Pointer to the video scene An intra-group index consisting of a synchronous block corresponding to a keyword in the page of the material and a position block consisting of a set of addresses where the keyword is described, and a plurality of the intra-group index A set block connected with pointers to, an inter-set index for the plurality of set blocks,
The inter-group index is searched according to the inputted keyword by the synchronous search index consisting of the following: a pair block pointing to a pair of a page of the material including the keyword and a moving image, and a pointer at the head of the found pair block A step of searching for a position block corresponding to the keyword by a group index, and a pointer to a synchronous block in the searched position block are repeated, and according to a display page pointer in the synchronous block, a document page and This is achieved by a multimedia synchronous search method comprising a step of displaying a pair of presented moving image scenes.
[0009]
The above-mentioned object of the present invention can be achieved more effectively by connecting pointers to moving image scenes in the synchronized block in a list structure, or by connecting the position blocks in a list structure.
[0010]
DETAILED DESCRIPTION OF THE INVENTION
The present invention searches for a material according to a keyword while synchronizing moving image data and presentation material (hereinafter referred to as “material”) stored as digital data, and stores the moving image data corresponding to the retrieved material page. The present invention relates to a multimedia synchronous search method for extracting scenes, and FIG. 1 is a conceptual diagram thereof.
[0011]
In the figure, the multimedia synchronous search mechanism searches the material according to the keyword given by the user, extracts the page in the material where the keyword appears, and uses the scene of the video data corresponding to that page together with the material page Return to the person.
[0012]
In summary, first create an index according to the content of the presentation material, associate the page of the presentation material with the scene of the video when the page is presented, and when a search request comes in, first enter a keyword. The presentation material is searched for, and a page corresponding to the keyword is found, and at the same time, a moving image scene corresponding to the page is found.
[0013]
The multimedia synchronization search mechanism has a synchronization block for associating a page of the material with a scene (s) in the moving image when the page is presented. The synchronization block holds a page and position information (of a plurality of scenes) in the moving image where the page has started to be posted. It also has a location block with an address indicating the location of the keyword in the material page along with a pointer to the synchronization block. By connecting the position blocks in a list structure, it is possible to cope with a case where a plurality of the same keywords appear. Furthermore, an index hierarchical structure is adopted, and an index for a set of a plurality of materials and moving image data is combined.
[0014]
FIG. 2 shows a configuration example of an index (index) for performing the multimedia synchronous search, which will be described based on this.
[0015]
Here, for example, when the material is a lecture material and the moving image data is a lecture video, the lecture video and the lecture material correspond to one lecture (group), and the inter-group index is used for a plurality of lectures. Points to the index. The intra-group index means an index within the lecture.
[0016]
The multimedia synchronization search index includes a synchronization block 1 composed of a pointer to a document page, a sequence of pointers to one or a plurality of moving picture scenes in which the pointer is presented, and the keyword corresponding to the keyword. To the position block 2 composed of a pair of addresses where the keyword in the page of the synchronization block 1 and the material is used, the intra-group index 3 for the pair of the material and the moving image, and a plurality of intra-group indexes 3 Are composed of a set block 4 to which the pointers are connected and an inter-set index 5 for the set block 4. Needless to say, when there is only one set of material and moving image, the set block 4 and the set index 5 are not necessary.
[0017]
In order to create a set of material and moving image data, the page of the material and the frame number of the recorded moving image are stored as a synchronization block when the material is presented. Next, the material is scanned to extract a keyword, a position block for the keyword is created, and an intra-group index 3 is constructed so that the position block 2 is pointed from the index. Further, when there are a plurality of sets, a set block 4 is created according to the keywords included in the set, and the set block 4 is pointed to from the set index 5. The intra-group index 3 and the inter-group index 5 may be an index using a tree structure such as a B-tree or an index using hashing (hash method).
[0018]
By doing so, it is possible to present a moving image scene corresponding to the keyword included in the material. Even if there are multiple pages of documents that contain the keyword, or even if a page of a document corresponds to multiple times in a video scene, it is possible to perform a search at high speed. Even if there are multiple materials and moving image data, it is possible to present a document page and moving image scene corresponding to the keyword at high speed.
[0019]
Next, the search procedure will be described based on the flowchart of FIG.
[0020]
First, the user inputs a keyword on the search screen (FIG. 5) and clicks the search key (step S201).
[0021]
When a keyword is input, the inter-group index 5 is repeated according to the given keyword, and a group block 4 indicating a group of material and moving image including the keyword is searched for (step S202).
[0022]
Next, the position block 2 corresponding to the keyword is searched by the group index 3 of the group pointed to by the head pointer of the group block 4 (step S203).
[0023]
Further, the pointer to the synchronization block 1 in the position block 2 is repeated, and according to the display page pointer in the synchronization block 1, the material page and the thumbnail of the moving image scene are displayed (FIG. 6, step S204).
[0024]
In this step S204, only the document page may be displayed, and the thumbnail of the moving image scene may be displayed in the case of “YES” in step S205 described later.
[0025]
If the page of the material is not what the user is looking for (click “No” in FIG. 6), it moves to the next location block, provided that it is not the last location block. (Step S211), the pointer to the synchronization block 1 in the position block 2 is repeated, and the thumbnail of the page of the material and the scene of the moving image is displayed according to the pointer of the display page in the synchronization block 1 (step S204). On the search screen (FIG. 6), the thumbnail of the material changes.
[0026]
In this way, the process is repeated until the page of the material that the user is looking for is found. If no search is made up to the last set block, the process ends without being detected (step S214). At this time, the screen of FIG. 9 is displayed.
[0027]
If the thumbnail of the displayed document page is what the user is trying to find (YES in step S205), clicking “Yes” in FIG. 6 switches to the screen in FIG. If the scene thumbnail is what the user is looking for, click “Yes” to display the enlarged screen of the document and the scene (FIG. 8), and the search ends (step S207).
[0028]
If the scene displayed in step S206 (FIG. 7) is not the scene that the user is trying to find, clicking “No” in FIG. 7 indicates that the scene is not the last scene in the same synchronization block. (NO in step S208), the next scene in the same synchronization block is displayed (in FIG. 7, only the thumbnail of the scene is switched). In this way, the process is repeated until the scene that the user is looking for is found.
[0029]
If there is no scene being searched for in the same synchronization block, the process moves to the next position block (step S211), and the procedure from step S204 is repeated.
[0030]
If the desired document page / scene pair is not found even after repeating the above procedure up to the last position block in the same set block (YES in step S210), the process moves to the next set block (step S213), step Repeat the procedure from S203.
[0031]
The search is performed up to the last set block. If no search is found, the screen of FIG. 9 is displayed and the search ends.
[0032]
Further, when a plurality of keywords are given, it is possible to narrow down by taking a logical sum or a logical product with the synchronous blocks searched by the respective keywords.
[0033]
FIG. 4 is a diagram showing a system for realizing the functions as described above. In the figure, reference numeral 30 denotes a center that provides such integrated multimedia information. is there.
[0034]
In the center 30, a web server 32, a web search engine 33, and a moving image server 34 are connected via a network switch 35, and hard disks 32a, 33a, 34a having sufficient capacity are connected to the respective machines. ing. Here, data is stored in the hard disk 32a, a search index is stored in the hard disk 33a, and moving image data is stored in the hard disk 34a.
[0035]
Each material such as material and moving image data stored in the hard disk is basically stored as provided so that the person who created the material can modify it. For example, if a document is provided in PDF or PPT, its contents will not be changed to XML. With regard to moving images, in consideration of the communication speed on the user side, moving images having different formats such as MPEG2, MPEG4, and Real are stored.
[0036]
With the above configuration, the intra-group index 3 and the inter-group index 5 are placed in the Web search engine 33. That is, the synchronization block is placed in the Web search engine 33, and has a pointer that points to an image frame in the moving image server 34 and a document page in the Web server 32. It should be noted that not only the above-described index (index) is used for moving image search, but MPEG7 is used together, or annotation, additional information, etc. are described using XML.
[0037]
The moving image server 34 is tightly coupled with the web server 32 and the web search engine 33, and has a function of providing a VOD (Video On Demand) service capable of playing a video from an arbitrary position.
[0038]
The reason why the Web server 32 and the Web search engine 33 are used together is to prevent the performance of the Web server 32 from being deteriorated due to the load of search processing.
[0039]
An authoring tool 36 for editing and preparing content is also connected to each machine via the network switch 35.
[0040]
In addition, in preparation for an increase in the number of users, a plurality of cache servers 37 for maintaining the rapid provision of data are also prepared and connected to a network switch.
[0041]
The video camera 38 is used to take pictures of lecture scenes in the classroom, and the taken moving image data is sent to the moving image server 34 via the network switch 35 and stored in the hard disk 34a. The projector 39 projects material and moving image data onto a screen (not shown) in the classroom.
[0042]
The backup system 40 is installed so as not to stop the operation even when the server or the like is down.
[0043]
As an example of actual operation in the above configuration, the user accesses the Web server 32 from the terminal 10 through the Internet 20 or further through the gateway 31 in the center 30, and inputs a keyword from the displayed screen, and the network switch The Web search engine 33 connected to 35 searches using the synchronization block index, and returns the synchronization block to the user terminal 10 in a form including the result in JavaScript.
[0044]
The user requests necessary data from the Web server 32 and the moving image server 34 in accordance with the JavaScript and the synchronization block information (including inter-group index information). The web server 32 and the moving image server 34 that have requested the data perform a synchronous search by picking up necessary data according to the synchronization block and returning it to the user terminal 10.
[0045]
In addition, the above example is a method in which processing written in JavaScript is performed on the user side, but it is not particularly limited to JavaScript, and all processing may be performed on the search engine side. Needless to say.
[0046]
【The invention's effect】
So far, even if there is presentation material associated with the video, only the index is provided for the video and the material for the video, but there is no index (index) to search while synchronizing the material and the video .
[0047]
With the present invention, for example, it has become possible to synchronize lecture videos and lecture materials, search for lecture materials by keyword, and view lecture videos of scenes in which the content is being lectured. The effect can be obtained.
[0048]
Also, when there is a set of multiple materials and videos as well as a set of one material and video, a keyword is given to select a set of materials and videos that contain the keyword from among them, and corresponding scenes Can also be output.
[0049]
As described above, the multimedia synchronous search method according to the present invention can be applied to various fields such as a paper for presentation papers, lecture videos, study videos, and PR presentation videos, and is expected to have a great effect. The
[Brief description of the drawings]
FIG. 1 is a diagram showing a concept of a multimedia synchronous search method according to the present invention.
FIG. 2 is a diagram illustrating a configuration example of an index (index) for performing multimedia synchronous search;
FIG. 3 is a flowchart showing a multimedia synchronous search procedure according to the present invention.
FIG. 4 is a diagram showing a system configuration of multimedia synchronous search according to the present invention.
FIG. 5 is an example of a keyword input screen.
FIG. 6 is an example of a search result display screen.
FIG. 7 is an example of a search result display screen.
FIG. 8 is an example of a search result display screen.
FIG. 9 is an example of a search result display screen.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 Synchronous block 2 Position block 3 Set index 4 Set block 5 Set index 10 User terminal 20 Internet 30 Center 31 Gateway 32 Web server 33 Web search engine 34 Movie server 35 Network switch 36 Authoring tool 37 Cache server 38 Video camera 39 Projector 40 Backup system

Claims (5)

Find the presentation materials by keyword for a user to input, at the same time to extract the relevant page of the presentation materials corresponding to the keyword, multimedia synchronization and display find a video of the scene corresponding to the extracted pages search A method comprising the steps of:
A keyword is extracted from the presentation material in advance, and information on the position of the keyword in a plurality of pages where the keyword appears is associated with the keyword in the presentation material, and a page on which the keyword appears is presented. Holds information about the scenes in the video
When the keyword is input, all corresponding pages of the presentation material corresponding to the keyword are displayed as candidates,
Displaying the scene in the video associated with the one page selected by the user as a candidate,
A multimedia synchronized search method characterized in that, starting from a scene in the moving picture selected by the user, a subsequent moving picture and presentation materials synchronized with the moving picture are displayed in pairs . A multimedia synchronous search method for searching for presentation materials by a keyword input by a user, searching for a page of the material corresponding to the keyword, and simultaneously searching for and displaying a video scene corresponding to the page, the method Is
An index that associates a keyword with a page of presentation material in which the keyword is described in advance is created, and a pointer to the page of the presentation material and a pointer to a scene of one or more moving images where the keyword is presented and synchronization block consisting of a column,
Corresponding to the keyword according to the input keyword by a synchronous search index consisting of the synchronous block corresponding to the keyword in the page of the presentation material and a position block consisting of a set of addresses of positions where the keyword is described Searching for a position block to perform,
Hauling a pointer to sync blocks in the searched position block, according to the display page pointer in the synchronizing block, and displaying the scene of a moving image and a page of the presentation materials which it is presented in pairs,
A multimedia synchronous search method comprising: 3. The multimedia synchronized search method according to claim 2, wherein pointers to moving image scenes in the synchronized block are connected in a list structure. 4. The multimedia synchronous search method according to claim 2, wherein the position blocks are connected in a list structure.   The user terminal, a web server storing the presentation material, a web search engine storing the synchronous search index, and a moving image server storing the moving image data are connected via a network,
  When the user inputs the keyword from the terminal, the presentation material page searched by the web search engine and a video scene on which the page is presented are displayed in a pair on the terminal. Item 5. The multimedia synchronized search method according to any one of Items 2 to 4.

Images