JP2006085379A - Information processor and its control method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an information processor and its control method, and a program for efficiently and suitably presenting a retrieval result intended by a user. <P>SOLUTION: Multi-media data are retrieved on the basis of retrieval conditions. A desired retrieval object is set from retrieval purpose candidates. The multi-medial data acquired as a retrieval result are analyzed according to an analytic rule corresponding to the retrieval conditions. A display rule corresponding to the set retrieval purpose is selected, and display contents to be displayed as the retrieval result are decided on the basis of the selected display rule and the analytic result. The retrieval result is displayed according to the display contents. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an information processing apparatus that searches for multimedia data composed of at least one of document data, image data, audio data, or a combination thereof, a control method thereof, and a program. is there.

  With the high performance of small computer systems and the increase in capacity of storage devices, data such as document data, image data, audio data, and other mixed data have come to be distributed. Furthermore, with the spread of the Internet, which is a global computer network in recent years, and the lower prices of mobile communication devices, it has become an era when these data can be easily obtained and enjoyed. The amount of information handled by individuals has increased, and the function of searching for such information has become very important.

  In order to efficiently use such a large amount of multimedia data, various search methods and devices equipped with them have been proposed. For example, a method of searching for multimedia data that matches a search condition by managing the multimedia data in a storage device such as a database, adding metadata such as feature amounts and keywords, and providing a search condition There is.

  Since the amount of information handled by individuals is large and the contents are diversified and complicated, it is difficult to obtain only the necessary information from a large database according to search conditions. Normally, users narrow down candidates by search conditions, You have to search for the desired information. When displaying the search results, it is common to display a list of identification information such as candidate titles and thumbnails that meet the search conditions, and the user can select this identification information to display more detailed information. Let

  In a conventional multimedia data search device for searching multimedia data, a list of search results is displayed according to the suitability with the search query, such as being ranked according to the suitability and similarity with the search condition. There were many (for example, refer patent document 1).

In the future, it is desirable not only to provide a search function for multimedia data, but also to display search results so that the user can easily find and use desired information.
JP 2001-109779 A

  When a large amount of information is searched and used, such as information collection, necessary information must be searched, selected and arranged according to the purpose from the search results.

  In the conventional method, since a list of search results is displayed according to the compatibility with the search query, information of various contents related to the search query is displayed in an inconsistent manner. In other words, since the display considering the relevance and connection due to the contents of a large number of data obtained as a search result is not performed, when viewing the search result, the user may review the information once viewed many times. The contents must be memorized and organized in the head, and adjusted in consideration of the browsing order of the information. In particular, when the number of search results is large, detailed information must be viewed one by one while repeating scrolling and page breaks, which requires much labor, time, and familiarity.

  The present invention has been made to solve the above-described problems, and an object thereof is to provide an information processing apparatus capable of efficiently and suitably presenting a search result intended by a user, a control method therefor, and a program. And

In order to achieve the above object, an information processing apparatus according to the present invention comprises the following arrangement. That is,
An information processing apparatus for searching multimedia data composed of at least one of document data, image data, or audio data, or a combination thereof,
Storage means for storing the multimedia data to be searched;
Input means for inputting search conditions for searching the multimedia data;
Search means for searching multimedia data from the storage means based on the search condition;
Search purpose candidate holding means for holding search purpose candidates by the search means;
Setting means for setting a desired search purpose from search purpose candidates held by the search purpose candidate holding means;
Analysis rule holding means for holding analysis rules for analyzing multimedia data obtained as a search result by the search means;
An analysis unit that selects an analysis rule corresponding to the search condition from the analysis rule holding unit, and analyzes the multimedia data obtained as a search result by the search unit according to the selected analysis rule;
Display rule holding means for holding display rules related to the search purpose candidates in order to determine the contents to be displayed as the search results;
A display rule corresponding to the search purpose set by the setting unit is selected from the display rule holding unit, and the display is displayed as the search result based on the selected display rule and the analysis result of the analysis unit. Display content determination means for determining the content;
Search result display means for displaying the search result according to the display content determined by the display content determination means.

Preferably, the analysis means classifies the multimedia data obtained as a search result by the search means according to the meaning and content thereof;
A multimedia data obtained as a search result by the search means, and a correlation means for performing a correlation for a group of multimedia data classified by the classification means;
The display content determination means determines display contents to be displayed as the search result based on the classification result by the classification means and the correlation result by the correlation means.

Preferably, the classification means classifies the multimedia data obtained as a search result by the search means according to a hierarchical structure,
The association means associates multimedia data obtained as a search result by the search means with a group of multimedia data hierarchically classified according to the hierarchical structure by the classification means.

Preferably, the analysis unit performs analysis when storing the multimedia data in the storage unit as a search target, and associates the analysis result with the multimedia data as metadata.
The display content determining means determines display content to be displayed as the search result based on metadata associated with the multimedia data.

Preferably, the setting unit acquires an acquisition target candidate related to the search condition from the search purpose candidate holding unit;
Selecting means for selecting a desired search purpose from the search purpose candidates acquired by the acquisition means.

Preferably, the display rule includes a definition for determining a presentation order according to a relationship between the multimedia data,
The display content determination means is a multimedia obtained as a search result of the search means based on the classification result of the classification means, the correlation result of the correlation means, and the search purpose set by the setting means. The configuration of the multimedia data to be displayed is determined from the data, and the presentation order of the multimedia data to be displayed is determined based on the correlation result of the correlation means.

Preferably, the display rule includes a definition of a priority with respect to a relationship between the multimedia data and a presentation order,
The display content determination means includes a group of multimedia data classified by the classification means, a multimedia data obtained as a search result of the plurality of search means, and a group of multimedia data classified by the classification means; When there is a relationship, the presentation order of the multimedia data to be displayed is determined based on the priority.

Preferably, the multimedia data is still image data or moving image data,
The search result display unit continuously reproduces and displays still image data or moving image data as the search result in accordance with the display content determined by the display content determination unit.

Preferably, the classifying unit uses the still image or the moving image data, which is multimedia data obtained as a search result by the searching unit, as a unit of a hierarchy to which the still image or the moving image data belongs. Classify according to the hierarchical structure
The association means associates multimedia data obtained as a search result by the search means with a group of multimedia data classified hierarchically according to the hierarchical structure.

In order to achieve the above object, a method for controlling an information processing apparatus according to the present invention comprises the following arrangement. That is,
Control an information processing apparatus that retrieves multimedia data from a storage unit that stores multimedia data composed of at least one of document data, image data, audio data, or a combination thereof. A control method,
An input step for inputting a search condition for searching the multimedia data;
A search step of searching multimedia data from the storage unit based on the search condition;
A search purpose candidate holding step of holding a search purpose candidate in the search step in a storage medium;
A setting step for setting a desired search purpose from the search purpose candidates held in the storage medium in the search purpose candidate holding step;
An analysis rule holding step for holding an analysis rule for analyzing multimedia data obtained as a search result by the search step in a storage medium;
An analysis step of selecting an analysis rule corresponding to the search condition from the storage medium held in the analysis rule holding step, and analyzing multimedia data obtained as a search result in the search step according to the selected analysis rule When,
In order to determine the content to be displayed as the search result, a display rule holding step for holding a display rule related to the search purpose candidate;
The display rule corresponding to the search purpose set in the setting step is selected from the storage medium held in the display rule holding step, and based on the selected display rule and the analysis result of the analysis step, the A display content determination step for determining display content to be displayed as a search result;
A search result display step of displaying the search result according to the display content determined in the display content determination step.

In order to achieve the above object, a program according to the present invention comprises the following arrangement. That is,
Control an information processing apparatus that retrieves multimedia data from a storage unit that stores multimedia data composed of at least one of document data, image data, audio data, or a combination thereof. A program for realizing control,
A program code of an input process for inputting a search condition for searching the multimedia data;
Based on the search condition, a program code of a search step for searching multimedia data from the storage unit,
A program code of a search purpose candidate holding step for holding a search purpose candidate in the search step in a storage medium;
A program code of a setting step for setting a desired search purpose from the search purpose candidates held in the storage medium in the search purpose candidate holding step;
A program code of an analysis rule holding step for holding an analysis rule for analyzing multimedia data obtained as a search result by the search step in a storage medium;
An analysis step of selecting an analysis rule corresponding to the search condition from the storage medium held in the analysis rule holding step, and analyzing multimedia data obtained as a search result in the search step according to the selected analysis rule Program code,
In order to determine the content to be displayed as the search result, a program code of a display rule holding step for holding a display rule related to the search purpose candidate;
The display rule corresponding to the search purpose set in the setting step is selected from the storage medium held in the display rule holding step, and based on the selected display rule and the analysis result of the analysis step, the A program code of a display content determination step for determining display content to be displayed as a search result;
And a program code of a search result display step for displaying the search result according to the display content determined in the display content determination step.

  ADVANTAGE OF THE INVENTION According to this invention, the information processing apparatus which can present the search result which a user intends efficiently and suitably, its control method, and a program can be provided.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  The present invention provides a search in a system that uses multimedia data composed of at least one of document data, image data (still image or moving image data), audio data, or a combination thereof. When displaying the search results searched by the engine using browsing software or search applications, the data obtained as search results is handled in a specific way so that it can be easily understood and effectively presented to the user. The configuration is provided.

[Embodiment 1]
FIG. 1 is a block diagram showing a basic configuration of a computer apparatus in which a multimedia data search system, which is an information processing apparatus according to Embodiment 1 of the present invention, is constructed.

  In FIG. 1, reference numeral 100 denotes a CPU comprising a microprocessor, which performs operations for various processing, logical judgments, etc., and is connected to each bus via an address bus AB, a control bus CB, and a data bus DB. Control the components. The control content is instructed by a program on the ROM 101 or RAM 102 described later.

  Further, a plurality of computer programs can be operated in parallel by the function of the CPU 100 itself and the computer program mechanism.

  The address bus AB transfers an address signal indicating a component to be controlled by the CPU 100. The control bus CB transfers a control signal of each constituent element to be controlled by the CPU 100. The data bus DB performs data transfer between the component devices.

  Reference numeral 101 denotes a ROM, which is a read-only fixed memory. The ROM 101 stores a computer program area and a data area in which a control procedure by the CPU 100 described later is stored.

  Reference numeral 102 denotes a RAM, which is a writable random access memory that is used as a computer program area and a data area in which a control procedure by the CPU 100 described later is stored, and from each component other than the CPU 100. It is also used as a temporary storage area for various computer programs and various data.

  These storage media such as the ROM 101 and the RAM 102 store various data such as computer programs and data for realizing the data management of the first embodiment, and the CPU 100 reads out and executes the program codes stored in these storage media. The function is realized by doing so, but the type of the storage medium is not limited.

  Also, a storage medium storing the program and data according to the present invention is supplied to the system or apparatus, and the program is copied from the storage medium onto the rewritable storage medium such as the RAM 102. As a storage medium, a CD-ROM, a floppy (registered trademark) disk, a CD-ROM, a hard disk, a memory card, a magneto-optical disk, or the like can be used.

  A hard disk (DISK) 103 functions as an external memory for storing various data. The DISK 103 has a built-in storage medium that can read and write a large amount of information at a relatively high speed, and can store and read various data in the storage medium as necessary. The stored computer program and data are called up completely or partially on the RAM 102 when necessary according to an instruction from the keyboard (KB) 107 or instructions from various computer programs.

  Reference numeral 104 denotes a MemCard (memory card), which is a detachable storage medium. Information is stored in this storage medium, and the stored information is referenced and transferred by connecting the storage medium to another device. It becomes possible.

  Reference numeral 107 denotes a keyboard (KB) which includes various function keys such as alphabet key, hiragana key, katakana key, character symbol input key such as a punctuation point, and cursor movement key for instructing cursor movement. A pointing device such as a mouse can also be included.

  A cursor register (CR) 108 stores an address indicating a position displayed on the CRT 111. The CPU 100 can read and write the contents of the CR 108. A CRT controller (CRTC) 110, which will be described later, displays a cursor at a position on the CRT 111, which is a display device, corresponding to the address stored in the CR.

  Reference numeral 109 denotes a display buffer memory (DBUF), which stores a pattern of data to be displayed. Reference numeral 110 denotes a CRT controller (CRTC), which plays a role of displaying the contents stored in the CR 108 and the DBUF 109 on the CRT 111. The CRT 111 is a display device using a cathode ray tube or the like, and the dot pattern display pattern and cursor display on the CRT 111 are controlled by the CRTC 110. Note that another display device such as an LCD can be used instead of the CRT 111.

  A character generator (CG) 112 stores character and symbol patterns to be displayed on the CRT 111.

  Reference numeral 105 denotes an NCU, which is a communication device for communicating with an external device such as another computer device. By using this device, the program and data of the first embodiment can be shared with other devices. become.

  In FIG. 1, through a network (LAN), a personal computer (PC (personal computer)) 114, an image receiving / accumulating / displaying device (TV / VR) 115 such as a television broadcast or a video taken by itself, a home Is connected to a game computer (GC) 116 for free use, and information can be freely exchanged therewith. Needless to say, the device of the present invention and the device connected via the network may be anything. The network type may be anything, and the network may be connected to an external network instead of the closed network as shown in the figure.

  Reference numeral 106 denotes a receiving device (DTU) that realizes a broadcast communication receiving function using an artificial satellite or the like. The parabolic antenna (ANT) 113 receives a signal such as a radio wave broadcast via the artificial satellite. And has a function of acquiring broadcast data. There are various types of broadcast-type communications, such as those broadcast on terrestrial radio waves, those broadcast on coaxial cables and optical cables, and those distributed on LANs and large-scale networks. Any form can be considered as long as it is for broadcast communication.

  Next, the functional configuration of the multimedia data search system according to the first embodiment will be described with reference to FIG.

  FIG. 2 is a diagram showing a functional configuration of the multimedia data search / display system according to the first embodiment of the present invention.

  2 are realized by the CPU 100 in FIG. 1 executing a control program stored in the ROM 101 to control each component.

  Reference numeral 201 denotes a search query input processing unit for inputting requirements (search query (search condition)) relating to a desired search target. A search query holding unit 202 temporarily stores the search query input by the search query input processing unit 201. The search query holding unit 202 is realized by the RAM 102, for example.

  A search processing unit 203 refers to the search query held by the search query holding unit 202 and searches for multimedia data that matches the search query. A search result holding unit 204 temporarily holds a search result searched by the search processing unit 203. The search result holding unit 204 is realized by the RAM 102, for example.

  Reference numeral 205 denotes a metadata holding unit, which stores metadata attached to multimedia data stored in the multimedia data holding unit 206. A multimedia data holding unit 206 stores multimedia data to be searched.

  An analysis rule selection processing unit 207 refers to the search query held by the search query holding unit 202 and selects an analysis rule corresponding to the search query from the analysis rules stored in the analysis rule holding unit 208. select.

  An analysis rule holding unit 208 stores analysis rules for analyzing multimedia data obtained as a search result. Reference numeral 209 denotes a decision analysis rule holding unit that temporarily holds information for identifying the analysis rule selected by the analysis rule selection processing unit 207. The decision analysis rule holding unit 209 is realized by the RAM 102, for example.

  A search result analysis processing unit 210 analyzes the search result held in the search result holding unit 204 according to the analysis rule held in the decision analysis rule holding unit 209.

  An analysis result holding unit 211 temporarily stores the analysis result analyzed by the search result analysis processing unit 210. The analysis result holding unit 211 is realized by the RAM 102, for example.

  A display content determination processing unit 212 uses the display rule held by the determination display rule holding unit 217 to determine display content to be displayed as a search result according to the analysis result held in the analysis result holding unit 211. .

  A display content holding unit 213 temporarily stores the display content determined by the display content determination processing unit 212. The display content holding unit 213 is realized by the RAM 102, for example.

  Reference numeral 214 denotes a search result display unit that displays the search result with reference to the display content held by the display content holding unit 213. Reference numeral 215 denotes a display rule holding unit, which stores display rules for determining display contents to be displayed as search results.

  Reference numeral 216 denotes a display rule selection processing unit, which refers to the search query held by the search query holding unit 202 and the search purpose held by the decision purpose holding unit 220 and displays the search query and the display rule corresponding to the search purpose. The display rule stored in the display rule holding unit 215 is selected.

  A decision display rule holding unit 217 temporarily holds information for identifying the display rule selected by the display rule selection processing unit 216. The determination display rule holding unit 217 is realized by the RAM 102, for example.

  A target candidate holding unit 218 holds a target candidate (search target candidate) serving as a display condition for displaying a search result. A search purpose setting processing unit 219 refers to the search query held by the search query holding unit 202, and sets a search purpose corresponding to the input search query to the target candidate held by the goal candidate holding unit 218. Choose from.

  A determination purpose holding unit 220 temporarily stores the search purpose set by the search purpose setting processing unit 219. The determination purpose holding unit 220 is realized by the RAM 102, for example.

  In the first embodiment, for example, when multimedia data such as a still image or a moving image shot at home is searched and a shot of the still image or moving image obtained as a search result is reproduced, the display is effectively performed. Is assumed.

<Search process>
In the first embodiment, a search is performed using metadata associated with each data to be searched. The metadata is given by manually assigning data such as titles and keywords, or by analyzing the multimedia data to be searched and extracting feature quantities such as media feature quantities.

  In search, users are given search conditions (search queries) such as search keywords and keyword lists, and the search conditions are compared with the metadata associated with each data, and metadata matching the search conditions is given. Get multimedia data. Further, the similarity between the search condition and the metadata of each acquired multimedia data may be calculated and acquired based on the calculation result.

  Here, the similarity is a value indicating how appropriate a specific character string in the search condition input by the user is as an expression indicating a relationship with each data.

  Next, an example of the search condition setting screen according to the first embodiment will be described with reference to FIG.

  FIG. 3 is a diagram showing an example of a search condition setting screen according to the first embodiment of the present invention.

  This search condition setting screen is illustrated as a screen when a window system is used. The search condition setting screen in FIG. 3 is a screen example when the window system is used in a dedicated application. However, when displaying on a small screen such as a digital camera or a video camera, the display according to the screen size is displayed. It is desirable to do the way.

  Reference numeral 31 denotes a title bar, which is a part for displaying the title of the search condition setting screen and operating the entire screen such as movement and size change. 32 and 33 are buttons for providing functions related to the search condition setting screen, 32 is a help display button, and 33 is a button for closing the search condition setting screen.

  Reference numerals 34 and 37 denote input areas for inputting search conditions for performing a search.

  The input area 34 is an input area for displaying the type and theme of the event that is the highest concept provided in the first embodiment and prompting the user to make a selection. The user can select a plurality of types of displayed candidates. A desired one can be selected by a check box.

  In the first embodiment, events such as “wedding”, “athletic meet”, etc. at the time of taking an image at home are displayed as candidates, and FIG. 3 shows a state where “wedding” is selected. Yes. By selecting the event / theme, a specific search condition for actually searching for an image such as a keyword is generated. In the first embodiment, image contents such as moving images and still images shot mainly in the home are handled, and therefore the contents can be assumed to some extent. Therefore, an event is selected from the list. Alternatively, the user may directly input a search condition such as a search keyword.

  The input area 37 is a text input area for inputting a keyword that the user wants to specify in addition to the search condition generated by selecting an item in the input area 34, and a plurality of keywords are separated by a delimiter (for example, In addition, it is possible to specify the logical operations AND and OR by radio buttons.

  In the first embodiment, the input to the input area 37 is optional and may not be input.

  Although not shown in the first embodiment, importance or similarity distance of a plurality of inputted keywords or phrases may be set, and information such as date range is specified as another search condition. You may do it.

  Reference numeral 35 denotes an OK button. When the OK button 35 is operated, the contents input to the input areas 34 and 37 as the search condition are finally determined, and the process proceeds to the next process. On the other hand, 36 is a cancel button. When the cancel button 36 is operated, the input contents in the input areas 34 and 37 are discarded.

  Next, using the definition of the relationship of events shown in FIG. 5, the sub-event corresponding to the lower concept corresponding to the higher concept set in the input area 34 of FIG. 3 is set as a search keyword. Then, using the metadata assigned to each information shown in FIG. 4, a plurality of still images and moving image shots obtained by photographing the wedding are searched.

  For example, as sub-events constituting the event “wedding”, a plurality of sub-events such as “greeting”, “entrance of bride and groom”, and “cake cut” are used as search keywords. Further, when a keyword is input in the input area 37 of FIG. 3, this is also used for the search.

  Next, an example of metadata (held by the metadata holding unit 205) added to image data (multimedia data) such as still images and moving image shots to be searched will be described with reference to FIG. .

  FIG. 4 is a diagram showing an example of metadata according to the first embodiment of the present invention.

  FIG. 4 shows that a title, a photographer, a shooting date, a data type (moving image / still image), a category, a keyword, an importance level, and the like are assigned to each image data as metadata. Here, an example in which character information that can be handled as simple keywords is mainly stored is shown, but information other than character information such as an image feature amount can be given as metadata.

  In FIG. 4, as an example, in a certain image A, the title is “cake sword”, the photographer is “Information Taro”, the shooting date is “2003-04-01T12: 30”, the data type is “Movie”, and the category is "Wedding-wedding-cake sword", "Bride and groom", "Wedding cake" as keywords, and "High" as importance are assigned. These metadata are stored in a storage medium such as the DISK 103, the ROM 101, and the RAM 102, for example.

  By using this metadata, it is possible to search image data such as still images and moving image shots stored in a database on the storage medium.

<Display search results>
Next, a method for effectively displaying the search result will be described in detail.

  In the first embodiment, image data such as a still image or a moving image that is a search result is analyzed, and effective display contents suitable for the user's purpose are determined according to the analysis result.

1. Analysis of search results Data can be considered to be diverse in terms of data management, but the type and contents of data handled in a certain situation are predictable to some extent or limited to a certain range. In addition, depending on the type of content of each data, it is possible to predict to some extent empirically the specific content, things, people, and situations that appear in each data. The relationship etc. can be estimated to some extent.

  Therefore, in the first embodiment, data analysis rules corresponding to the data handling situation, data range, content type, etc. are defined in advance and stored in a storage medium such as the DISK 103, ROM 101, RAM 102, etc. The data obtained as a search result is analyzed. The analysis rule may be defined for each search engine or for each database in which data to be searched is stored.

  The data analysis method differs depending on the content and type of the data to be searched, and the situation and range of the search, but in the first embodiment, it is expressed in the data obtained as the search result. Classify and relate according to meaning and content.

  First, when classification is performed according to the content and type of data, for example, classification can be performed by paying attention to common factors (factors to which the data belongs) for events, categories, scenes, central event objects, and the like. Events, categories, scenes, etc. can be expected to be composed of a plurality of events, categories, scenes, etc. Therefore, a hierarchical structure of events, categories, scenes, etc. is defined as an analysis rule, and data is classified according to these hierarchical structures.

  Next, an example of the analysis rule having this hierarchical structure will be described with reference to FIG.

  FIG. 5 is a diagram showing an example of an analysis rule having a hierarchical structure according to the first embodiment of the present invention.

  FIG. 5 shows a hierarchical structure of classification elements such as events, categories, and scenes. FIG. 5 shows the inclusive relation / hierarchical relation of individual sub-classification elements XA to XI included in a certain classification element X. The classification element X first includes classification elements XA to C, and further indicates that the classification element XA includes classification elements XD to F. A keyword may be associated with the end element of the hierarchical structure and used for data classification. Information in this hierarchical structure is stored in a storage medium such as the DISK 103, the ROM 101, and the RAM 102, for example.

  For example, in the first embodiment, since image data such as videos and still images taken mainly at home is a search target, events such as “wedding”, “athletic meet”, etc. when shooting images at home A hierarchical structure can be defined in units of and themes.

  If the classification element X in FIG. 5 is an event called wedding, it can be classified into categories such as the introduction of the bride and groom, the process up to the wedding, the scenery of the venue, the wedding, and the reception. The wedding category in these categories consists of entrance, vows, exchange of rings, etc., and the reception category consists of entrance, cake entry, speech, etc., and further speech. This category can be composed of scenes such as guest speech, friend representative speech, and the like.

  Further, if the classification element X is a theme of child growth, it can be classified into categories such as birth, Shichigosan, kindergarten era, elementary school era, and the like. For example, the category of the kindergarten era can be constituted by small events such as an entrance ceremony, an athletic meet, and a school performance.

  In the first embodiment, each piece of data obtained as a search result is classified into categories according to the above-described hierarchical structure using metadata attached to image data as shown in FIG. For example, in FIG. 4, category information is given to the data in advance according to the hierarchical structure of FIG. 5, and can be classified into categories using this.

  Also, as shown in FIG. 5, a keyword is associated with the end element of the hierarchical structure, and by comparing this with metadata such as a keyword and a title assigned to each data, the search result can be determined. It can also be classified into categories by estimating the content.

Further, a category or the like related to the search query may be estimated from the search query, and classification may be performed only for the category belonging to the lower layer.

Next, the individual data obtained as a search result and the group of data as the classification result are related.

  In the first embodiment, for example, spatial positional relationship, temporal and semantic context, similarity relationship, equality / parallel relationship, contrast (opposite) relationship, causal relationship such as cause and result, overview and details, materialization / example・ Association rules for finding mutual relationships such as supplement, modification, etc. are held as analysis rules.

  Here, an example of the association rule will be described with reference to FIG.

  FIG. 6 is a diagram showing an example of an association rule based on the meaning / content of image data according to the first embodiment of the present invention.

  FIG. 6 shows association rules for search results obtained from search queries belonging to the category “wedding”.

  For example, a “category exterior” and “inside the venue”, which are subcategories of the category “venue scenery” in the event “wedding”, have a spatial positional relationship of perspective, outside, inside, and part. In addition, it can be said that there is a causal relationship between the cause and the result of the category “Bride and groom's sorrow” and the category “wedding”. In addition, the category “state of waiting room” and “wedding” correspond to a modification relationship or a supplementary relationship.

  In the first embodiment, the association is performed according to the association rule of FIG. 6 using metadata such as titles and keywords assigned to the image data as shown in FIG. Further, the relationship between before and after may be performed according to the shooting date and time of the image data.

  In the first embodiment, data obtained as a search result is further classified, and individual data and a group of data as a classification result are related.

  Here, an example of group association will be described with reference to FIG.

  FIG. 7 is a diagram illustrating an example of classifying data obtained as a search result according to the first exemplary embodiment of the present invention, and associating individual data and a group of data that is the classification result.

  In FIG. 7, G represents a group of data, D represents data, and R represents a relationship. A line indicated by R represents a start point, and ● represents an end point.

  For example, the classification element G1 and the classification element G2 have a relationship of R13, and the classification element G1 and the classification element G3 have a relationship of R11. The classification elements G11 and G12 included in the classification element G1 have a relationship of R11.

  Further, the data D111 and D112 included in the classification element G11 have a relationship of R13 with D111 as the start point and D112 as the end point. D224 is an element included in the classification element G2, but is not related to any element.

  These pieces of information are temporarily stored in a storage medium such as the RAM 102.

  Next, details of the search result analysis processing by the search result analysis processing unit 210 will be described with reference to FIG.

  FIG. 8 is a flowchart showing search result analysis processing by the search result analysis processing unit according to the first embodiment of the present invention.

  First, in step S901, data obtained as a search result is classified according to an analysis rule. This classification method is as described above.

  Next, in step S902, the data obtained as the search result and the group of data obtained as the classification result are related. The relation method is as described above.

  As described above, in the first embodiment, as the analysis process of the search result analysis processing unit 210, the search result classification and the correlation between the search result and the classification result are performed, but the data to be searched is handled. Various other analysis methods are conceivable depending on the situation, the range of data, the type of content, characteristics, and the like.

2. Setting User Purpose In the first embodiment, the user specifies a search purpose, and necessary information is displayed in an easy-to-understand manner according to the specified search purpose.

  In particular, in the first embodiment, a candidate for displaying a search result is created in advance according to the type of data content to be searched. At the time of a search, a search target candidate corresponding to a specified search query is presented to the user as a display condition for selection.

  For example, in the first embodiment, candidates for viewing purposes (search purposes) for daily events and themes such as weddings, travel, and child growth are created.

  Here, an example of the viewing purpose will be described with reference to FIG.

  FIG. 9 is a diagram showing an example of the viewing purpose of the image data according to the first embodiment of the present invention.

  For example, the viewing purpose in the event “wedding” can be specified by the viewer, so “bridal couple”, “relative”, etc. are selected as the purpose. Such information is stored in a storage medium such as the DISK 103, the ROM 101, and the RAM 102, for example.

  Since these purposes differ depending on the data to be searched, it is necessary to create a template suitable for each.

  Next, an example of a display condition setting screen for setting a viewing purpose as a display condition will be described with reference to FIG.

  FIG. 10 is a diagram showing an example of a display condition setting screen according to the first embodiment of the present invention.

  The display condition setting screen in FIG. 10 is displayed after the OK button 35 in FIG. 3 is pressed. This is illustrated on the screen when a window system is used. In addition, the display condition setting screen in FIG. 10 is a screen example when the window system is used in a dedicated application. However, when displaying on a small screen such as a digital camera or a video camera, a display according to the screen size is displayed. It is desirable to do the way.

  Reference numeral 261 denotes a title bar, which is a portion for performing the title display of the display condition setting screen and the operation of the entire screen such as movement and size change. Reference numerals 262 and 263 are buttons for providing functions related to the search condition setting screen, 262 is a help display button, and 263 is a button for closing the search condition setting screen.

  Next, the input area 264 is an input area for displaying the display conditions for displaying the search results and prompting the user to make a selection. The input area 264 corresponds to the search query by referring to the contents (viewing purpose candidates) in FIG. The contents (viewing purpose candidates) are displayed. The user can select a desired one from a plurality of displayed candidates. In the first embodiment, “bride and groom”, “relative”, and the like, which are viewing objectives of the viewer corresponding to the event “wedding” that is the search query, are displayed, and “bride and groom” and “friend” are selected. It shows the state.

  Reference numeral 265 denotes an OK button. When this OK button 265 is operated, the content input to the input area 264 as a display condition is finally confirmed, and the process proceeds to the next process. Reference numeral 266 denotes a cancel button. When the cancel button 266 is operated, the input contents in the input area 264 are discarded.

  In the first embodiment, candidates for search purposes are created in units of events, categories, etc., but can be used for other data by selecting a reference unit according to the type and content of various data. It is possible to make it easy to do.

3. Determination of Display Contents Next, a method for determining the display contents of search results suitable for the user's search purpose based on the obtained analysis results and displaying the search results in an easy-to-understand manner for the user will be described in detail.

  Humans connect and understand fragmentary information in their heads by integrating and linking various linguistic and non-verbal symbols. When complicated mediation of imagination is required, it is difficult to understand, and information presentation without any temporal, spatial, or semantic continuity will be confusing. Therefore, it is important to present information in such a way that humans can interpret it naturally.

  In the first embodiment, a display rule corresponding to a search target is defined in advance by applying a human understanding pattern. By determining the display contents based on the above analysis results according to the display rules, the search results can be shown in an easily understandable manner.

  In particular, in the first embodiment, as the display contents, the configuration and components of the search result, and the expansion method (presentation order of the components), the connection method of the components, and the like are determined as display rules. This display rule may be defined for each search engine or may be defined for each type of database in which data to be searched is stored.

  Further, information that the user particularly needs from the search results differs depending on the search purpose for the user to search. Therefore, in the first embodiment, contents that the user thinks are particularly necessary in the search results are predicted from the designated search purpose, and data related to the contents is selected and displayed. That is, in the first embodiment, display rules are defined in accordance with the type of data content to be searched and the search purpose of the user. At the time of search, a display rule appropriate for the search query and search purpose specified by the user is selected, and the display content is determined.

  When the search result is shown, it is easy for the user to understand if it is divided into blocks of data representing one content corresponding to a chapter or paragraph in a sentence and a sequence or scene in a video. The elements of each block include the central element (high importance element) that is the center of the block, supplementary elements that are connected to the central element in terms of content, elements that connect blocks, central elements / supplemental elements There are connecting elements for connecting the two, explanation elements for helping understanding of the contents, and the like. In the first embodiment, a group of data obtained by classifying search results is handled as a block of data.

  Specifically, the block of data in the first embodiment is a set group of a plurality of pieces of image information representing scenes and sequences determined by the above classification. Each piece of data represents a still image or a shot of a moving image.

  When presenting search results, it is possible to create an easy-to-understand story and explanation flow by combining individual data and blocks of data and gradually building an understanding framework (starting up, breaking down, etc.) step by step. . Also, the appropriate story structure and its components differ depending on the user's search purpose. Therefore, in the first embodiment, a story suitable for the search purpose of the user is generated based on the data analysis result.

  Next, the contents (display rules) of the elements for each search purpose for configuring a framework (for example, a start-up roll) according to the user's search purpose will be described with reference to FIG.

  FIG. 11 is a diagram showing the contents of elements for each search purpose for constituting a framework in accordance with the search purpose of the user according to the first embodiment of the present invention.

  In FIG. 11, first, “introduction of the bride and groom” and the like are components of “start” for the viewer “friend” of the event “wedding”. In addition, there are “venue scenery”, “state before ceremony” and the like as components of “approval” for the viewer “friend”. In addition, there are “wedding”, “banquet” and the like as components of “turn” for the viewer “friend”. In addition, as a component of “conclusion” for the viewer “friend”, there are “state after ceremony”, “thank you / greeting” and the like.

  Each component of the start-up and rolling can be further configured by a framework such as a start-up and rolling. Although not shown in the drawing, for example, “entrance” or the like is a constituent element of “start” for the viewer “friend” of the event of “banquet”. In addition, as a component of “approval” for “friend”, there is “cake sword”. In addition, “changing color”, “side entertainment”, and the like are components of “turning” for the viewer “friend”. In addition, as a constituent element of “Yui” for the viewer “Friend”, “Bouquet gift” and the like can be cited.

  One or more constituent elements are selected from these, and the story structure is determined by a framework such as an arrangement. A data block belonging to a framework such as a start-and-run connection and a method of expanding individual data belonging to the block will be described next.

  In addition, if necessary, an element that explicitly separates a framework such as a start-and-run process (break and start) and an element that connects these elements may be included. In the case of the first embodiment, for example, a subtitle screen corresponds to this.

  Moreover, you may show a search result, without producing | generating a story. When a story is not generated, all the searched data is set as a display target, and the above-described data block and an individual data development method belonging to the block are determined.

  Next, determination of a data block and an individual data expansion method (presentation order) will be described.

  When presenting search results, expand each component in an order that humans can naturally understand, for example, from the overview or from the whole to the details, according to the spatial and temporal arrangement, in the order in which the explanation is understood. Then it becomes easy to understand. This can be derived from the relationship of each element obtained by analysis. Therefore, in the first embodiment, the presentation order based on the mutual relationship between the components is defined as a display rule, which is stored in a storage medium such as the DISK 103, the ROM 101, and the RAM 102. By using this display rule, the presentation order of the components is determined.

  Here, an example of the presentation order will be described with reference to FIG.

  FIG. 12 is a diagram illustrating an example of the presentation order according to the first embodiment of the present invention.

  FIG. 12 defines the order of presentation and the priorities based on the data blocks and the interrelationships between the individual data. The priority indicates the strength of dependency and cohesion between the components when the components are presented according to the mutual relationship.

  When the priority is strong, it is difficult to understand if the components are presented separately, so that the data must be continuously arranged in the presentation order.

  If the priority is medium, the components do not necessarily have to be continuous if the order of presentation is observed, but if they are not continuous, elements that can understand the relationship must be supplemented. Represents that.

  When the priority is weak, it is indicated that the constituent elements do not need to be continued if the presentation order is observed.

  In FIG. 12, for example, when there is a space or a semantic whole and part relationship between data A and data B, data B (partial) is also included in the order of data A (whole) to data B (partial). ) To data A (whole) in order, both of which are strong.

In addition, when there is a causal relationship between data A and data B, it can be understood naturally by presenting data A representing the cause to data B representing the result. Conversely, data B (result) can be presented in the order of data A (cause), but in this case, it is difficult to understand without any explanation or presentation that aids understanding. Moreover, when it is not necessary to order according to the mutual relationship of data like parallel or an equal relationship, you may arrange in random order, alphabetical order, etc.

For example, in the first embodiment, “the situation near the venue” and “the situation at the venue” have a relationship between perspective and outside, so it is preferable to display “the situation near the venue” to “the situation at the venue”. I understand.

  Next, based on the structure of the search result, its constituent elements, and the expansion method of each constituent element, connection is made so that the data block delimiters and relationships with other blocks become clear. That is, if it is clearly indicated that the data blocks differ depending on the display and the flow cannot be understood naturally, a connecting element for explaining the mutual relationship is necessary to help the user understand.

  In the first embodiment, when an image is reproduced, a sequence and a scene are connected by a cut connection, a transition effect, a screen that captures between white and black, and the like is clearly indicated. If the relationship between the sequence and the scene is not clear, the mutual relationship is shown by using a material image, a transition effect, a sound effect, and the like. In addition, when it is difficult to understand the development and contents, it is used as an explanatory element such as a telop or a supermarket so that it can be easily understood.

  Individual data is also linked so that the relationship with the central element and the relationship with other data can be understood. In this case, if the flow cannot be understood naturally, a connecting element is necessary.

  In the first embodiment, for example, shots of still images and moving images are usually connected by cuts, and when a connection element is required, a transition effect, a material image, or the like can be used.

  FIG. 13 is a diagram defining a connection ID indicating a connection method with respect to the mutual relationship and presentation order of the components according to the first embodiment of the present invention. The medium having the medium priority in FIG. 12 differs in connection method depending on whether or not the elements to be connected are continuous.

  FIG. 14 is a diagram showing display contents corresponding to each connection ID according to the first embodiment of the present invention as display rules.

  In FIG. 14, the display contents differ depending on whether the element to be connected is a shot, a scene, or a sequence. The display content corresponding to the link ID in FIG. 14 differs depending on the type and content of the data to be searched, and it is necessary to define the corresponding content.

  From FIG. 13 and FIG. 14, for example, if shots and scenes with context are displayed continuously from front to back, they are connected by cutting, and if sequences are displayed continuously from front to back, a fade-out or pause screen is inserted. You can see that it is better to connect. In addition, when displaying shots or scenes having contexts from the back to the front, it is necessary to clearly indicate the relationship by inserting a crossfade, a material image, BGM, or the like. Furthermore, in the case of a sequence, it can be seen that explanatory elements such as telop and supermarket are necessary.

  Specifically, when a search is performed for a “wedding” event, for example, if the relationship is equal, parallel, or similar, a fade-out display is displayed. In the case of a scene or sequence, a black screen is displayed for the next scene. Display differently by displaying before the sequence. Also, if there is a context or order of understanding, they are displayed connected by cutting.

  Next, display content determination processing by the display content determination processing unit 212 will be described with reference to FIG.

  FIG. 15 is a flowchart showing display content determination processing by the display content determination processing unit according to the first embodiment of the present invention.

  First, in step S1401, a display rule for easily indicating the search result is selected. The display rule is necessary in steps S1402 to S1404. In the first embodiment, the display rule corresponding to the type of data to be searched and the search query is selected.

  Next, in step S1402, based on the above-described analysis result, the story structure and its constituent elements (data group or data) when the search result is shown are determined.

  Details of the processing in step S1402 will be described with reference to FIG.

  FIG. 16 is a flowchart showing details of the process in step S1402 according to the first embodiment of the present invention.

  First, in step S1501, it is determined whether or not an easy-to-understand story (flow of explanation) structure for displaying the search result is generated. If the story structure is not generated (NO in step S1501), the process ends. On the other hand, if a story structure is generated (YES in step S1501), the process advances to step S1502.

  In step S1502, the story structure for displaying the search result is determined according to the analysis result. As described above, when a search result is shown, an easy-to-understand story can be created by combining data and data blocks and constructing a framework for understanding such as starting and rolling. In order to determine a story, one or a plurality of constituent elements suitable for the user's purpose are selected from the constituent element candidates that make up the start-up roll as shown in FIG. Inquiry rolls can be further composed in stages, such as roll-in rolls, and one or more components are selected from each candidate and assembled in stages to determine the story structure of the entire search. .

  Next, in step S1503, in accordance with the determined story configuration, data blocks and data that are constituent elements of the framework such as start-up conversion are determined from the data obtained as search results, and the processing is terminated.

  Returning to the description of FIG.

  In step S1403, the expansion method (presentation order) of each component determined in step S1402 is determined. As described above, when each component is presented, it is easier to understand if it is shown in the order that humans understand. Accordingly, the presentation order of the individual components is determined according to the analysis result using the display rule that defines the presentation order based on the mutual relationship of the components in FIG.

  In step S1404, a method for linking the components is determined. As described above, in order to show the search result in an easy-to-understand manner, it is necessary to link the blocks so that the data block delimiters and relationships with other blocks are clear. Also, it is necessary to link individual data so that the relationship with the central element and the relationship with other data can be understood. Therefore, using the display rules as shown in FIG. 13 and FIG. 14, the connection method of the constituent elements is determined based on the above-described interrelationship of the constituent elements.

  Next, an example of the search result display screen of Embodiment 1 will be described with reference to FIG.

  FIG. 17 is a diagram showing an example of a search result display screen according to the first embodiment of the present invention.

  FIG. 17 is an example of a search result display screen in which search results are displayed on the CRT according to the display contents described above. This is a screen example when the window system is used in a dedicated application. However, when displaying on a small screen such as a video camera, it is desirable to perform a display method according to the screen size.

  Reference numeral 161 denotes a title bar, which is a part for performing title display on the search result display screen and operation of the entire screen such as movement and size change. Reference numerals 162 and 163 denote buttons for providing functions related to the search result display screen, 162 denotes a help display button, and 163 denotes a button for closing the search result display screen.

  Reference numeral 164 denotes a display area for displaying a search condition input by the user, the number of search results, and the like. In the first embodiment, the keywords used as search conditions and the number of data obtained as search results are displayed.

  Reference numerals 165 and 166 denote display areas for displaying search results. In the display area 165, the image data is reproduced according to the display contents determined based on the data analysis result. The display area 166 displays text information such as a title, description, and comments.

  Reference numeral 167 denotes a playback button for instructing the start of playback of image data. Reference numeral 168 denotes a stop button for instructing to stop reproduction of the image data. Reference numeral 169 denotes an end button for ending the display of the search result display screen.

  In the first embodiment, an analysis rule and a display rule are defined in units of events, which is an example suitable for analyzing and displaying the content of home video content and the like. However, according to the present invention, it is possible to cope with content other than video easily by selecting a unit serving as a reference according to the content.

  Next, an overview of the entire process executed in the first embodiment will be described with reference to FIG.

  FIG. 18 is a flowchart showing an outline of the entire processing executed in the first embodiment of the present invention.

  First, in step S2201, search result analysis processing is executed. The details are as shown in FIG.

  Next, in step S2202, a user purpose setting process for setting a user search purpose is executed. This is a display condition setting screen as shown in FIG. 10, where candidates for search purposes are presented to the user and prompted for input.

  Next, in step S2203, display content determination processing is executed based on the analysis result in step S2201. The details are as shown in FIG.

  Next, in step S2204, search result display processing is executed according to the display content determined in step S2203.

  As described above, according to the first embodiment, when displaying the search result, the data obtained as the search result is analyzed, and the display content is dynamically changed based on the analysis result and the search purpose related to the search. To display. This makes it possible to display data that seems to be necessary for the user in an easy-to-understand manner according to the relevance of the data type and content, and is necessary even when searching a large amount of information such as collecting information. Information can be easily searched, selected, organized and used.

  In the first embodiment, on the basis of the result of analyzing the data, in particular, the display contents are determined by determining the configuration and constituent elements of the search result, the expansion method (presentation order of the constituent elements), and the connecting method of the constituent elements. However, it can also be determined according to the type and content of data to be searched, the analysis method, and the like.

[Embodiment 2]
In the second embodiment, a case where the retrieved document data is effectively displayed according to the user's purpose will be described.

<Search process>
As in the first embodiment, the search is performed using the metadata assigned to the search target. In the second embodiment, the metadata can be provided with character information extracted by analyzing document data, in addition to extracting and assigning manpower and media feature quantities. In addition, information such as the creation date and time and the creator can be given by extracting information automatically given by the OS.

  First, an example of the search condition setting screen according to the second embodiment will be described with reference to FIG.

  FIG. 19 is a diagram showing an example of a search condition setting screen according to the second embodiment of the present invention.

  This search condition setting screen is an example of a screen when a window system is used, and it is assumed that document data on a computer network is searched and the result is displayed. This search condition setting screen is a screen example when a window system is used in a dedicated application, but a similar function may be provided by a Web browser or the like.

  Reference numeral 171 denotes a title bar, which is a part for displaying the title of the search condition setting screen and for operating the entire screen such as movement and size change. Reference numerals 172 and 173 are buttons for providing functions related to the search condition setting screen, 172 is a help display button, and 173 is a button for closing the search condition setting screen.

  Reference numeral 174 denotes an input area for inputting a search condition for performing a search. The input area 174 is a text input area for inputting a keyword. A plurality of keywords are separated by a delimiter (such as a space) and input, and either a logical operation AND or OR can be designated by a radio button. It has become. In addition to keywords, phrases and sentences can be entered.

  As other search conditions, the number of search results to be displayed on one screen and the language in which information to be searched is described are specified.

  Although not shown in the second embodiment, importance or similarity distances of a plurality of inputted keywords and phrases may be set, and as other search conditions, a date range or a search target is set. Information such as a position may be designated.

  Reference numeral 175 denotes an OK button. When this OK button 175 is operated, the content set in the input area 174 as a search condition is finally confirmed, and the process proceeds to the next process. On the other hand, 176 is a cancel button. When the cancel button 176 is operated, the input contents in the input area 174 are discarded.

  In the search condition setting screen of FIG. 19, the user arbitrarily inputs a keyword, phrase, sentence, etc., and the search condition is specified, but the user specifies the genre, category, etc. to which the information belongs. You can also search from within. For example, the category type and genre provided in the second embodiment may be displayed in a list or tree, and the user may select a desired one.

  When the keyword “notebook PC” is input in the input area 174 of FIG. 19, related information is searched using metadata assigned to each data to be searched.

  Here, an example of metadata will be described with reference to FIG.

  FIG. 20 is a diagram illustrating an example of metadata according to the second embodiment of the present invention.

  FIG. 20 shows that a title, a creator, a creation date, a category, a keyword, and the like are assigned to each data as metadata. Here, an example is shown in which character information that can be handled as simple keywords is mainly stored, but information other than character information such as feature amounts according to each data characteristic can also be added as metadata. is there.

  In FIG. 20, as an example, in a certain document data A, “AAA note price information” as a title, “Information Taro” as a creator, “2003-04-01T08: 30” as a creation date, and “Economy-Enterprise— Computer-product-price "," notebook PC "as a keyword, and product name" AAA notebook "are assigned. These metadata are preserve | saved at storage media, such as DISK103, ROM101, RAM102, for example.

  In the second embodiment, character information is given as a search query, and a search is performed according to the character information. Images, sounds, etc. are designated as a search query, and the media feature amount extracted from the search query is compared with the media feature amount of each data. You can also search by doing this. Further, the search may be performed by associating the media feature amount with the character information given as the search query.

  In the second embodiment, the search is performed using the metadata assigned to each data. However, the search can be performed by directly analyzing the document data.

  As described above, data such as character information stored in the database on the storage medium can be searched.

<Display search results>
Next, a method for effectively displaying the search result will be described in detail.

  In the second embodiment, document data as a search result is analyzed, and effective display contents are determined according to the analysis result.

  The basic processing procedure is the same as that of the first embodiment, and the type of data to be searched, the analysis rule for analyzing the search result, and the display rule for determining the display contents are different from those of the first embodiment.

1. Analysis of Search Results Analysis of search results is performed by analyzing data obtained as search results by using data analysis rules stored in a storage medium such as DISK 103, ROM 101, and RAM 102, as in the first embodiment. To realize. In the second embodiment, a category related to a search query is estimated from a search query specified at the time of search, and an analysis rule corresponding to the category is selected.

  Next, as in the first embodiment, multimedia data obtained as a search result is classified according to an analysis rule having a hierarchical structure of categories as shown in FIG. For example, document data obtained via a computer network such as the Internet can generally be classified into categories such as economy, politics, art, religion, etc., and the category of economy is corporate, finance, real estate, shopping, investment, etc. Further, the category of companies can be composed of smaller categories such as industry types such as electrical machinery, automobiles, and construction, product information, service information, and the like.

  In the second embodiment, individual data obtained as a search result is classified into categories according to the above-described hierarchical structure using metadata attached to document data as shown in FIG. For example, in FIG. 20, category information is given beforehand as metadata according to the hierarchical structure of FIG. 5, and can be classified into categories using this.

  In addition, a keyword is assigned to each classification element of the hierarchical structure in FIG. 5, and what kind of search result is obtained by comparing this with metadata such as a keyword and a title assigned to each document data. The contents can be estimated and classified into categories.

  Further, a category or the like related to the search query may be estimated from the search query, and classification may be performed only for the category belonging to the lower layer.

  In the second embodiment, the data may be classified according to common factors such as the data type of data obtained as a search result, the data management structure (directory, folder, URI / URL, etc.), and the like.

  When classifying by data type, for example, it is classified by media type such as text, moving image, still image or the like, or even in a document, whether only text or character information and image are mixed. This may be preliminarily assigned with a media type as one of metadata, but can also be estimated from a file name (extension). In the case of an HTML document or an XML document, by analyzing a tag or text described in the document, it can be estimated whether the document is composed only of text or an image is embedded in the document. .

  In the case of a still image, it may be analyzed for whether it is a natural image or chart data and used for classification. This can also be applied in advance as metadata, or can be estimated by image feature quantities such as edges.

  When classifying based on the data management structure or the positional relationship in which the data is stored, for example, the classification is based on a hierarchical structure of locations based on directories, folders, URI / URLs, network addresses, and the like.

  In addition, it is possible to classify by analyzing the media feature amount of each content such as an image or sound, and the similarity or clustering technology may be used for the classification.

  In the second embodiment, as in the first embodiment, an association rule for associating individual data obtained as a search result and a group of data as a classification result is held as an analysis rule.

  Here, an example of the association rule will be described with reference to FIG.

  FIG. 21 is a diagram showing an example of association rules based on the meaning and contents of data according to the second embodiment of the present invention.

  FIG. 21 shows an association rule for a search result obtained from a search query belonging to the category “computer-product”.

  For example, the category “product information” and its subcategory “product specification” have a relationship of outline and details, and whole and part. In addition, the category “accessory” is complementary to the category “product specification”.

  In the second embodiment, as in the first embodiment, the association is performed according to the association rule of FIG. 21 using the metadata attached to the data as shown in FIG. Further, the relationship between before and after may be performed according to the data creation date and update date.

  In the second embodiment, the analysis is performed by using the metadata assigned to each data. However, the document data is directly analyzed, and the media feature amount of each content such as an image or sound is also analyzed. By doing so, the correlation can be found.

  Further, as in the first embodiment, the data obtained as a search result is classified, and individual data and a group of data as a classification result are related (FIG. 7).

  In the second embodiment, the search result analysis processing by the search result analysis processing unit 210 is the same as that of the first embodiment except that the analysis rule to be used is different.

2. Setting of User Purpose Also in the second embodiment, as in the first embodiment, a candidate for displaying a search result is created in advance according to the type of data content to be searched. In the second embodiment, a search target candidate for a category is presented to the user as a display condition for selection.

  Here, an example of the search purpose will be described with reference to FIG.

  FIG. 22 is a diagram showing an example of a document data search purpose according to the second embodiment of the present invention.

  For example, the search purpose in the category “computer-product” is to select “new purchase”, “market research”, and the like. These pieces of information are stored in a storage medium such as the DISK 103, the ROM 101, and the RAM 102.

  Next, an example of a display condition setting screen for setting a search purpose as a display condition will be described with reference to FIG.

  FIG. 23 is a diagram showing an example of a display condition setting screen according to the second embodiment of the present invention.

  The display condition setting screen in FIG. 23 is displayed after the OK button 175 in FIG. 19 is pressed. This is illustrated on the screen when a window system is used. In addition, the display condition setting screen in FIG. 23 is a screen example when the window system is used in a dedicated application, but a similar function may be provided by a Web browser or the like.

  Reference numeral 281 denotes a title bar, which is a portion for performing title display on the display condition setting screen and operations for the entire screen such as movement and size change. Reference numerals 282 and 283 are buttons for providing functions related to the search condition setting screen, 282 is a help display button, and 283 is a button for closing the search condition setting screen. For example, it is a part that performs overall operations such as movement and size change.

  Next, the input area 284 is an input area for displaying the display conditions for displaying the search result and prompting the user to select, and corresponds to the search query by referring to the contents (search target candidates) in FIG. Display the contents (search target candidates). The user can select a desired one from a plurality of displayed candidates. In the second embodiment, “new purchase”, “market research”, etc., which are search purposes corresponding to a search query belonging to the category “computer-product”, are displayed, and “new purchase” “market research” is selected. Indicates the state.

  Reference numeral 285 denotes an OK button. When this OK button 285 is operated, the content input to the input area 284 as a display condition is finally confirmed, and the process proceeds to the next process. Reference numeral 286 denotes a cancel button. When the cancel button 286 is operated, the input contents in the input area 284 are discarded.

3. Determination of Display Contents Also in the second embodiment, the display contents of search results suitable for the user's search purpose are determined based on the obtained analysis results. Here, in the second embodiment, the flow of the display content determination processing by the display content determination processing unit 212 is as shown in FIG. 15 as in the first embodiment, but each display rule for determining the display content is different. Different.

  Therefore, with reference to FIG. 15, the display content determination processing by the display content determination processing unit 212 in the second embodiment will be described.

  First, in step S1401, when a display rule is selected, a category related to the search query is estimated from the search query specified at the time of search, and a display rule corresponding to the category is selected.

  Next, in step S1402, the elements to be displayed and the configuration are determined from the search results. In the second embodiment, the display elements indicate the document data obtained as the search results and are obtained at the time of the search. Those having a high degree of similarity are determined to be particularly important elements, and this is set as a central element. Further, the data block in the second embodiment is a group of a plurality of document data representing categories and genres determined by classification.

  Specifically, based on the analysis result of the data, a framework such as a start-up roll suitable for the user's search purpose is generated.

  FIG. 24 is a diagram showing the contents of elements for each search purpose for constituting a framework in accordance with the search purpose of the user according to the second embodiment of the present invention.

  FIG. 24 shows the contents of the elements constituting the incoming and outgoing link suitable for the user's purpose for the search result obtained from the search query belonging to the category “computer-product”. In FIG. 24, there are “market trend”, “latest model spec” and the like as components of “start” for the search purpose “new purchase”. In addition, “product information” or the like is a component of “acceptance” for the search purpose “new purchase”. In addition, “maker comparison”, “product evaluation”, and the like are components of “transfer” for the search purpose “new purchase”. Further, “shop information” or the like is cited as a component of “conclusion” for the search purpose “new purchase”. Although not shown in the figure, each constituent element of the start-and-roll connection can be further configured by a framework such as a start-and-end connection.

  In addition, if necessary, elements (separated from the beginning) and elements that connect these may be inserted. In the case of the second embodiment, for example, a blank line, an item name, or the like corresponds to this.

  Moreover, you may show a search result, without producing | generating a story.

  Next, in step S1403, the display element determined in step S1401 and the method of expanding the data block including the display element (presentation order) are determined using FIG. 13 as in the first embodiment.

  For example, in the second embodiment, the “accessory” category has a supplementary relationship with the “product spec” category.

  Next, in step S1404, based on the configuration of the search result, its components, and the relationship and expansion method of each component, the relationship between each component, the relationship with the central element, and the relationship with other components are determined. Decide how to consolidate to be clear. Individual data is also linked so that the relationship with the central element and the relationship with other data can be understood.

  In the second embodiment, for example, a line break, a space, a symbol, or the like is clearly indicated so that a delimiter between components can be clearly understood. In addition, a block of data may be indicated by surrounding it with a frame. If the relationship is not clear, use a design such as indentation, symbols, or colors to show these relationships. In particular, if the flow cannot be understood naturally, information that indicates the relationship with the previous component is required. Further, the frame and characters may be displayed in different colors.

  FIG. 25 is a diagram showing display contents corresponding to each connection ID according to the second embodiment of the present invention as display rules.

  In FIG. 25, the display contents differ depending on whether the elements to be connected are individual data, paragraphs (paragraphs), or chapters. From FIG. 13 and FIG. 25, for example, in the case where data and paragraphs having a context are displayed continuously from front to back, they are connected by displaying “→” symbols, numbers, etc., and chapters are continuously displayed from front to back. It can be seen that it is better to connect by line feed and number display.

  In addition, when data or paragraphs having the opposite relationship are displayed continuously, it is necessary to clearly indicate the relationship by displaying a “⇔” symbol or the like. Furthermore, in the case of chapters, it can be seen that explanation elements such as subtitles are required.

  In addition, the equivalent / parallel / similar relationship indicates that the information name is displayed in a separate line by displaying it in a separate line, and a uniform symbol such as “●” is displayed in front of the information name. . Also, if there is a context or order of understanding, a number may be given before the information name. When there is an inclusive / hierarchical relationship, it indicates that the relationship is expressed by a line break and indentation.

  Also, if the number of search results is large or the amount of data displayed on the screen is large and it is difficult to see on one screen, the search results are composed of multiple screens, and the screen is changed by hyperlinks, buttons, etc. Or a paragraph break or interrelationship with other components. The screen transition method can be determined based on the block configuration and development, the interrelationship between each component, the number of elements that can be displayed on one screen, and the like.

  In addition, an appropriate screen transition method varies depending on a device for displaying a search result such as a PDA, a type of data to be searched, and the like. For example, when the hierarchical relationship is clearly indicated by screen transition, the screen transition is determined according to the hierarchical structure, and the elements of the lower hierarchy are indicated by screen switching. In addition, when the data has a clear order such as a context and it is difficult to grasp the contents unless it is viewed in an appropriate order, the following information may be limited and shown according to the above-described development.

  Next, an example of the search result display screen according to the second embodiment will be described with reference to FIG.

  FIG. 26 is a diagram showing an example of a search result display screen according to the second embodiment of the present invention.

  FIG. 26 is an example of a search result display screen in which search results are displayed on the CRT according to the display contents described above. This is an example of a screen when a window system is used in a dedicated application. However, when displaying on a small screen such as a mobile phone or a PDA, it is desirable to perform a display method according to the screen size.

  Reference numeral 231 denotes a title bar, which is a portion for performing title display on the search result display screen and operations for the entire screen such as movement and size change. Reference numerals 232 and 233 are buttons for providing functions related to the search result display screen, 232 is a help display button, and 233 is a button for closing the search result display screen.

  Reference numeral 234 denotes a display area for displaying the search conditions input by the user, the number of search results, and the like. In the second embodiment, a keyword used as a search condition and the number of data obtained as a search result are displayed.

  Reference numeral 235 denotes a display area for displaying search results. The display area 235 displays display contents determined based on the data analysis result. In the second embodiment, first, “AAA”, “BBB”, and “CCC”, which are character strings representing the contents of data chunks, are displayed. Since there is an order in this hierarchy, a number indicating the order is displayed.

  Next, “AAA-a1”, “AAA-a2”, and “AAA-a3”, which are character strings representing the contents of data chunks, are displayed for the layer below “AAA”. Since these are in an equal or parallel relationship, “●” is added to the head.

  Next, as the search results belonging to “AAA-a1”, the document names “AAA-a11” to “AAA-a15” are the search results belonging to “AAA-a2”, and the document names “AAA-a21”, “AAA”. -A22 "is displayed. In the same way, the document names as search results are displayed for the other layers.

  Reference numeral 236 denotes a button for terminating the display of the search result display screen.

  In the example of FIG. 26, the search result is displayed on one screen. However, as described above, the search result may be composed of a plurality of screens, and the screen may be changed. Further, an HTML document may be generated according to the display contents described above and displayed on a Web browser.

  Next, an example of a search result display screen displayed using a Web browser will be described with reference to FIG.

  FIG. 27 is a diagram showing an example of a search result display screen according to the second embodiment of the present invention.

  FIG. 27 is an example of a search result display screen in which search results are displayed on the CRT using a Web browser. In FIG. 27, instead of displaying the search results on one screen, the search results are shown as a plurality of screens by linking them.

  Reference numeral 241 denotes a title bar, which is a portion for performing title display on the search result display screen and operation of the entire screen such as movement and size change. Reference numerals 242 and 243 are buttons for providing functions related to the search result display screen, 242 is a help display button, and 243 is a button for closing the search result display screen.

  Reference numeral 244 denotes a display area for displaying the search conditions input by the user, the number of search results, and the like. In the second embodiment, keywords used as search conditions and the number of data obtained as search results are displayed.

  Reference numeral 245 denotes a display area for displaying search results. The display area 245 displays the display content determined based on the data analysis result. In the second embodiment, first, “AAA”, “BBB”, and “CCC”, which are character strings representing the contents of data chunks, are displayed. Since this hierarchy has an order, a number indicating the order is displayed. Yes.

  Next, “AAA-a1”, “AAA-a2”, and “AAA-a3”, which are character strings representing the contents of data chunks, are displayed for the layer below “AAA”. “AAA-a1” to “AAA-a3” are underlined (anchors), indicating that other HTML documents are linked. When the user clicks on the character string, the screen jumps to the search result display screen belonging to each.

  As described above, according to the second embodiment, the same effect as that of the first embodiment can be obtained for document data having character information.

[Embodiment 3]
In the above embodiment, metadata such as keywords is assigned to multimedia data, and this is used for search and analysis of search results. However, what is used for search and analysis of search results? A method according to the type / characteristic of each multimedia data and the purpose of the user can be used.

  In the above embodiment, the DISK 103, the ROM 101 and the RAM 102 in FIG. 1, or the storage medium such as a memory card, the metadata shown in FIGS. 4 and 20, and the analysis rules and diagrams shown in FIGS. Although the display rules shown in FIGS. 11 to 14 and 24 are stored, these pieces of information can be acquired from an external device on the network, stored in a storage medium, and used.

  Although the embodiments have been described in detail above, the present invention can take an embodiment as, for example, a system, an apparatus, a method, a program, or a storage medium, and specifically includes a plurality of devices. The present invention may be applied to a system that is configured, or may be applied to an apparatus that includes a single device.

  In the present invention, a software program (in the embodiment, a program corresponding to the flowchart shown in the figure) that realizes the functions of the above-described embodiment is directly or remotely supplied to the system or apparatus, and the computer of the system or apparatus Is also achieved by reading and executing the supplied program code.

  Accordingly, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention.

  In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, or the like.

  As a recording medium for supplying the program, for example, floppy (registered trademark) disk, hard disk, optical disk, magneto-optical disk, MO, CD-ROM, CD-R, CD-RW, magnetic tape, nonvolatile memory card ROM, DVD (DVD-ROM, DVD-R) and the like.

  As another program supply method, a client computer browser is used to connect to an Internet homepage, and the computer program of the present invention itself or a compressed file including an automatic installation function is downloaded from the homepage to a recording medium such as a hard disk. Can also be supplied. It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the present invention.

  In addition, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to users, and key information for decryption is downloaded from a homepage via the Internet to users who have cleared predetermined conditions. It is also possible to execute the encrypted program by using the key information and install the program on a computer.

  In addition to the functions of the above-described embodiments being realized by the computer executing the read program, the OS running on the computer based on an instruction of the program is a part of the actual processing. Alternatively, the functions of the above-described embodiment can be realized by performing all of them and performing the processing.

  Furthermore, after the program read from the recording medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion board or The CPU or the like provided in the function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

It is a block diagram which shows the basic composition of the computer apparatus with which the multimedia data search system of Embodiment 1 of this invention is constructed | assembled. It is a figure which shows the function structure of the multimedia data search display system of Embodiment 1 of this invention. It is a figure which shows an example of the search condition setting screen of Embodiment 1 of this invention. It is a figure which shows an example of the metadata of Embodiment 1 of this invention. It is a figure which shows an example of the analysis rule which consists of hierarchical structure of Embodiment 1 of this invention. It is a figure which shows an example of the association rule by the meaning and content of the image data of Embodiment 1 of this invention. It is a figure which shows an example which classify | categorizes the data obtained as a search result of Embodiment 1 of this invention, and performs correlation of the group of the data which is individual data and a classification result. It is a flowchart which shows the search result analysis process by the search result analysis process part of Embodiment 1 of this invention. It is a figure which shows an example of the viewing-and-listening purpose of the image data of Embodiment 1 of this invention. It is a figure which shows an example of the display condition setting screen of Embodiment 1 of this invention. It is a figure which shows the content of the element for every search objective for comprising a framework along the search objective of the user of Embodiment 1 of this invention. It is a figure which shows an example of the presentation order of Embodiment 1 of this invention. It is the figure which defined connection ID which shows the connection method with respect to the mutual relationship of the component of Embodiment 1 of this invention, and a presentation order. It is the figure which showed the display content corresponding to each connection ID of Embodiment 1 of this invention as a display rule. It is a flowchart which shows the display content determination process by the display content determination process part of Embodiment 1 of this invention. It is a flowchart which shows the detail of a process of step S1402 of Embodiment 1 of this invention. It is a figure which shows an example of the search result display screen of Embodiment 1 of this invention. It is a flowchart which shows the outline | summary of the whole process performed in Embodiment 1 of this invention. It is a figure which shows an example of the search condition setting screen of Embodiment 2 of this invention. It is a figure which shows an example of the metadata of Embodiment 2 of this invention. It is a figure which shows an example of the association rule by the meaning and content of the data of Embodiment 2 of this invention. It is a figure which shows an example of the search objective of the document data of Embodiment 2 of this invention. It is a figure which shows an example of the display condition setting screen of Embodiment 2 of this invention. It is a figure which shows the content of the element for every search objective for comprising a framework along the search objective of the user of Embodiment 2 of this invention. It is the figure which showed the display content corresponding to each connection ID of Embodiment 2 of this invention as a display rule. It is a figure which shows an example of the search result display screen of Embodiment 2 of this invention. It is a figure which shows an example of the search result display screen of Embodiment 2 of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 201 Search query input process part 202 Search holding part 203 Search processing part 204 Search result holding part 205 Metadata holding part 206 Multimedia data holding part 207 Analysis rule selection processing part 208 Analysis rule holding part 209 Decision analysis rule holding part 210 Search result Analysis processing unit 211 Analysis result holding unit 212 Display content determination processing unit 213 Display content holding unit 214 Search result holding unit 215 Display rule holding unit 216 Display rule selection processing unit 217 Decision display rule holding unit 218 Purpose candidate holding unit 219 Search purpose setting unit Processing unit 220 Determination purpose holding unit

Claims (11)

An information processing apparatus for searching multimedia data composed of at least one of document data, image data, or audio data, or a combination thereof,
Storage means for storing the multimedia data to be searched;
Input means for inputting search conditions for searching the multimedia data;
Search means for searching multimedia data from the storage means based on the search condition;
Search purpose candidate holding means for holding search purpose candidates by the search means;
Setting means for setting a desired search purpose from search purpose candidates held by the search purpose candidate holding means;
Analysis rule holding means for holding analysis rules for analyzing multimedia data obtained as a search result by the search means;
An analysis unit that selects an analysis rule corresponding to the search condition from the analysis rule holding unit, and analyzes the multimedia data obtained as a search result by the search unit according to the selected analysis rule;
Display rule holding means for holding display rules related to the search purpose candidates in order to determine the contents to be displayed as the search results;
A display rule corresponding to the search purpose set by the setting unit is selected from the display rule holding unit, and the display is displayed as the search result based on the selected display rule and the analysis result of the analysis unit. Display content determination means for determining the content;
An information processing apparatus comprising: a search result display unit that displays the search result according to the display content determined by the display content determination unit. The analyzing means classifies the multimedia data obtained as a search result by the search means according to the meaning and content thereof;
A multimedia data obtained as a search result by the search means, and a correlation means for performing a correlation for a group of multimedia data classified by the classification means;
2. The information processing according to claim 1, wherein the display content determination unit determines display content to be displayed as the search result based on a classification result by the classification unit and a correlation result by the correlation unit. apparatus. The classification means classifies the multimedia data obtained as a search result by the search means according to a hierarchical structure,
The association means relates the multimedia data obtained as a search result by the search means and the group of multimedia data hierarchically classified according to the hierarchical structure by the classification means. The information processing apparatus according to claim 2. The analysis means performs analysis when storing the multimedia data as a search target in the storage means, and associates the analysis result with the multimedia data as metadata,
The information processing apparatus according to claim 2, wherein the display content determination unit determines display content to be displayed as the search result based on metadata associated with the multimedia data. The setting means obtains a search purpose candidate related to the search condition from the search purpose candidate holding means;
The information processing apparatus according to claim 1, further comprising: a selection unit that selects a desired search purpose from the search purpose candidates acquired by the acquisition unit. The display rule includes a definition for determining a presentation order according to a relationship between the multimedia data,
The display content determination means is a multimedia obtained as a search result of the search means based on the classification result of the classification means, the correlation result of the correlation means, and the search purpose set by the setting means. The composition of multimedia data to be displayed is determined from among the data, and the presentation order of the multimedia data to be displayed is determined based on the correlation result of the correlation means. The information processing apparatus described. The display rule includes a definition of a priority with respect to a relationship between the multimedia data and a presentation order,
The display content determination means includes a group of multimedia data classified by the classification means, a multimedia data obtained as a search result of the plurality of search means, and a group of multimedia data classified by the classification means; The information processing apparatus according to claim 6, wherein when there is a relationship, the presentation order of the multimedia data to be displayed is determined based on the priority. The multimedia data is still image data or moving image data,
The search result display unit continuously reproduces and displays still image data or moving image data as the search result according to the display content determined by the display content determination unit. Information processing device. The classifying unit classifies the still image or the moving image data, which is multimedia data obtained as a search result by the searching unit, according to a hierarchical structure in which a common factor to which the still image or the moving image data belongs is a unit of the hierarchy. And
9. The association means for associating multimedia data obtained as a search result by the search means with a group of multimedia data hierarchically classified according to the hierarchical structure. The information processing apparatus described in 1. Control an information processing apparatus that retrieves multimedia data from a storage unit that stores multimedia data composed of at least one of document data, image data, audio data, or a combination thereof. A control method,
An input step for inputting a search condition for searching the multimedia data;
A search step of searching multimedia data from the storage unit based on the search condition;
A search purpose candidate holding step of holding a search purpose candidate in the search step in a storage medium;
A setting step for setting a desired search purpose from the search purpose candidates held in the storage medium in the search purpose candidate holding step;
An analysis rule holding step for holding an analysis rule for analyzing multimedia data obtained as a search result by the search step in a storage medium;
An analysis step of selecting an analysis rule corresponding to the search condition from the storage medium held in the analysis rule holding step, and analyzing multimedia data obtained as a search result in the search step according to the selected analysis rule When,
In order to determine the content to be displayed as the search result, a display rule holding step for holding a display rule related to the search purpose candidate;
The display rule corresponding to the search purpose set in the setting step is selected from the storage medium held in the display rule holding step, and based on the selected display rule and the analysis result of the analysis step, the A display content determination step for determining display content to be displayed as a search result;
A search result display step of displaying the search result according to the display content determined in the display content determination step. Control an information processing apparatus that retrieves multimedia data from a storage unit that stores multimedia data composed of at least one of document data, image data, audio data, or a combination thereof. A program for realizing control,
A program code of an input process for inputting a search condition for searching the multimedia data;
Based on the search condition, a program code of a search step for searching multimedia data from the storage unit,
A program code of a search purpose candidate holding step for holding a search purpose candidate in the search step in a storage medium;
A program code of a setting step for setting a desired search purpose from the search purpose candidates held in the storage medium in the search purpose candidate holding step;
A program code of an analysis rule holding step for holding an analysis rule for analyzing multimedia data obtained as a search result by the search step in a storage medium;
An analysis step of selecting an analysis rule corresponding to the search condition from the storage medium held in the analysis rule holding step, and analyzing multimedia data obtained as a search result in the search step according to the selected analysis rule Program code,
In order to determine the content to be displayed as the search result, a program code of a display rule holding step for holding a display rule related to the search purpose candidate;
The display rule corresponding to the search purpose set in the setting step is selected from the storage medium held in the display rule holding step, and based on the selected display rule and the analysis result of the analysis step, the A program code of a display content determination step for determining display content to be displayed as a search result;
And a program code of a search result display step for displaying the search result according to the display content determined in the display content determination step.

Images