JP2007241474A - Information processing apparatus and method, program, and storage medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To efficiently and accurately retrieve and use the subject of retrieval from within multimedia documents. <P>SOLUTION: An information processing method, which is capable of processing electronic documents including data with different attributes, includes analyzing in attribute units the layout structure of data contained in electronic documents (S502); and comparing the layout structure of data of a first attribute selected with the layout structure of data of a second attribute for which the data of the first attribute is referred to (S506). Based on the comparison result, if the layout structure of the data of the first attribute is different from the layout structure of the data of the second attribute, additional data is created and, according to the layout structure of one of the data, the layout structure of the other is changed (S507). The additional data created is registered as data for retrieval for associating the data of the first attribute with the data of the second attribute (S508). <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a technique for retrieving and using an electronic document (hereinafter also referred to as “multimedia document”) including moving images, still images, audio data, text, and the like as constituent elements.

  In recent years, with the development of digital data processing / communication technology, systems capable of handling multimedia documents in which moving images, still images, audio, text, and the like are mixed have been provided. According to such a system, it is possible to create and record (accumulate) a multimedia document as content, and to search and use the accumulated multimedia document.

Focusing on each component of the multimedia document (moving image, still image, audio data, text, etc.), the technology for realizing the search is different for each of moving image, still image, audio, text, and the like. Therefore, when searching for necessary information from moving images, still images, audio data, texts, etc., it is necessary to apply individual search techniques. When searching for moving images, still images, audio, and text as specific components, there is a search technology that combines each component's own information with other information in the multimedia document to characterize each component. . According to this search technique, for example, in a still image search, a still image is characterized by combining a title in the document and a description of the still image as other information in the multimedia document. In this search technique, there is a problem of how to acquire text such as a title and an explanation in a multimedia document for the still image as information characterizing the still image. For example, in Patent Document 1, text information related to image information in a multimedia document is extracted, information on the scale of text to be searched is extracted, and used at the time of search.
JP 2000-306103 A

  However, in the above-mentioned Patent Document 1, the relationship between the image to be detected and the text is specified, but information processing that matches the layout (arrangement structure) of the image and the text is not performed. Therefore, when searching and using in multimedia documents, the relationship between components such as texts and corresponding charts cannot be accurately grasped, and the search target is searched and used efficiently and accurately. There is a problem that it becomes difficult to do.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to provide a technique that enables a search target to be efficiently and accurately searched from a multimedia document and used.

In order to achieve the above object, an information processing apparatus according to the present invention provides:
An information processing apparatus capable of processing an electronic document including data with different attributes,
Layout analysis means for analyzing the arrangement structure of data included in the electronic document in attribute units;
A comparing means for comparing the arrangement structure of the selected first attribute data with the arrangement structure of the second attribute data referring to the first attribute data;
If the data structure of the first attribute is different from the data structure of the second attribute based on the comparison result of the comparing means, additional data is generated, and the other is matched to one of the arrangement structures. Changing means for changing the arrangement structure of
Registration means for registering the additional data generated by the changing means as search data that associates the data of the first attribute and the data of the second attribute.

  According to the present invention, it becomes possible to efficiently search and use a search target from a multimedia document with high accuracy.

  Embodiments of the present invention will be described with reference to the accompanying drawings. FIG. 1 is a diagram schematically illustrating an internal configuration of an information processing apparatus according to an embodiment of the present invention.

  In the figure, a CPU 101 executes various controls in the information processing apparatus. The ROM 102 stores a boot program executed when the information processing apparatus is started up and various data. The RAM 103 stores a control program to be processed by the CPU 101 and provides a work area when the CPU 101 executes various controls.

  The input unit 104 includes a keyboard, a mouse, a stylus pen, a dial, and the like, and provides various input operation environments for the user.

  The external storage device 105 includes a recording medium such as a hard disk, a floppy (registered trademark) disk, an optical disk, a magnetic disk, a magneto-optical disk, a magnetic tape, and a nonvolatile memory card, and a drive that drives the storage medium and records information. Composed. The external storage device 105 is assumed to have a storage capacity that can record and use multimedia documents and information used when searching and reusing them. When the information processing apparatus provides a search service as a search server, the search additional information is stored in the external storage device 105 as search metadata.

  The display unit 106 is configured by a display or the like, and displays various input operation states to the user. The communication unit 107 is a component for communicating with other devices, and has a wired communication function such as RS232C, USB, IEEE1394, P1284, SCSI, modem, Ethernet (registered trademark). The communication unit 107 has various communication functions such as Bluetooth, infrared communication, and IEEE802.11b. Reference numeral 108 denotes a LAN, and the communication unit 107 can communicate the information processing apparatus with other devices via the LAN 108. A connector 109 is a connector for connecting the information processing apparatus to another device via the communication unit 107. It is possible to connect a storage device or another device to the end of the connector 109 to use the storage device from the information processing apparatus.

  Reference numeral 115 denotes an imaging unit that can capture a moving image or a still image. Each component in the information processing apparatus is connected by an internal bus 110 and can transmit and receive data to and from each other. The information processing apparatus can also store the multimedia document transmitted from another apparatus via the LAN 108 and the communication unit 107 in the external storage device 105. The information processing apparatus can generate a multimedia document that combines a still image captured by the imaging unit 115 and a document input by the input unit 104. The information processing apparatus can store the generated multimedia document in the external storage device 105.

  FIG. 2 is a diagram exemplifying a configuration example for one page of the multimedia document 201 to be processed in the present embodiment. Reference numeral 202 denotes a document portion of the multimedia document 201, and 203 denotes a document heading. The heading 203 includes a document part 204, a graph 205, a caption (description) 206, a graph 207, a caption (description) 208, a graph 209, a caption 210, and a page number 211.

  FIG. 3 is a diagram exemplifying a data structure of a basic part describing a result of the layout analysis of the multimedia document 201 performed by the CPU 101 of the information processing apparatus. (3-1) to (3-32) shown on the right in the figure correspond to the respective row numbers of the layout analysis result. The display of the layout analysis result is not limited to the XML format shown in FIG. 3, but can be expressed by other structured documents such as the SGML format.

  The (3-1) line indicates the start of a page and indicates that the page number of the multimedia document 201 is 11. The (3-32) line indicates the end of the corresponding page.

  From (3-2) to (3-4) lines, information for specifying the size of the corresponding page is described. The upper left corner of the page is the origin (O), the y-axis is the forward direction in the lower direction of the page, and the x-axis is the forward direction in the right direction (FIG. 2).

  Here, it is described in the (3-3) line that the size is 600 in the x direction and 900 in the y direction. Hereinafter, information such as position and size is described in this coordinate system. Note that the page size is not limited to vertical and horizontal numerical values, but may be specified by a standard such as A4 or B5, or may be specified by a page layout setting value such as m rows and xn columns.

  Lines (3-5) to (3-31) list the constituent elements in the page, and express each constituent element in a unit of a block. Here, lines (3-6) to (3-21) represent blocks of the text area of the document part 202, the headline 203, and the document part 204 of the multimedia document 201 shown in FIG.

  The (3-6) line is a text format in which the number for identifying the block given by the layout analysis (hereinafter referred to as “block identification number”) is “1101”, and the attribute of the data is a text. It shows that it is a block.

  The (3-7) to (3-9) lines indicate position information in the page of the block specified by the block identification number 1101. In the (3-8) line, the position information is described as (50, 40), and this position information corresponds to the point G1 in the block of the document area including the document part 202, the heading 203, and the document part 204 ( (See FIG. 2).

  Lines (3-10) to (3-12) show information on the size (size) of blocks in the document area. In FIG. 3, the size is (x, y) = (400, 500). With reference to the point G1 of the block in the document area, the position and size of the block in the document area in the page can be specified based on the size information.

  Lines (3-13) to (3-20) show the contents described in the document area block. When the document is scanned from a paper medium or the like, it is realized by converting the image into characters and text information by using a technique such as character recognition. Alternatively, text information or the like can be acquired based on electronic data transmitted from another device via the LAN 108.

  The (3-15) line contains contents corresponding to the headline portion 203, and XML information indicating that the headline number is “5” and the headline title is “analysis of customer segment”. is described. In line (3-16), the contents of the text data are described.

  The (3-17) to (3-19) lines describe XML information indicating itemized items. In the (3-18) line, the content corresponding to the document part of the item of the bullet item is described, and XML information (tag <UL>) indicating that the expression “ski customer” is an underlined expression. And </ UL>).

  The (3-22) line indicates that the block identification number given by the layout analysis is “1102”, and that the data attribute is a graph-type block into which a graphic is inserted.

  The (3-23) to (3-25) lines indicate the position information within the page of the block specified by the block identification number 1102. In the (3-24) line, the position information is described as (60, 560), and this position information corresponds to the point G2 in the block of the graphic area including the pie chart 205 and the pie chart 207 (see FIG. 2). reference).

  Lines (3-26) to (3-28) show information on the size (size) of the block in the graph format. In FIG. 3, the size is (x, y) = (100, 80). Based on the size information, the position and size of the graph block in the page can be specified based on the point G2 of the graph block.

  In lines (3-29) to (3-31), a graph 205 (Graph 1), a graph 207 (Graph 2), and a graph 210 (Graph 3) are described. Note that the present invention is not limited to various types of graphs, and a table format display, a photograph, drawing data, and the like can be used as components of the multimedia document.

  As a result of the layout analysis, components (document portion 202, graph 205, etc.) included in multimedia document 201 are extracted. For each block, a block identification number, attribute information, block position information, size information, a document, a graph, or the like is extracted for each block.

  The CPU 101 performs layout analysis on the multimedia document stored in the external storage device 105. Then, a feature amount is extracted for each component (for each block) from the layout analysis result, and this is registered as metadata in the external storage device 105 as search information that can be processed. Is possible. The feature amount extraction based on the layout analysis enables the search without depending on various formats in which various multimedia documents are recorded. Furthermore, since only the metadata search needs to be processed at the time of search, the processing time required for the search process can be shortened. This process facilitates retrieval and use without depending on various formats in which various multimedia documents are recorded.

  FIG. 4 is a diagram schematically showing the result (FIG. 3) of the layout analysis performed on the multimedia document shown in FIG.

  Reference numeral 401 denotes a part corresponding to the page number, and reference numeral 402 denotes text data corresponding to the sentence part 202 in FIG. Reference numeral 403 corresponds to the headline 203 in FIG. 2, and reference numeral 404 denotes text data corresponding to the document portion on the next line of the headline 203.

  Reference numeral 405 denotes a bulleted portion in FIG. 2, which includes bulleted items (406, 409). Under the bullet item 406, text data relating to the content (407, 408) of the bullet item is arranged. In the contents of itemized items (407, 408), text data and figure numbers (3-a), (3-b) referenced therein are described.

  Reference numeral 410 denotes a figure area. In the figure area 410, chart data corresponding to the figure number (3-a), the figure number (3-b), and the figure number (3-c) are arranged (412, 413, 414). In the following description, chart data is used as an example, but a moving image or a still image may be used.

  When the data arrangement structure (406 to 409) of the bulleted portion 405 is compared with the data arrangement structure (412 to 414) of the figure area 410, the data corresponding to the item 406 is not in the figure area 410. They are different.

  If the itemized portion is a description that gives meaning in order or a description that gives meaning to the confrontation, the same relationship is also established for each referenced chart data (412, 413, 414). Often to do. For this reason, by combining the arrangement structure of the document part and the chart part, it is possible to more accurately grasp the relationship between the constituent elements in the multimedia document.

  FIG. 5 is a diagram for explaining a flow of processing for matching the arrangement structure of the document part and the chart part in the multimedia document in the present embodiment. This process is executed under the control of the CPU 101 in the information processing apparatus.

  In step S501, the process is started. In step S502, layout analysis is performed on the multimedia document in the external storage device 106 or the like.

  In step S503, it is searched whether or not there is a caption for the chart selected from the graphic or tabular data (chart data) obtained from the layout analysis result. Here, as an example, it is assumed that the graphs 205, 207, and 209 in FIG. 2 are selected. If it is determined that there is no caption, the process ends. If it is determined that there is a caption, the process proceeds to step S504. Here, 206, 208, and 210 in FIG. 2 are searched as captions.

  In step S504, data including the same text (expression) as the captions 206, 208, and 210 is searched for in a nearby text data block.

  In step S505, if data including the same text (expression) as the captions 206, 208, and 210 is not retrieved, the process ends. On the other hand, if it is determined in step S505 that data including the same text (expression) as the captions (descriptions) 206, 208, and 210 is searched, the process proceeds to step S506.

  In step S506, it is determined whether the arrangement structure of the caption (description) is the same as the arrangement structure of the data of the document portion including the same text (expression) as the caption (description). Specifically, in FIG. 4, the layout structure of the chart data (412 to 414) of the caption (3-a, b, c) and the document part (itemized part) including the caption (3-a, b, c) ) (407 to 409). If these arrangement structures are the same, the process proceeds to step S509 and the process ends. On the other hand, if the arrangement structure is different, the process proceeds to step S507, and the arrangement structure is changed according to the more complicated structure. This is for preventing a part of data having a complicated arrangement structure from being deleted when the simple structure is adopted. In the case of FIG. 4, the arrangement structure of the document part (itemized part) (407 to 409) is more complicated than the chart data (412 to 414) in that there is no data corresponding to the itemized item 406 in the figure area 410. It is determined that there is. Based on the determination result, the arrangement structure of the chart data (412 to 414) of the caption (3-a, b, c) is changed in accordance with the arrangement structure of the document part (itemized part) (407 to 409).

  In changing the arrangement structure, additional data as shown by 615 in FIG. 6 is generated. This additional data functions as search data for associating the chart data (612, 613) and the document part (itemized part) (407, 408). The additional data is stored (registered) as search metadata in the external storage device 105 or the like, and is referred to when searching or using the chart data (612, 613) or the document part (itemized part) (407, 408).

  In step S508, the layout analysis result obtained by changing the arrangement structure of the chart data (412 to 414) of the caption (3-a, b, c) is stored in the external storage device 105 and the process is terminated.

  FIG. 6 shows corresponding chart data (412 to 412) according to the arrangement structure of the document parts (itemized parts) (407 to 409) including the captions “3-a”, “3-b”, and “3-c”. It is a figure which shows the result of having changed the arrangement structure of 414). The same parts as those in FIG. 4 showing the result of layout analysis before changing the arrangement structure are given the same reference numerals.

  The arrangement structure of the chart data 612 and 613 is changed according to the arrangement structure of the document parts (itemized parts) (407 and 408) including the captions “3-a” and “3-b”. As data for matching the arrangement structure, a “diagram area” 615 is generated, and the arrangement structure is changed so that the chart data 612 and 613 are included under the figure area 615.

  As a result, the description (for example, the relationship such as comparison / conflict / enumeration) of the bulleted portion (405 to 409) can be reflected in the diagram area (410, 615, 612, 613, 414), and the correspondence between them It becomes possible to clarify. By combining the arrangement structure of the document part and the figure area, it is possible to more accurately grasp the relationship between the constituent elements in the multimedia document.

  The figure area 615 corresponding to the itemized item 406 can be used as search metadata. When searching the corresponding chart data from the itemized section 407, the corresponding chart data 612 is searched by searching the metadata (the figure area 615) corresponding to the item 406 without parsing the explanation text. Can be used.

  Similarly, when searching for the corresponding bulleted part 407 from the chart data 612, the bulleted part corresponding to the chart is searched by searching the bulleted item 406 corresponding to the figure area 615 regardless of the syntax analysis. Can be used.

  In the present embodiment, the number of data that can be used as search metadata (the figure region 615) is one, but a plurality of data may be used. When a plurality of search metadata are generated, it is possible to set a quantitative importance for each search metadata. For example, when a plurality of search metadata has a hierarchical structure, the depth of the hierarchy may be evaluated, and quantitative importance may be set for each search metadata. As the evaluation of the importance of the search metadata, for example, the importance of the first hierarchy is 100, the importance of the second hierarchy is 80, the importance of the third hierarchy is 60, depending on the depth of the hierarchy. It is also possible to quantify the importance. It is also possible to determine the priority of search using this importance as an index.

  As described above, according to the present embodiment, it is possible to efficiently search and use a search target from a multimedia document with high accuracy.

(Other embodiments)
Needless to say, the object of the present invention can also be achieved by supplying a storage medium storing software program codes for realizing the functions of the above-described embodiments to a system or apparatus. Needless to say, this can also be achieved by the computer (or CPU or MPU) of the system or apparatus reading and executing the program code stored in the storage medium.

  In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention.

  As a storage medium for supplying the program code, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a nonvolatile memory card, a ROM, or the like can be used.

  Further, the functions of the above-described embodiment are realized by executing the program code read by the computer. In addition, an OS (operating system) running on a computer performs part or all of actual processing based on an instruction of a program code, and the above-described embodiment is realized by the processing. Needless to say.

It is a figure which illustrates roughly the internal structure of the information processing apparatus concerning embodiment of this invention. It is the figure which illustrated the example of a structure for one page of the multimedia document used as the process target in embodiment of this invention. It is a figure which shows the data structure which describes the result of having performed the layout analysis of the multimedia document concerning embodiment of this invention. FIG. 3 is a diagram schematically showing a result of layout analysis performed on the multimedia document shown in FIG. 2. It is a figure explaining the flow of a process which matches the arrangement structure of the document part and figure part in a multimedia document in embodiment of this invention. It is a figure which shows the result of having changed the arrangement structure of corresponding chart data according to the arrangement structure of the document part (itemized part) containing a caption.

Claims (10)

An information processing apparatus capable of processing an electronic document including data with different attributes,
Layout analysis means for analyzing the arrangement structure of data included in the electronic document in attribute units;
Comparing means for comparing the arrangement structure of the selected first attribute data with the arrangement structure of the second attribute data referring to the first attribute data;
If the data structure of the first attribute is different from the data structure of the second attribute based on the comparison result of the comparing means, additional data is generated, and the other is matched to one of the arrangement structures. Changing means for changing the arrangement structure of
Registration means for registering the additional data generated by the changing means as search data relating the data of the first attribute and the data of the second attribute;
An information processing apparatus comprising:   The changing means changes the data attribute structure of the first attribute or the data attribute structure of the second attribute in accordance with a more complicated arrangement structure based on the comparison result of the comparing means. The information processing apparatus according to claim 1.   2. The information according to claim 1, wherein, when a plurality of additional data is generated by the changing unit, the registration unit sets importance for each of the plurality of additional data and registers the data as search data. Processing equipment.   The information processing apparatus according to claim 1, wherein the first attribute data includes moving images, still images, and chart data, and the second attribute data includes text data. An information processing method capable of processing an electronic document including data of different attributes,
A layout analysis step of analyzing the arrangement structure of data included in the electronic document in attribute units;
A comparison step of comparing the arrangement structure of the selected first attribute data with the arrangement structure of the second attribute data referring to the first attribute data;
If the data structure of the first attribute is different from the data structure of the second attribute based on the comparison result of the comparison step, additional data is generated, and the other is combined with one of the arrangement structures. A change process for changing the arrangement structure of
A registration step of registering the additional data generated by the changing step as search data relating the data of the first attribute and the data of the second attribute;
An information processing method comprising:   The changing step is characterized in that, based on the comparison result of the comparison step, the data attribute structure of the first attribute or the data attribute structure of the second attribute is changed in accordance with the more complicated arrangement structure. The information processing method according to claim 5.   6. The information according to claim 5, wherein when a plurality of additional data is generated by the changing step, the registration step sets importance for each of the plurality of additional data and registers the data as search data. Processing method.   The information processing method according to claim 5 or 6, wherein the first attribute data includes moving images, still images, and chart data, and the second attribute data includes text data.   A program for causing a computer to execute the information processing method according to claim 5.   A computer-readable storage medium storing the program according to claim 9.

Images