JP2601111B2 - Document element retrieval device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a document element for decomposing a document printed on paper into document elements such as texts and images and storing and retrieving the document elements in a multimedia file such as an optical disk. Ru about the search equipment of <br/>.

[0002]

2. Description of the Related Art A conventional search apparatus is disclosed in IEE, Computer (September 1987), No. 17
Pages 41 to 41 (J. Conklin, Hyperte)
xt: An Introduction and Su
rvey, Computer, pp. 17-41, S
As discussed in Eptember 1987), there is a hypertext system as a system having nodes and links. However, this is not a system for handling documents printed on paper as the object of the present invention. In addition, unlike the present invention, there is no user interface for separating and registering document elements based on a paper document. For this reason, it was not clear what an easy-to-use interface system for registering document elements was.

A conventional document database system is described in pages 410 to 413 of Introduction, Tow, Modern, and Information Retrieval (1983).
Page (G. Salton and MJ McGil)
l, Introduction Modern I
nformation Retrieval, McGr
aw-Hill Book Company, pp. 4
10-413, 1983), is a search in bibliographic units based on bibliographic items and keywords.
It has been difficult to search for a desired document based on the contents of the document such as a photograph or a chart. Further, the system is not a system for handling a document printed on paper as the object of the present invention. Further, as a system for storing document images, an electronic filing system has been developed by Hitachi Review, 36, Number 4 (1987), pages 213 to 220 (S. Ito).
h and N.H. Takahashi, HITFILE
650 Optical Disk Filing
System, Vol. 36, no. 4, 1987, p
p. 213-220), the document is stored as an image in page units, and the registration date of the document,
The document number or the like is used as an index in the search. Therefore, it is not enough to separate, organize, and search figures and photographs of academic literature and the like. For this reason, it has not been clear what kind of an easy-to-use user interface method for retrieving document elements.

[0004]

The prior art described above does not consider separating, accumulating, and retrieving images of document elements such as photographs and charts in a document image. There was a problem that it was not enough to organize and search. SUMMARY OF THE INVENTION An object of the present invention is to devise an easy-to-use user interface method for retrieving document elements such as photographs and charts in a paper document. In the present invention, a document printed on paper is decomposed into elements and document information is registered. A document is represented by a network consisting of nodes indicating elements and links indicating relationships between the elements, and this network allows browsing via a screen and an operation of registering a new document. Texts and images are associated with the nodes of this network as document elements. Also,
As the link, the document logical structure and the reference relation are handled.

Another object of the present invention is to display a full-color element image at high speed. In order to browse through the element images, full color images are sequentially displayed on the display screen. If the time required to display one element image is long, the waiting time of the user increases, the browsing efficiency is low, and the usability is not good. The time that the user can wait for screen display in such a browse search is considered to be limited to 2 to 3 seconds per one element image. If the display takes more time, the user's patience limit for the waiting time is exceeded, and the usability deteriorates. Therefore, the element images are displayed at high speed.

Another object of the present invention is to perform an interactive search in browsing element images. An interactive operation is performed to search for a target element image via the display screen. As an interactive operation performed by the user to search for an element image, the display mode is switched to display the element images one after another on the screen, or the element image found visually is specified. In this interactive search,
The search efficiency differs depending on the search method up to the target element image. When a user reads a book, information is obtained by repeatedly reading and reading quickly, but in the same way, the display of element images is displayed at high speed and at low speed so that the screen can be glanced and glanced at To be able to

Another object of the present invention is to narrow down element images by character string search. It is not efficient to search for a desired element image only by browsing from a large number of element images. For this reason, a keyword is previously assigned to an element image, and a set of the corresponding element images is obtained by a character string search. The element images in the document include, for example, captions of figures and tables, which are used as keywords. Therefore, the caption is extracted together with the element image, and this character string is assigned as a key word for a search.

[0008] In a conventional electronic filing system for storing paper documents, document elements are arranged one-dimensionally continuously, and a user is physically forced to read a document one-dimensionally. Searching is difficult. Another object of the present invention is to organize and conceptualize document information, so that a user can not only read in the order of text, but also
The entire structure of chapters and sections is displayed in a tree structure, and important parts can be skipped, for example, introductions and conclusions can be read first. Further, the reference relationship can be traced based on the pointer.

Another object of the present invention is to provide a user interface method related to a registration function in a search system for separating an element from a document printed on paper, registering the element in a multimedia file, and performing a search. .
As a registration function, a user interface system is provided for each of registration of element images, registration of relevance, and display of registration results.

[0010]

In order to achieve the above object, the present invention comprises a frame memory for displaying elemental images at high speed, and displays a reproduced video signal of the frame memory in combination with a video signal for display.

According to another aspect of the present invention, a display speed switching function is provided. Further, when sequentially displaying the element images, a function of switching a search direction is provided so as to enable display in the forward direction and display in the reverse direction. Further, a feed display for sequentially displaying the element images in a window and a still display for gazing at the corresponding element image are possible, and a function of switching between the feed display and the still display is provided.

In order to achieve the above-mentioned other object, a user interface for registering document elements is provided so that document information can be organized and conceptualized. Here, a registration operation is performed interactively through a display screen, and an operation function of cutting out and registering an element image from a document image, an operation of providing an association between elements, and a function of displaying a registration result are provided. It is. In addition, the registration screen and the search screen method were clarified.

[0013]

A search system for performing a search on a document printed on paper by using the content as a clue comprises registration, search, printing, editing, and communication functional units. In this system, a multimedia file is stored with a search attribute added so that a document can be accessed for each element. Among these functional units, the registration unit captures an image of the surface of the document with an input device such as a scanner, and extracts the surface image as a document image. In addition, it is possible to register a document in a structured manner, to add relevance, and to register a figure and a table separately. Here, the document image is decomposed into elements, a document structure such as a layout structure is extracted, and registered in a multimedia file. Further, the registration unit gives relevance between elements of the document, and separates and registers the charts in the document so that the charts can be searched. On the other hand, the search unit searches for elements of the document stored in the multimedia file, and has functions of a related search and a chart search. Among them, the related search obtains desired information while interactively tracing the relevance of the document elements via a screen of a display device provided in the search system. In the chart search, a chart corresponding to the axis name such as the vertical axis or the horizontal axis of the chart is searched for and displayed on a screen. In addition, the element images are sequentially displayed on the screen at high speed, and a search for browsing is performed.

[0014]

An embodiment of the present invention will be described below with reference to FIG. FIG. 1 is a diagram illustrating a configuration of a search system that performs a search on a document printed on a paper surface using the document content as a clue. Here, in particular, diagram areas such as photographs and illustrations are stored as element images, and these element images are browsed (browsing) via a screen. In this search system, first, in the document image input unit 100,
A surface image of a paper document is input by a color scanner. For the input document image, a document element separation unit 1
In step 01, document elements such as text and charts are separated and identified, and the attribute registration unit 102 registers the attributes of the document elements. A multimedia file 10 composed of a large-capacity storage device such as an optical disk or a magnetic disk
The element images and the search attributes are registered and stored in the element image file 104 and the search attribute file 105 in No. 3 respectively. As a search function for an element image, a normal character string search function and an interactive browse search function are required. Here, the character string search unit 106 has a normal keyword search function for performing a search based on a character string given to an element image such as a color chart, and thereby narrows down a large number of candidate document element images. Do. In addition, the browsing search unit 107 needs a browsing function of sequentially displaying the corresponding element images on the screen and interactively searching for a desired chart. A full-color element image is sequentially displayed on the screen, and the target element image is displayed. It has a function to find it visually and save it as a hard copy. The processing for automatically separating elements and registering attributes in the document element separation unit 101 is described in Procedure, Second, International, Symposium, On, Interoperable Information System (1988), No. 213.
To 238 (H. Fujisawa, H. Yas)
hiro, J .; Higasino, Y .; Shima,
Y. Nakano, T .; Murakami, Docum
ent Analysis and Decompos
ition Method for Multimed
ia Contents Retrieval, Pr
oc. of the second Int. Sym
Posium on Interoperable I
nformation Systems (ISIIS8
8), p. 231-238, 1988.11. )It is described in.

FIG. 2 shows another embodiment of the document element registration unit of the search system. This figure is a diagram for explaining a document element registration process. In the document structuring and attribute registration processing unit 200 indicated by the document image 210 input by the scanner 211, elements are automatically separated and associated, and search attributes serving as clues for search are stored in the multimedia file 201. register. At this time, 202
In the editing processing section, the registration error in the multimedia file is corrected and a search attribute is added. Thereby, the reliability of data is improved. In order to make the search system easy to use, as a user interface,
The component is specified from the document image 204 displayed on the screen 203 via the screen, the search attribute is changed, added or deleted, and the search attribute such as the content and type of the registered component is displayed on the screen. . The search attributes to be registered include element content, element type (chart, text),
There are layouts of elements, relationships between elements, and the like. In addition, as a registration function in the user interface, each function of changing a search attribute, adding a search attribute, deleting a search attribute, and displaying a search attribute is provided.

FIG. 3 is a diagram for explaining the relevance of document elements. In the element registration user interface, these elements and their relationships are registered. The document image 300 is composed of elements 301, 302, 303, and 30 composed of text and charts.
4,305. These elements are the partial images 311, 312, 316, 3 cut out from the document image.
13, 314, 315 are represented by rectangular areas, and the relations 322, 323, 32 of "is a part of"
4, 325, 326, and 327 are linked to the image 300 of the entire page. In addition, the elements have a hierarchical relationship or a reference relationship of a logical structure, and for example,
FIG. 311 and the description 312 of the figure are connected by a link 320 indicating a reference relationship. In the user interface of the search system, such a document image element is registered using the display device 203, the mouse 205, and the keyboard 206 shown in FIG.

In the element registration user interface, the registered elements are changed, deleted, and added. In this element registration, the user interface has a function of setting the content of the element, ie, a partial image, a function of setting the type of element, and a function of setting layout and association between elements. In the element image setting function, the element is designated as a rectangular area in the document image, and a partial image of the area is cut out. As a result, elements can be changed or added. FIG. 4 is an explanatory diagram of a rectangle setting interface in a document image. On the display screen 400,
The image 401 of the entire page is displayed and a cut-out rectangle is set.
02 and a method of inputting coordinate values 411 and 412 in the coordinate setting window 410 using a keyboard can be used together. In the former method, an image of the entire page is displayed, the cursor on the display is moved, and the position coordinates are input. At this time, since the element image is represented by a rectangular area, the vertex 403 at the upper left corner of the rectangle is displayed.
And the vertex 404 at the lower right corner is set with the cursor and the coordinates are input.

When changing or deleting a registered element, it is necessary to select and specify a target element from a plurality of elements. FIG. 5 shows a screen used for a document element selection interface. In the document 501 displayed on the display screen 500, a plurality of elements 510, 51
1, 512, 513, 514, 515, and 516 are displayed in a rectangular area, the cursor 540 is moved to the corresponding rectangle, and the mouse is clicked to display, for example, the element 51.
Specify 1. Or, the name list window 5 in the screen
At 20, a list of names given to the elements is displayed,
Character strings 530,531,532,533,5 of the name
By pointing 34, 535, 536 with the cursor 541, the corresponding element is designated. If there are many elements,
By specifying the arrow 521 for scrolling,
Scroll the window. FIG. 6 is a diagram illustrating reduction and enlargement display of an element image. Displaying the document elements registered in the multimedia file as described above and allowing the user to check the contents of the file are indispensable in realizing the user interface. For this reason,
Display the element image on the display. Due to the limitations of display resolution and size, it is not possible to simultaneously display a large number of element images on the display. Therefore, as shown in this figure, a function of displaying the element image 603 as it is on the display 602 and a function of reducing and displaying the element image 601 as shown on the display 600 are provided.

The document element registration interface has functions of an operation of cutting out and registering an image of an element from a document image, an operation of giving a relationship between elements, and a display operation of a registration result. The operation of registering the elements of the document image is performed using a cursor, a mouse, and a keyboard via a display screen. Types of registration operation include change, addition, and deletion of an element. FIG. 7 is a registration screen of an element image. To perform the registration operation interactively, the elements already registered in the multimedia file are displayed on the display screen 700 as partial images as shown in FIG.
Is displayed in page units. Here, a document image is displayed on the screen as shown at 701. In addition, as shown in 702, the element images are displayed in texts 710, 712, 713, 715 and charts 711, 714 with their types classified by character strings and the like, and the arrangement of each element in the entire page is represented by a rectangular area. Show. Further, in order to confirm the registration result of the chart, the charts 722 and 723 are separated and displayed from the document image as shown on the display 720. Here, the document image has a function of displaying only the outer frame on the document image 721.
The user interface for image registration includes a mode for registering an original image of the entire document page and a mode for registering a partial image from the original image. In the screen for registering the original image of the entire page of the document, the screen is divided into, for example, left and right windows, the right window is an attribute window, and a file name, an image size, and an image type are input as attributes of the original image. Or display. At the top of the attribute window, a menu of operation commands is displayed. By moving the cursor to one of these and clicking the mouse button, the operation command is selected and executed. Operation commands include “registration”, “display”, “list”, “end”, and “copy” commands. Among them, the “register” command newly stores the original image in the multimedia file. The "display" command displays the original image in the left window, and displays the original image by specifying a file name. The “list” command displays a list of file names of the original images that have already been registered. The "copy" command makes a hard copy of the display screen.

In the screen of the user interface for registering a partial image from the original image of the entire document, the screen is divided into, for example, left and right windows, and the registered element images are displayed in a red window in the left window. The window on the right is an attribute window for inputting and displaying a file name of a partial image, a file name of a thinned image, a thinning rate, and the like. At the top of this attribute window, a menu of operation commands is displayed.
It has commands for "new", "change", "delete", "list", "end", and "copy". Among these commands, the “new” command newly registers a partial image in the multimedia file. The “change” command corrects and changes the cutout position and the like of the already registered partial image. The “delete” command deletes a registered partial image.
The “list” command displays the file names of the registered partial images. In the element image registration operation, on the operation screen of the "change" command, select the "change" command from the menu at the upper right, move the cursor to the partial image to be changed, and click the mouse to select the element specify. The specified partial image is displayed in a red line frame in the left window of the screen, and the cutout position is specified by the cursor. Here, the changed cutout position is indicated by a blue line frame. At the upper part of the screen, there are "confirmation" and "copy" as command menus. The changed position is visually confirmed, and the element image changed by selecting the "confirmation" command is stored in the multimedia file.

FIG. 8 shows a display screen for character lines and characters. On the display 800, a frame 801 of the document is displayed, and for a text element,
The character lines 802, 803, and 804 are displayed. Furthermore, for the text area separated for each character, as shown on the display 810, the separated characters in the frame 811 of the document are represented by rectangles 812, 813, 814, and 815.
Displayed with.

FIG. 9 is a diagram for explaining an operation for registering an element image. In the change operation of the element image registration operation, the cutout position of the element image displayed in a rectangle is changed. For this reason, as shown in this figure, first, in the document image 901 reduced and displayed on the display screen 900, the element image 904 of interest is selected from the element images 903 and 904. Since the element image is displayed as a rectangle, a cursor is set in the rectangle on the screen to specify a target element. And
The position of the upper left corner and the lower right corner of the rectangle is set as a new element cutout position with the cursor. The name of the element image, the name of the layout, the type of the element, and the like are corrected and input from the keyboard. In addition, the element image is newly registered in the multimedia file by the adding operation. Therefore, from the image of the entire page of the document displayed on the display screen, the cutout position of the element is newly set by the cursor, and the name of the element image, the name of the layout, the type of the element, etc. are newly input from the keyboard. . Further, in the deletion operation, the element image of interest is selected by the cursor from the element images displayed on the display screen, and the deletion processing is executed. In the change and addition operations, at the same time as the element images are registered, reduced images of the element images are generated and registered in the multimedia file. Enlarge the document image to change the rectangular area,
Additions and deletions can be made, and the displays 910, 9
20, the enlarged document images 911 and 921 are displayed, and character lines 912, 913, 914 and 915 or
Characters 921, 922, 923, 924, 930, 931
, A function of performing a registration operation.

FIG. 10 is a view for explaining an interface for registering layout names of elements. Display 10
At 00, the document image 1001 is displayed, a rectangular area can be set in the document image, and the name of the rectangle can be input by key. For example, the rectangular area 100
2 is given a title as a name. Also, the rectangular area 1
A summary is given to 005 as a name.

FIG. 11 is an explanatory diagram of a layout display screen. As shown in the figure, in the display 1100,
The layout of the elements of one page 1101 of the document is shown, and is reduced and displayed on the display screen 1100 so that the whole can be displayed. Here, the text area 1110 has a function of displaying a logical structure in a rectangular nested format. For example, a chapter title rectangle 1120 and section rectangles 1121 and 1123 are displayed in a chapter area 1111. You. Then, within the section, a rectangle 11 of the title of the section
22, 1123 etc. are displayed in a nested format.
By selecting and designating these reduced element images, as shown on the display 1130, a part 1131 of the enlarged document image can be displayed and the element image 1141 can be displayed.

FIG. 12 is a diagram for explaining the operation of registering the association. On the display 1200, the document image 12
01, a plurality of elements 1202, 1203 or 1204, 1205 are selected, added to the association, and registered in the multimedia file. Related elements include a link between text and text, a link between text and chart, and a link between charts. Here, an operation for giving a relationship between the diagram element 1202 and the text element 1203 will be described. The registered element images are displayed on the display screen 1200 in a format retaining the layout, and the types of the elements are also displayed.
As the display of the type of the element, for example, the element of the text is displayed by being surrounded by a blue rectangle, while the element of the chart is displayed by being surrounded by a red rectangle. The association is performed by specifying two pairs of elements. The element to be paired is selected by setting the cursor within the rectangle. The association is displayed by a line segment 1220 connecting the element 1212 and the element 1213. In the display 1211, the line segment 1 is located between the two elements related to each other.
220 and 1221 are connected.

In the present embodiment, a function of displaying a list of registration results is provided, whereby errors in element registration can be reduced and the reliability as data can be improved. FIG.
FIG. 3 is a diagram illustrating a method of displaying an element list display screen. In this list display, the registered result is displayed on the display screen 1300, and the registered contents are inspected while looking at the screen. For this reason, in the registration user interface,
A list of element images stored in the multimedia file is displayed on a display screen, and registered elements can be visually confirmed. Since the number of element images is large, a plurality of element images are simultaneously displayed on the display screen. Further, by turning the screen, the element images are sequentially displayed. Here, since a large number of element images are arranged and displayed on one display screen 1300, reduced images 1301, 1302, 1303, and 1303 of the element images are displayed as shown in FIG.
4,1305,1306,1307,1308,130
9, 1310 are displayed. By displaying such reduced images, it is possible to check the outline of the elements and to check a large number of element images at a time. Further, when the content of the element image is to be checked in detail, by selecting one of the element images arranged on the screen 1300, an image 1321 obtained by enlarging the element image is displayed on the screen 1330.
20 is displayed. In addition to displaying the enlarged image 1321, an element image previously associated with the image, for example, a legend 1322 in the figure is also displayed.

FIG. 14 is an explanatory diagram of the structure of a multimedia file. As shown in the figure, document data 1401 and control data 1400 for controlling access to these image data are stored in the multimedia file. The document image data 1401 includes image data (original image data) 1402 of one entire page of a document and partial image data 1403 as an element. Furthermore, reduced image data 1404 obtained by reducing these image data at the time of registration is also provided. The data format of the image is stored as uncompressed or compressed data depending on the presence or absence of data compression. The image can handle monochrome data collected by a single-color scanner or color data having three primary color components collected by a color scanner. Further, these pixel values are expressed as 8-bit grayscale values or 1-bit binary data.

FIG. 15 is a view for explaining the format of control data for controlling image data. The control data 1400 is
Original image file data 1500, partial image file data 1501, layout data 1502, relational data 1
503, character line data, character data, and the like. These data are provided with numbers 1510, 1520, 1530, and 1540 indicating the type of data in a 2-byte header portion, and control data is identified by the type number. The original image file data 1500 includes control information relating to the image of one page of the document, and includes an original image number 1511 for designating the original image, a file name 1513 in which the original image is stored, a horizontal image of the original image, Vertical size 1515,
There are 1516. The partial image file data 1501 is
The control information about the partial image cut out from the original image is provided. The control data includes an original image number 1521 indicating the original original image, a file name 1526 storing the partial image, and horizontal and vertical Size 152
7,1528. The layout data 1502 is linked with the partial image file data 1501 and stored, and has rectangular position coordinates 1534, 1535, 1536, and 1537 indicating a partial image on one page of the document. The layout type 1532 indicates whether the element is a text or a chart. Relational data 150
Reference numeral 3 denotes an element having a relation, and the relation is expressed by layout numbers 1541 and 1542 of the two elements. Like the layout data, the character line data and the character data have rectangular position coordinates indicating the character line and the character on one page of the document.

In the search system, a document image is input, and its components are registered in a multimedia file such as an optical disk. To register, use the mouse on the workstation,
Perform interactively using the keyboard. Therefore, this workstation has a color C for displaying a document image.
A display device having an RT is provided. A magnetic disk and a rewritable optical disk are connected to the workstation, constitute a multimedia file, and store an entire image and a partial image of a document. Further, a monochrome scanner and a color scanner can be used for inputting a document image. In the software configuration of the user interface for registering the element images, the overall control unit controls the entire processing of the user interface. It consists of a file input / output unit, original image registration unit, element image change / addition / deletion unit, and element display unit thereunder. Furthermore, as routines common to these processing units, an image display unit that displays the entire image and the partial image, an image data generation unit that generates a thinned image obtained by cutting out or reducing the partial image from the original image, and a window frame on the screen. There is a window control that generates and controls commands from the keyboard.

FIG. 16 is an explanatory diagram of a search for an element image. The search is a two-stage process, which includes a character string search and a browse search. In this search, first, desired element image candidates are narrowed down by a normal character string search for a set 1600 of element images such as charts and the like separated and extracted from the document image. Element images 1601, 1602, 160
3, 1604, 1605, and 1606 include keywords 1611, 1612, 1613, 1614, and 16 respectively.
15, 1616 are provided. The set 1620 of the narrowed-down element images is divided into groups 1621, 1622, and 1623 according to the keyword used for the character string search. Here, a keyword is assigned in advance for this character string search. A character string search is performed based on the keyword to obtain a set of candidate element images. Element images have captions for figures and tables, which can be used as clues for character string searches. For this group of element images, for example 1622, then the display screen 165
0 to interactively browse and search for a desired element image. At this time, in order to reduce the waiting time for browsing the full-color element images, the frame images 1642, 1643, and 164 are stored in the frame memory 1641.
4 is stored. The frame image is the element image 16
32, 1633. Frame memory 1641
Sequentially display the frame images stored in the display screen 1
650 in the window 1651.

FIG. 17 is a configuration diagram for high-speed browsing of a document element image. The frame memory 1702 includes a frame memory 1702 for browsing the document element image 1701 at high speed. The frame memory 1702 is connected to the workstation 1700 by a bus. In the workstation 1700, the element image is separated and extracted from the document image. The data is sent to the memory 1702 and stored as a frame image. From the workstation 1700, the frame memory 170 is designated by specifying the frame number.
2 is accessed, so that a frame image can be output as a video signal. In the superimposing device 1703, the workstation 1700
Are combined with the video signal of the frame image, and the combined video signal is displayed on the display device 1704. Display device 1
A window 704 displays an element image.
05 and a window 1706 for performing a search operation. The element images are sequentially sent to the element image display window 1704 at high speed in the forward or reverse direction.
Alternatively, it is displayed stationary. On the other hand, a window 1706 for performing a search operation executes a character string search and a browse search.

FIG. 18 shows a processing procedure for content search for a document element image. The processing procedure is as follows:
At 00, the element images are input to the frame memory, and at step 1801, the search attributes for each element image are stored in the search attribute file. Here, a character string of a caption such as a chart is separated and extracted in advance by a character recognition technique and used as a search attribute. Next, the process proceeds to a search process. First, a normal character string search is performed. At this time, in step 1802, the user specifies a character string for search as a keyword. In step 1803, a character string search is executed, and in step 1804, the identifier of the corresponding element image is stored as a search result of the character string search to form one group. In the next step 1805, the element images belonging to this group are set as the range of the browse search. Instead of browsing through a large number of element images, this character string search narrows down the element images to be browsed. Of course, the character string search can be omitted and all the element images can be sequentially browsed. Execution of the browse search in step 1806 is performed interactively via a screen. The browsing is performed by visually searching using both the high-speed display and the low-speed display to find a desired element image. When a desired element image is found, the mode is switched to the still image display mode, and an operation such as taking a hard copy in step 1807 is performed to save the image as a search result.

FIG. 19 is an explanatory diagram of a display screen of a workstation for performing a content search. The screen 1900 includes an element image display window 1901 for sequentially displaying element images, and a search operation window 1902 for performing an interactive search operation. The size of the display screen 1900 is 1280 (horizontal) × 1024 (vertical) pixels. The size of the element image display window 1901 is fixed at 640 pixels horizontally × 512 pixels vertically, but this size matches the size of the frame image. Element images separated and extracted from document images have various sizes, and therefore,
In order to match the size of the element image display window, processing of enlarging and reducing an image and embedding a margin is performed.

Element images such as figures and photographs are separated from a color document image, and attributes serving as clues for retrieval and element image data are registered. The element image is extracted from the document image as a rectangular area. The element images are stored in the frame memory so that they can be displayed at high speed via the screen. Since figures and tables of various sizes are described in the document, when storing the element image in the frame memory, the size of the element image is adjusted to the size of the frame image of 640 pixels (width) × 512 pixels (height). FIG. 20 shows a procedure for generating a frame image.
In this procedure, first, a document image is input in step 2000, and element images are separated in step 2001. In order to match the size of the separated element image with the size of the frame image, in step 2002, the element image is enlarged or reduced. Then, in step 2003, the element image is input to the frame image. At this time, the enlargement ratio and the reduction ratio are set to the same value in both the vertical and horizontal directions so that the element image is not distorted. For this reason, in step 2004, an embedded frame image is generated in a blank portion using, for example, white pixels. For simplicity, the enlargement ratio and the reduction ratio may be integers and their reciprocals. Next, in step 2005, a frame image is output.

FIG. 21 is an explanatory diagram of a process of processing a document element image. First, a separation processing unit 2102 separates an element image inside the document image 2100, for example, a photographic area 2101 indicated by a black line frame in the figure. Then, the element image 21
From 03, an enlargement / reduction processing unit 2104 generates an enlargement / reduction image 2105. Note that the reduction processing is realized by a method of averaging pixel values. Then, the embedding processing unit 2107 embeds the margin 2110 with pixel data having a specific value, and generates a frame image 2108 of 640 pixels horizontally × 512 pixels vertically. The element image to be registered is displayed on the screen, gamma correction (density correction) can be performed while looking at the screen, and the image after the gamma correction is stored in the frame memory as the element image.

In registering the document element image, the document image is displayed on the screen, and the automatically separated element image is displayed in a rectangular area. Also, element images can be separated and extracted interactively, and an element image such as a chart is specified and cut out with the mouse at the upper left point and the lower right point of the rectangular area. The cut-out image is registered as attribute images with attribute information. This registration operation is editable, and the registered element image can be deleted, newly generated, the position can be changed, and the like. FIG.
Shows the procedure for registering a document element image. Step 22
At 00, a menu command displayed on the screen is selected, and a registration operation from step 2220 is performed.
The operations are "new" 2201, "delete" 2202, "change" 2203, "list" 2204, "copy" 2205,
"End" 2206 is included. For example, "new" 2201
When the menu command is selected, the process proceeds to the process of inputting coordinates in step 2210, and the coordinates of the rectangular area on the screen are sequentially input. In step 2211, those values are registered. In the menu command of “delete” 2202, the registered element image is designated by the cursor in step 2212, and the element image is deleted in step 2213. In the command of “copy” 2205, step 22
At 17, a full color hard copy of the registration screen can be taken.

Using the character string assigned to the element image as a clue, a normal keyword search is performed. A set of corresponding element images is registered as a group. For example, for the sake of simplicity, element images are now described as e1, e2, e3, e4, e5.
e6. Assume that a plurality of groups are generated by keyword search, and the groups are G1, G2, and G3. Here, the element images obtained by the keyword key1 are e1, e2, and e4, and these are group G
It is set to 1. Similarly, the element images of the group G2 obtained by the keyword key2 are e3 and e5, and the element images of the group G3 are e6. When browsing element images, element images are displayed on the screen for each group in order,
Visually find the desired element image. At this time, the order in which the element images are displayed on the screen is determined according to the degree of matching in the keyword search. For example, the group G1 displays the element images e2, e4
It can be performed in the order of e1. Here, for the sake of explanation, it is assumed that the degree of matching in the keyword search is higher for the element image e2. Further, the display order can be simply the order obtained by the character string search.

In the registration of the group information, editing such as new creation, change, and deletion can be performed separately from the screen for registering the element images. In this operation, first, when a group number is specified, a list of display order numbers and frame numbers belonging to the corresponding group is displayed. In the change, a frame number is selected and input, and a display order number is input. When executed, a changed portion is registered, and a list of display order numbers and frame numbers in the set group is displayed again. In the new creation, a frame number is input, and then a display order number is input. Similarly, when execution is performed, the display is registered and the list of display order numbers and frame numbers in the set group is displayed again. In the deletion, a frame number is selected and input, and when executed, the frame number is deleted, and a list of display order numbers and frame numbers in the set group is displayed again.
In order to arrange the display order numbers, when the display order numbers are set to the same registered numbers at the time of input, the previously set numbers are prioritized, and the numbers are renumbered at the time of redisplay. . Regarding the handling of element images that do not belong to a group, element images for which these groups are not set can be accessed and edited by designating a group number 0. Further, in order to confirm the registered element image by display, the corresponding element image is displayed on the screen by inputting a frame number.

FIG. 23 is an explanatory diagram of browsing of document element images. The frame memory is accessed in units of frame images 2300, 2301, 2302, and 2303 obtained by enlarging and reducing the element images, and displayed on the search screen. The display includes a forward display 2310 for continuously displaying frame images in the forward direction and a backward display 2311 for displaying in the reverse direction. In addition, there is a stop display 2312 for displaying a certain specific frame image on the screen stationary. In the browsing operation, a frame image is displayed in a forward direction, and a user sequentially looks for a target frame image visually. When the user finds the corresponding frame memory, the display is switched to the stop display. This switching is performed by the user pressing a function key, and the user can observe the frame image in detail in a stopped display state. In addition, a reverse display function is provided so that the frame image can be returned to the original state when the display in the forward direction is excessive. Here, when the frame images to be displayed are exhausted, a new frame image is displayed from the first frame image, and the display is performed continuously.

FIG. 24 is an explanatory diagram of the procedure for switching the display mode of the document element image. The display speed of the frame image can be switched to an arbitrary speed in units of 1/30 seconds. Here, a high-speed display mode and a low-speed display mode are provided for browsing element images. Step 2400 for switching the display mode of the element image.
The function key pressed by the user is read with.
Next, in step 2401, the display mode is switched.
In the high-speed mode 2402, high-speed display is performed in step 2410, and five frame images are displayed per second. In the low-speed mode 2403, low-speed display is performed, and display is performed at a speed of one sheet per second. In the stop mode 2404, the stop display 2
At 414, the user can observe the frame image in detail.

The frame memory is configured to store a large number of frame images. Each frame image is assigned a frame number, and is accessed by this frame. There are two inputs to the frame memory, one of which can transfer elemental images from the workstation via the VME bus. The other is a video signal input.
Image data can be input in units of / 30 seconds. The output to the frame memory is also of two systems.
ME bus transfer and video signal output in 1/30 second units are possible. In elemental image browsing, input to the frame memory stores the elemental image from the workstation via the VME bus. As for the output, a frame image is output as a video from the frame memory, and combined with a display video signal and displayed. FIG. 25 is an explanatory diagram of an access method of the frame memory. Frame memory 2520
Is accessed by the frame number 2522.
When browsing an element image, first, the group table 2501 is accessed by the group number 2500. The group table 2501 includes a group number 2502, a frame number 2503, and a display order number 2
504 are stored. By designating a group number 2500, a set of corresponding frame numbers is extracted and stored in the display frame number table 2510 according to the display order. This display frame number table 251
The frame number 2511 is read sequentially from 0, and the frame memory 2520 is accessed.

There are two types of element image formats, one pixel 4 bytes / one-plane format and one pixel 1 byte / one-plane format. The format that can be used in the frame memory is a one-byte four-byte one-plane format, but is not limited thereto. The size of the frame image is 512 pixels vertically and 640 pixels horizontally. Further, the pixel data is a full color of a 32-bit configuration, and blue, green and red 8 bits are arranged from the lower byte. The most significant byte is dummy data.

FIG. 26 is an explanatory diagram of the data format of the search attribute file. The search attribute file is composed of original image file data 2600, element image file data 2601, group data 2602, and keyword data 2603, which are numbers 2610 and 2 indicating the type of 2 bytes.
620, 2640, and 2650. For example, the original image file data 2600 has the type number 2
610 is 1, followed by the original image number 2611, the original image type 2612, and the file name length of the original image data 2
613, file name 2614, image size 2615,
1616 is stored. Number 2611 of this original image
Manages each document image. The element image file data 2601 has a type number 2621 of 2;
Subsequently, an original image number 2621, an element number 2622, a file name 2625 storing element image data, position coordinates 2626 and 2627 in the original image, and the like are stored. The keyword data 2603 has the type 2
650 is 4 and is managed by the keyword data number 2651. Here, the number 2652 of the assigned element image, character string data 2657 of the keyword, and the like are stored.

FIG. 27 shows the configuration of the content search system. A color document image is input, and its element images are separated and registered. Then, the stored element images are interactively searched for via the display screen. In the document image input station 2701, a surface image of a color document is collected by the color scanner 2702, and the element images are separated and extracted. The registration station 2703 interactively performs an operation of registering an element image via a screen 2704. In the search station 2705, browsing of the element images stored in the frame memory 2707 is performed on the search screen 2706 to search for and obtain a desired element image. A color hard copy 2708 is provided for copying frame images.
A hard copy of an element image as a search result can be taken.

FIG. 28 is an explanatory diagram of an apparatus configuration for browsing element images. A frame memory unit 2801 is connected to the search station 2800 via a VME bus. The frame memory unit 2801 includes a semiconductor image memory 2809, stores a large number of frame images (640 × 512 pixel full-color images), and inputs and outputs frame images in 1/30 second units. The output video signal 2808 of the frame image is transmitted to the workstation 280 by the super imposer 2802.
0 on the display screen 2803. The frame memory unit includes a laser video disk 2811 and a U-matic VTR as auxiliary storage devices.
2812 are connected. Laser video disc 28
11 and a semiconductor image memory 2809 constitute a memory hierarchy. A plurality of contents of the frame memory storing the element images can be displayed on the screen.
You can scroll according to the frame number.
Note that a trackball is used for scrolling. The selectors 2805 and 2806 can switch between the laser video disk 2811, the U-matic VTR 2812, and the semiconductor image memory 2809, and the memory is used hierarchically depending on the capacity of the element image to be stored.

[0046]

According to the present invention, in searching for a document element image, an element image separated from the document image such as a full-color photograph or a chart can be browsed through the screen.
There is an effect that the element images are displayed one after another on the screen, and the user can visually find a desired element. Also,
A high-speed display mode and a low-speed display mode are provided to switch the display speed of the element images on the display, and the search direction can be switched sequentially and in the opposite direction. Therefore, there is an effect that the target element image can be efficiently reached. Further, since this system can organize and conceptualize the document information, there is an effect that the handling of the document information is facilitated and the strategic use of the document can be performed. Also,
Since the registration operation can be performed interactively via the display screen, there is an effect that an operation of cutting out and registering an element image from a document image, an operation of giving an element association, and the like can be executed with high reliability.

[Brief description of the drawings]

FIG. 1 is an explanatory diagram of a configuration of a document element image search system according to an embodiment of the present invention.

FIG. 2 is an explanatory diagram of a document element registration system.

FIG. 3 is an explanatory diagram of the relationship between elements and relevance of a document image.

FIG. 4 is an explanatory diagram of a rectangle setting interface in a document image.

FIG. 5 is an explanatory diagram of a document element selection interface.

FIG. 6 is an explanatory diagram of reduction and enlargement display of an element image.

FIG. 7 is an explanatory diagram of an element image registration screen.

FIG. 8 is an explanatory diagram of a character line and character display screen.

FIG. 9 is an explanatory diagram of an operation for registering an element image.

FIG. 10 is an explanatory diagram of a layout registration interface.

FIG. 11 is an explanatory diagram of a layout display screen.

FIG. 12 is an explanatory diagram of an association registration interface.

FIG. 13 is an explanatory diagram of an element list display screen.

FIG. 14 is a diagram showing a configuration of a multimedia file.

FIG. 15 is a diagram showing a data format of an element.

FIG. 16 is an explanatory diagram of an element image search.

FIG. 17 is an explanatory diagram of high-speed browsing of a document element image.

FIG. 18 is a diagram illustrating a procedure of content search of a document element image.

FIG. 19 is an explanatory diagram of a screen for content search.

FIG. 20 is a diagram showing a procedure for generating a frame image.

FIG. 21 is a diagram illustrating a process of processing a document element image.

FIG. 22 is a diagram illustrating a procedure of a registration operation of a document element image.

FIG. 23 is an explanatory diagram of a browsing method of a document element image.

FIG. 24 is an explanatory diagram of a procedure for switching a display mode of a document element image.

FIG. 25 is an explanatory diagram of an access method of a frame memory.

FIG. 26 is an explanatory diagram of a data format of a search attribute file for browsing.

FIG. 27 is a diagram showing an apparatus configuration of a content search system.

FIG. 28 is a diagram illustrating a device configuration for browsing element images.

[Explanation of symbols]

 103 multimedia file, 107 browse search, 320 reference relation, 510 document element, 603 element image, 700 registration display screen, 802 character lines, 812 characters, 1000 layout registration screen, 1401 document image data, 1502 layout data, 1702 frame memory, 1806 browse search execution step, 2108 frame image, 2310 forward display, 2311 reverse display, 2312 stop display, 2501 group table, 2510 display frame number table, 2601 element image file data.

 ──────────────────────────────────────────────────続 き Continuation of the front page (56) References JP-A-3-273363 (JP, A) JP-A-3-225565 (JP, A) JP-A-2-297182 (JP, A) JP-A-4- 62670 (JP, A) JP-A-60-123961 (JP, A) Information processing VOL. 28, NO. 6 (1987-6-15) P. 756-764 Information processing VOL. 28, NO. 6 (1987-6-15) P. 710-720 Information processing VOL. 28, NO. 6 (1987-6-15) P. 705-709

Claims (1)

(57) [Claims] And means for inputting a document image and registering a document element and a search attribute of the document image.
Interactively set the display position of the document element via the display screen
Means, a document element image obtained by separating the document image, and a layer of the document image.
Means for storing data and related information between document elements
And assign a keyword to the document element image and
Performs a character string search on the
Document element images belonging to a group in order on the display screen.
A document element search device having means for displaying next , wherein a rectangular frame is set on a display screen and vertex coordinates of the rectangular frame are set.
By holding, document element images are divided as rectangular areas.
Release the document element image in the text area or image area
The identification information is stored in advance and displayed on the display screen,
Select identification information for each separated document element image
And store the input identification information for each document element image
Means for specifying a plurality of document element images for the separated document element images.
And register the relationship between the specified document element images.
In addition, related information indicating the relationship between the specified element images
For document element images on the display screen as geometric figures such as
Display means and a rectangular frame in the child format for separated document element images.
As a search attribute name for each rectangular frame.
Title, author name, and / or abstract.
Key in a character string, and enter the character string
Means for holding in a document;
Elementary search device.