JP5312310B2 - SEARCH DEVICE, METHOD, AND PROGRAM - Google Patents

Description

  The present invention relates to a search apparatus, method, and program, and in particular, uses a region in a document in which the layout of an image and a character string is determined as a search query, and an index of the document in which the region appears and each position in the document. The present invention relates to a search apparatus, method, and program for creating and executing a search.

  In particular, a search that is applied when you want to uniquely identify a document that may contain an area in the document where the layout of images and character strings has been determined, and the position within the document, not exhaustively. The present invention relates to an apparatus, a method, and a program.

  There are not a few scenes in which it is necessary to uniquely identify from which document a part of a document is included in which document or at which position in which document.

  For example, if you have a catalog cutout at hand, you may want to find the original cutout catalog and read the continuation of the cutout. In this case, it is necessary to be able to uniquely identify which catalog the cutout is part of.

  The above example can be regarded as a search system that uses a partial area of a document as a query and inquires about a document name including the area or a document name and a position in the document from a huge amount of documents.

  As an example for realizing the above search system, the color distribution information of the entire area of each image in the document is stored in a database in advance, and the color distribution information of the entire area of the image in the document is used as a search query. A search system that inquires a database for the name of a document including an image whose color distribution information matches the document name, or the position of the document in the document (for example, see Non-Patent Document 1).

  Alternatively, there is a system that performs a search only by matching a partial area of each image. For example, each image in the document is previously divided into a predetermined number of partial areas, and color distribution information of each partial area is stored in a database. Next, an image in the document is divided into a predetermined number of partial areas, and the color distribution information of each partial area is used as a search query, and the document name including the partial area where the partial area matches the color distribution information, or A search system that inquires a database for a document name and a position in the document can be considered (for example, see Non-Patent Document 2).

M. J. Swain and D. H. Ballard, "Color Indexing," International Journal of Computer Vision, Vol. 7, pp. 11-32, 1991 WALRUS: A Similarity Retrieval Algorithm for Image Databases, Apostol Natsev, Rajeev Rastogi and Kyuseok Shim, IEEE Transaction on Knowledge and Data Engineering, Vol. 16, No. 3, pp. 301-316, 2004.

  The method of storing the color distribution information of the entire area of each image in the database and using the color distribution information of the entire area of a certain image as a search query is simple while matching the information in the database and the search query, There is a problem that the entire area of must be included in the search query.

  On the other hand, the method of storing the color distribution information of the partial area of each image in the database and using the color distribution information of the partial area of a certain image as a search query may not include the entire area of the image in the search query. There is a problem that the number of times of matching is greatly increased because the image is divided into a plurality of partial areas at the time of database storage and search execution, and color distribution information is matched between the partial areas of the database and the search query. .

  The present invention has been made in view of the above points, and a search apparatus capable of improving the probability that a correct search result is obtained when an index is given to a shooting target and reducing the calculation cost required for matching, and An object is to provide a method and a program.

  FIG. 1 is a principle configuration diagram of the present invention.

According to the present invention (Claim 1), a page break or a line feed position is fixed, and a document including the character string in a document including the character string is used as a search query, and the document in which the region appears A search device for creating a search index for responding to a search request to acquire a position in
A document group input means 10 for receiving an input of a document group to be indexed;
Character area specifying means 11 for specifying a character area in a document using a character area extraction method;
Peripheral area specifying means 12 for specifying a peripheral area that is an area existing around each character area;
A partial image specifying means 13 for comparing each peripheral area with a background area of a document using an image feature comparison method, and determining a part of the peripheral area different from the background area as a partial image area;
A character region, a partial image region around the character region, and an index output unit 14 that associates an appearance position in a document in which the character string appears and outputs it to the index DB 15.

Further, the present invention (Claim 2) is the search device according to Claim 1,
The document also includes images,
A partial area including an image and a character string in the document is set as a search query.

Further, the present invention (Claim 3) is the search device according to Claim 1 or 2,
A search target image input means for receiving an input of an image obtained by photographing a partial area of a document including an image and a character string;
Search target character area specifying means for specifying a character area in a search target image using a character area extraction method;
Search target peripheral area specifying means for specifying a peripheral area that is an area existing around each character area;
A search target partial image specifying unit that compares each peripheral region with a background region of the search target image using an image specific comparison method, and determines a partial image region that is different from the background region in the peripheral region;
Query means for querying the index DB for the character area and the document in which the partial image area around the character area appears or the position in the document;
Search result output means for outputting the result of the inquiry.

Further, the present invention (Claim 4) is the search device according to Claim 3,
Inquiry means are
Means for accessing a database in which the position and content of each document are associated in advance and obtaining the content associated with the document position as a search result as a search result;

  FIG. 2 is a diagram for explaining the principle of the present invention.

According to the present invention (Claim 5), a page break or a line feed position is fixed, and a document including the character string in a document including the character string is used as a search query, and the document in which the region appears and the document A search method for creating a search index for responding to a search request for acquiring a position in a device for performing a search,
A document group input step (step 1) in which the document group input means receives an input of a document group to be indexed;
A character area specifying means specifies a character area in the document using a character area extraction method (step 2),
A peripheral area specifying step (step 3) in which the peripheral area specifying means specifies a peripheral area that is an area existing around each character area;
The partial image specifying unit compares each peripheral area with the background area of the document using an image feature comparison method, and determines a partial image area that is different from the background area as a partial image area. Step 4) and
The index output means performs an index output step (step 5) of associating the character region, the partial image region around the character region, and the appearance position in the document in which the character string appears in association with the index DB. .

Further, the present invention (Claim 6) is the search method of Claim 5,
The document also includes images,
A partial area including an image and a character string in the document is set as a search query.

Further, the present invention (Claim 7) is the search method according to Claim 5 or 6,
A search target image input unit for receiving an input of an image obtained by photographing a partial region of a document including an image and a character string;
A search target character area specifying unit, wherein the search target character area specifying means specifies a character area in the search target image using a character area extraction method;
A search target peripheral region specifying means for specifying a peripheral region that is a region existing around each character region;
A search target part in which the search target partial image specifying unit compares each peripheral area with the background area of the search target image using an image specific comparison method, and determines a part of the peripheral area different from the background area as a partial image area. An image identification step;
An inquiry step for inquiring the index DB about the character region and the document in which the partial image region around the character region appears or the position in the document;
The search result output means further performs a search result output step of outputting the result of the inquiry.

The present invention (Claim 8) provides a search method according to Claim 7,
In the inquiry step,
A step of accessing a database in which the position and content of each document are associated in advance and obtaining the content associated with the document position as a search result as a search result is further performed.

  The present invention (Claim 9) is a search program for causing a computer to function as each means constituting the search device according to any one of Claims 1 to 4.

  As described above, according to the present invention, a specific document is selected from a document group by using only a character string and a partial image around the character string instead of using the entire area of the image at the time of index creation and search execution. It is possible to respond to a search request for uniquely acquiring a specific position. For example, even if there is only a part of the catalog cut out at hand, it is possible to determine which part of the book the cut out is.

  Further, in a method for realizing the above processing, the color distribution information of the entire area of each image is stored in a database, and the color distribution information of the entire area of a certain image is used as a search query. There was a problem that the entire area of the image had to be included in the search query. However, in the method of the present invention, only a region including a part of an image and a character string need be included in the search query. In the case of a system in which an area photographed with a handy device is used as a search query, only a narrow area of a document needs to be photographed, so that a query image can be photographed clearly. In the case of a system that can acquire the content associated with the area captured by the handy device, it is possible to inquire for each narrow part of the document, so that the user who associates the content with the document can perform the association at a fine granularity. The user who acquires the content associated with can acquire the content with fine granularity.

  Further, in a method for realizing the above processing, which is a method for storing color distribution information of a partial area of each image in a database and using the color distribution information of a partial area of an image as a search query, There is a problem that the number of times of matching is greatly increased because the image is divided into a plurality of partial areas at the time of database storage and search execution, and color distribution information etc. is matched between the partial areas of the database and the search query. It was. However, in the method of the present invention, only the matching of the partial images around the character area is performed, so that the number of matching operations can be greatly reduced and the processing can be performed at high speed.

  In addition, the information on all areas of each image is stored in the database in common with both the method of handling the above-mentioned image in the whole area unit and the method of dividing the image into a plurality of areas. There was a problem that it was necessary. However, according to the method of the present invention, only the partial image around the character area and the information on the character area are stored in the database, so that the storage area can be reduced.

It is a principle block diagram of this invention. It is a figure for demonstrating the principle of this invention. It is a block diagram of the index preparation part in the 1st Embodiment of this invention. It is an example of a PDF file (book name: vegetable, file name: vegetable3.pdf, page: third page) in the first embodiment of the present invention. It is a flowchart of the process of the index creation in the 1st Embodiment of this invention. It is a data structured list of the page of the inputted document in the 1st embodiment of the present invention. It is an example of character area specification with respect to the PDF file (Book name: vegetable, File name: vegetable3.pdf, Page: 3rd page) in the first embodiment of the present invention. It is a data structured list of the specified character area in the 1st embodiment of the present invention. It is an example of the peripheral region specification for the PDF file (book name: vegetable, file name: vegetable3.pdf, page: third page) in the first embodiment of the present invention. It is a data structured list of the specified peripheral region in the first exemplary embodiment of the present invention. It is another implementation | achievement method (an example in case a character area is "me") of the surrounding area specification in the 1st Embodiment of this invention. It is an example of partial area specification with respect to the PDF file (book name: vegetable, file name: vegetable3.pdf, page: third page) in the first embodiment of the present invention. It is a data structured list of the specified partial image area in the first embodiment of the present invention. 3 is a data structured list of color distribution information of a partial image area in the first embodiment of the present invention. It is a flowchart of the process of the index output part in the 1st Embodiment of this invention. It is an example of the data output to index DB in the 1st Embodiment of this invention. It is a block diagram of the search device in the 2nd Embodiment of this invention. It is a flowchart of the search process in the 2nd Embodiment of this invention. It is an example of the document (paper medium) of the search object in the 2nd Embodiment of this invention. It is an example of the input search object image (JPEG) in the 2nd Embodiment of this invention. It is an example of the file name of the search object image in the 2nd Embodiment of this invention. It is a figure which shows the character area specific process of the search object image (2009-05-20-18-00-00.jpg) in the 2nd Embodiment of this invention. It is a structured list of character area data in a 2nd embodiment of the present invention. It is an example which specifies a peripheral area | region from the search object image (2009-05-20-18-00-00.jpg) in the 2nd Embodiment of this invention. It is a structured list of peripheral area data in the second embodiment of the present invention. It is another implementation | achievement method (an example in case a character area is "G") of the surrounding area specification in the 2nd Embodiment of this invention. It is an example which specifies a partial image area | region from the search object image (2009-05-20-18-00-00.jpg) in the 2nd Embodiment of this invention. It is a structured list of partial image area data in the second embodiment of the present invention. It is a structured list of data of color distribution information of a partial image area in the second embodiment of the present invention. It is a flowchart of the inquiry process in the 2nd Embodiment of this invention. It is an example of the search result based on the data of the character area in the 2nd Embodiment of this invention. It is a flowchart of index DB search in the 2nd Embodiment of this invention. It is an index DB search result list in the second embodiment of the present invention. It is a flowchart of the aggregation process of the index DB search result in the 2nd Embodiment of this invention. It is an aggregation list in the 2nd embodiment of the present invention. It is an example of content DB in the 2nd Embodiment of this invention. It is a flowchart of content DB search processing in the 2nd Embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  In the following, the process of creating an index and the process of executing a search using the index will be described separately.

[First Embodiment]
In the present embodiment, a process for creating an index in the search device will be described.

  FIG. 3 shows the configuration of the index creation unit in the first embodiment of the present invention.

  The index creation unit shown in FIG. 1 includes a document input unit 10, a character region specifying unit 11, a peripheral region specifying unit 12, a partial image region specifying unit 13, an index output unit 14, and an index DB 15.

  The document is a paper book composed of a character string or an image, or a set of pages including the character string and the image. In this embodiment, each page of the book is read by a scanner (general device), and as shown in FIG. 4, the character string and image layout do not change even in different browsing environments (OS, PDF browsing software, etc.). Convert to a file (such as a PDF file). It is assumed that the background color of the document is known as white.

  In addition, the character region specifying unit 11, the peripheral region specifying unit 12, and the partial image region specifying unit 13 have a memory (not shown) that temporarily stores information such as a character string in the course of processing.

  The processing flow for creating an index in the present embodiment is shown below.

  FIG. 5 is a flowchart of index creation processing according to the first embodiment of this invention.

  In the following, an input step for inputting a document to be indexed (steps 110 and 120), a character region specifying step for specifying a character region from the page in the document (step 130), a peripheral region (from the page in the document) A peripheral region specifying step (step 140) for specifying each character region in the page), and a partial image region (region of a partial image used for index creation among the peripheral regions in the page) from the page in the document. The partial image region specifying step (step 150) to be specified and the output step (step 160) for outputting an index for executing the search will be described.

  Step 110) The document input unit 10 receives input of a document group to be analyzed. It is assumed that the format of each document is a PDF file as shown in FIG. 4, and the book name and page as the scan source are known.

  Step 120) The document input unit 10 lists the documents included in the document group, the title of the scan source, and the pages in the data structure shown in FIG. Each element in the list shall uniquely indicate each document. For convenience of explanation, the file name is used as information uniquely identifying each document. However, other information may be used as long as the information can uniquely identify the document, such as a hash value of the file. Absent.

  Step 130) The character area specifying unit 11 reads each document listed in the list of FIG. 6, and uses the character area extraction method disclosed in Japanese Patent Laid-Open No. 2000-48120 for each page of each document, as shown in FIG. A rectangular area (character area) including each character is specified as described above, and the character area ID, character, book name, page, coordinates, and size of each character area are listed in the data structure shown in FIG. The character area ID is an identifier uniquely indicating each character area provided for convenience of explanation. In the present embodiment, the coordinates are expressed in units of pixels when the upper left corner of the document is the origin, the x axis is in the right direction, and the y axis is in the lower direction. As another implementation method, the coordinates may be represented by rows / columns in each page.

  In this embodiment, the size represents the length of the character area in the x-axis direction and the length in the y-axis direction in units of pixels.

  Step 140) The peripheral area specifying unit 12 reads each record described in the list of FIG. 8, specifies an area (peripheral area) adjacent to the character area represented by each record as shown in FIG. The peripheral area ID, coordinates, size, character area ID, and adjacent direction of the peripheral area are listed in the data structure shown in FIG.

  In the present embodiment, the peripheral area is an area that is adjacent to the character area in one of the upper, lower, left, and right directions, is an area that has a similar shape to the character area, and includes the character area and the peripheral area. The similarity ratio is 1: N (here, N = 1), and is an area that does not include all or part of one or more character areas. As another implementation method, an area including all or a part of one or more character areas may be used as shown in FIG.

  In the present embodiment, the peripheral area ID is an identifier that uniquely indicates each peripheral area provided for convenience of explanation.

  In the present embodiment, the coordinates are expressed in pixel units in the upper left corner of the surrounding rectangular area when the origin is the upper left corner of the document, the x axis is in the right direction, and the y axis is in the lower direction. As another realization method, the coordinates may be represented by rows and columns on each page.

  In the present embodiment, the size represents the length in the x-axis direction and the length in the y-axis direction of the peripheral region in units of pixels.

  In the present embodiment, the character area ID is a character area ID of an adjacent character area.

  Further, in the present embodiment, the adjacent direction represents the direction of the peripheral area adjacent to the character area as viewed from the character area in the vertical and horizontal directions. As another realization method, it may be expressed by how much the line segment connecting the centroid of the character area and the centroid of the peripheral area is inclined from the horizontal direction.

  Step 150) The partial image area specifying unit 13 reads each record described in the list of FIG. 10, and removes the area (part) that is the background area from the peripheral area represented by each record as shown in FIG. Image area) is specified, the partial image area ID, coordinates, size, character area ID, and adjacent direction of each partial image area are listed in the data structure shown in FIG. 13, and the color distribution information of each partial image area is shown in FIG. List with the data structure shown in.

  In the present embodiment, the method described in Non-Patent Document 1 described above is used when removing the background region from the peripheral region and when listing the color distribution information of each partial image region. . As described above, the color distribution information A of white, which is the background color of the document that has been determined in advance, is calculated, and the color distribution information B of each peripheral region is calculated by the same method. The peripheral area is determined as a partial image area, and B is added to the list in the format of FIG. As another realization method, when the color dispersion (for example, the dispersion value of the frequency of the R value, the G value, and the B value) in a peripheral region exceeds a certain value, that is, in a region where a plurality of colors are mixed. May be determined not as a background area but as a partial image area.

  In the present embodiment, the partial image area ID is an identifier that uniquely indicates each partial image area provided for convenience of explanation.

  In the present embodiment, the coordinates, size, character area ID, and adjacent direction are the same as the definitions in the peripheral area.

  Step 160) The index output unit 14 reads each record described in the list of FIGS. 8, 13, and 14 and associates FIG. 8 and FIG. 13 with the character area ID according to the flowchart shown in FIG. FIG. 14 is associated with the partial image area ID, and is output to the index DB 15 as an index having the format shown in FIG.

  Step 161) Read all the character area lists in FIG.

  Step 162) Set the count i = 1.

  Step 163) If i ≦ X, the process proceeds to step 164. If i> X, the process ends.

  Step 164) The following processing is performed.

    Step 1641) All Y records having the same character area ID as the i-th record in the character area list in FIG. 8 are read from the partial image area list in FIG.

    Step 1642) The counter j is set to j = 1.

    Step 1643) If j ≦ Y, the process proceeds to Step 1644, and if j> Y, the process ends.

    Step 1644) All Z records (record group z) having the same partial image area ID as the j-th record in the partial image area list of FIG. 13 are selected from the color distribution information list of the partial image area of FIG. Read.

    Step 1645) The character, book name, page, coordinates of the i-th record in the character area in FIG. 8, the adjacent direction of the j-th record in the list of partial image areas in FIG. 13, and the color distribution information of the partial image area in FIG. The R value, G value, B value, and frequency of the record group z in the list are associated with each other and output to the index DB 14 as one record of the index.

    Step 1646) The counter j is set to j = j + 1, and the process proceeds to Step 1643.

[Second Embodiment]
In the present embodiment, a flow of processing for executing a search by a search device using the index DB generated in the first embodiment is shown.

  FIG. 17 shows the configuration of the search device according to the second embodiment of the present invention.

  The search apparatus shown in FIG. 1 includes a search target image input unit 20, a character region specifying unit 21, a peripheral region specifying unit 22, a partial image region specifying unit 23, an inquiry unit 24, a search result output unit 25, a content DB 26, an index DB 15, The aggregate list storage unit 27 and the handy device 200 are included.

  The index DB 15 is a DB generated in the first embodiment.

  The handy device 200 is a general device having a camera, a display, and a data transmission / reception function. For example, the handy device 200 corresponds to a camera-equipped mobile phone.

  It is assumed that the document is a book of paper medium consisting of a character string, an image, or a set of pages including the character string or image, and the search target image is a part of the document. It is assumed that the background color of the document is known as white.

  In addition, the character area specifying unit 21, the peripheral area specifying unit 22, the partial image area specifying unit 23, and the inquiry unit 24 have a memory (not shown) that temporarily stores information such as a character string in the course of processing. .

  Below, the flowchart of the process which performs the search in this Embodiment is shown.

  FIG. 18 is a flowchart of search processing according to the second embodiment of the present invention.

  In the following, an input step for inputting a search target image to be searched (steps 201 and 203), a character region specifying step for specifying a character region from the search target image (step 204), and a peripheral region from within the search target image. Step of specifying peripheral region (step 205), step of specifying partial image region (step 206) for specifying a partial image region from within the search target image, document matching the search target image, position in the document, and pre-association with the position An inquiry step (steps 207 to 210) for inquiring the requested content and an output step (steps 211 and 212) for outputting the result of the inquiry will be described.

  Step 201) Using the handy device 200, a portion of the page in the document (book) as shown in FIG. 19 is photographed to create a JPEG format file as shown in FIG. The search target image is used.

  It should be noted that the file format may be other than JPEG as long as the captured image and character string information are included.

  In the present embodiment, there is only one search target image, but a plurality of search target images may be prepared.

  Step 202) The search target image input unit 20 receives an input of the search target image from the handy device 200 via the network.

  In addition, you may receive the input of a search object image via storage media, such as an SD card, not via a network.

  In the present embodiment, there is only one search target image. However, when there are a plurality of search target images, they may be received simultaneously.

  Step 203) The search target image input unit 20 assigns a unique file name in the system to the search target images (or a search target image group if there are a plurality of search target images), and lists them in the data structure shown in FIG.

  For convenience of explanation, the file input time is used as the file name uniquely indicating the search target image. However, as long as the information such as a hash value of the file can uniquely identify the search target image. Other information may be used.

  Hereinafter, in the present embodiment, only the case where there is one input search target image will be described. However, when a plurality of search target images are input, a file name that can uniquely distinguish them is assigned. These are added to the list and repeated for each search target image in the subsequent processing.

  Step 204) The character area specifying unit 21 reads each search target image listed in the list of FIG. 21, and uses the character area extraction method such as the above-mentioned Japanese Patent Laid-Open No. 2000-48120 for each search target image. As shown in FIG. 22, a rectangular area (character area) including each character is specified, and the character area ID, character, and size of each character area are listed in the data structure shown in FIG.

  In the present embodiment, the character area ID is an identifier that uniquely indicates each character area provided for convenience of explanation.

  In this embodiment, the size represents the length of the character area in the x-axis direction and the length in the y-axis direction in units of pixels.

  Step 205) The peripheral area specifying unit 22 reads each record listed in the list of FIG. 23, specifies an area (peripheral area) adjacent to the character area represented by each record as shown in FIG. The peripheral area ID, size, character area ID, and adjacent direction of the peripheral area are listed in the data structure shown in FIG.

  In the present embodiment, the peripheral area is an area that is adjacent to the character area in any of the upper, lower, left, and right directions, is an area that is similar to the character area, and is adjacent to the character area. The similarity ratio of the areas is 1: N (N = 1 here), and is an area that does not include all or part of one or more character areas. As another implementation method, as shown in FIG. 26, an area including all or a part of one or more character areas may be used.

  In the present embodiment, the peripheral area ID is an identifier that uniquely indicates each peripheral area provided for convenience of explanation.

  In this embodiment, the size expresses the length in the peripheral region x-axis direction and the length in the y-axis direction in units of pixels.

  In the present embodiment, the character area ID is a character area ID of an adjacent character area.

  Further, in the present embodiment, the adjacent direction represents the direction of the peripheral area adjacent to the character area as viewed from the character area in the vertical and horizontal directions. As another realization method, it may be expressed by how much the line segment connecting the centroid of the character area and the centroid of the peripheral area is inclined from the horizontal direction.

  Step 206) The partial image region specifying unit 23 reads each record described in the list of FIG. 25, and removes the region (part) that is the background region from the peripheral region represented by each record as shown in FIG. 28), the partial image area ID, size, character area ID, and adjacent direction of each partial image area are listed in the data structure shown in FIG. 28, and the color distribution information of each partial image area is shown in FIG. List by data structure.

  In the present embodiment, when the background area is excluded from the peripheral areas, and when the color distribution information of each partial image area is listed, it is the same as step 150 in the first embodiment. The method is used.

  In the present embodiment, the partial image area ID is an identifier that uniquely indicates each partial image area provided for convenience of explanation.

  In the present embodiment, the size, the character area ID, and the adjacent direction are the same as the definitions in the peripheral area.

  Step 207) The inquiry unit 24 reads each record described in the list of the character area data in FIG. 23, the partial image area data in FIG. 28, and the color distribution information in the partial image area in FIG. According to the flowchart, FIG. 23 and FIG. 28 are associated with the character area ID, and FIG. 28 and FIG. 29 are associated with the partial image area ID to create a list in the format shown in FIG.

  FIG. 30 is a flowchart of inquiry processing in the second embodiment of this invention.

    Step 310) Read all X lists of character areas in FIG.

    Step 320) The counter i is set to i = 1.

    Step 330) If i ≦ X, the process proceeds to Step 340, and if i> X, the inquiry process is terminated.

    Step 340) The following processing is performed.

    Step 341) All Y records having the same character area as the i-th record in the character area list in FIG. 23 are read from the partial image area list in FIG.

    Step 342) The counter j is set to j = 1.

    Step 343) If j ≦ Y, the process proceeds to Step 344, and if j> Y, the process ends.

    Step 344) All Z records (record group z) having the partial image area ID of FIG. 28 are read from the color distribution information list of the partial image area of FIG.

    Step 345) The character of the i-th record in the list of character areas in FIG. 23, the adjacent direction of the j-th record in the list of FIG. 28, and the R value of the record group z in the color distribution information list of the partial image area in FIG. , G value, B value, and frequency are associated and output as one record in the search result list of FIG.

    Step 346) Set the counter j to j = j + 1, and proceed to Step 343.

    Step 350) Set the counter i to i = i + 1, and proceed to Step 330.

  Step 208) The inquiry unit 24 makes an inquiry to the index DB 15 having the data structure of FIG. 13 according to the flowchart of FIG. 32, and obtains the search result shown in FIG.

  FIG. 32 is a flowchart of the index DB search in the second embodiment of this invention.

    Step 401) Read all X items in the character area list of FIG.

    Step 402) Set the counter i to i = 1.

    Step 403) If the counter i ≦ X, the process proceeds to step 404. If i> X, the process is terminated.

    Step 404) The name, page, and coordinates of all Y records in the record group having the same character, adjacent direction, R value, G value, B value, and frequency as the i-th record in the character area list of FIG. 16 is read from the index DB 15 and added to the index DB search result list of FIG.

    Step 405) Set the counter i to i = i + 1, and proceed to Step 403.

  Step 209) The inquiry unit 24 reads each record of the search result of the index DB 15 listed in the list of FIG. 33, and aggregates the search results having similar coordinates into groups according to the flowchart of FIG. The number of collected original records (aggregation number) is aggregated, and the list shown in FIG. 35 is output to the aggregation list storage unit 27.

  FIG. 34 is a flowchart of index DB search result aggregation processing according to the second embodiment of this invention.

    Step 510) Read all X search result lists in the index DB 15 of FIG.

    Step 520) The counter i is set to i = 1.

    Step 530) If the counter i ≦ X, go to Step 540, and if i> X, go to Step 560.

    Step 540) The following processing is performed.

    Step 541) It is determined whether there is a record having the same book title / page as the i-th record in FIG. 33 and having coordinates close to each other (for example, within 20 pixels in the x-axis direction and the y-axis direction). If yes, then go to Step 542; if not, go to Step 543.

    Step 542) The i-th record in the list of FIG. 33 is aggregated to the aggregation destination record having the closest coordinates (when there are a plurality of coordinates, the coordinates are closest to the origin), and the aggregation number of the aggregation destination record is increased by one. The process proceeds to step 550.

    Step 543) The i-th record in the list of FIG. 33 is set as a new aggregation destination record, the aggregation number of the aggregation destination record is set to 1, and stored in a memory (not shown) together with the title, page, and coordinates. 550.

    Step 550) The counter i is set to i = i + 1, and the process proceeds to Step 530.

    Step 560) In step 530, if i> X, the list, page, coordinates, and number of aggregations of the aggregation destination records stored in the memory (not shown) are used as the list of FIG. Output to.

  In the present embodiment, it is determined that the coordinates are close if the coordinates are within a range of 20 pixels in the x-axis direction and the y-axis direction.

  Step 210) The inquiry unit 24 reads the record having the largest aggregation number in the list of the aggregation list storage unit 27 in FIG. 35, and stores it in the content DB 26 having the data structure in FIG. 36 created in advance by the system operator or the like. On the other hand, in accordance with the flowchart of FIG. 37, the content associated with the position whose coordinates are close to the record is inquired, and the obtained content is passed to the search result output unit 25.

  FIG. 37 is a flowchart of content DB search processing according to the second embodiment of the present invention.

    Step 601) Read the record A having the maximum number of aggregations in the list of the aggregation list storage unit 27 in FIG.

    Step 602) If the content DB 26 includes the record B having the same book title / page as the record A, the process proceeds to step 603. Otherwise, the process proceeds to step 605.

    Step 603) If the coordinates of record A and record B are close (within 20 pixels in the x-axis direction and y-axis direction), the process proceeds to step 604, and if not, the process proceeds to step 605.

    Step 604) The contents of record B are output.

    Step 605) Determine “no content”.

  In the present embodiment, if the coordinates are within 20 pixels in the x-axis direction and the y-axis direction, it is determined that the coordinates are close.

  Step 211) If the content is received from the inquiry unit 24, the search result output unit 25 transmits the content to the handy device 200 via the network. If the inquiry unit 24 determines “no content”, nothing is handed to the handy device 200.

  Note that when the content is transferred from the search result output unit 25 to the handy device 200, the content may be transferred via a storage medium such as an SD card, not via a network.

  Step 212) The handy device 200 displays the content delivered from the search result output unit 25 by a predetermined means.

  In the present embodiment, when a content whose extension ends with html is passed, the content is displayed by a Web browser built in the handy device 200.

  The operation of each component shown in FIG. 3 and FIG. 17 is constructed as a program and can be installed and executed in a computer used as an index creation unit and a search device, or distributed via a network. Is possible.

  Further, the constructed program can be stored in a portable storage medium such as a hard disk, a flexible disk, or a CD-ROM, and can be installed or distributed in a computer.

  The present invention is not limited to the above-described embodiments and examples, and various modifications and applications can be made within the scope of the claims.

DESCRIPTION OF SYMBOLS 10 Document input part 11 Character area specific part 12 Peripheral area specific part 13 Partial image area specific part 14 Index output part 15 Index DB
20 Search target image input unit 21 Character region specifying unit 22 Peripheral region specifying unit 23 Partial image region specifying unit 24 Inquiry unit 25 Search result output unit 26 Content DB
27 Aggregation list storage

Claims (9)

A page break or a line feed position has been determined, and a search request for acquiring a document in which the area appears and a position in the document using a partial area including the character string as a search query in the document including the character string is satisfied. A search device for creating a search index for performing a search,
A document group input means for accepting input of a document group to be indexed;
A character region specifying means for specifying a character region in the document using a character region extraction method;
A peripheral area specifying means for specifying a peripheral area which is an area existing around each character area;
A partial image specifying means for comparing each peripheral area with a background area of the document using an image feature comparison method, and determining a partial image area that is different from the background area among the peripheral areas;
Index output means for associating the character area, the partial image area around the character area, and the appearance position in the document in which the character string appears, and outputting the associated position to the index DB;
A search device comprising: The document also includes an image,
The search device according to claim 1, wherein a partial area including an image and a character string in the document is used as a search query. A search target image input means for receiving an input of an image obtained by photographing a partial area of a document including an image and a character string;
Search target character area specifying means for specifying a character area in the search target image using a character area extraction method;
Search target peripheral area specifying means for specifying a peripheral area that is an area existing around each character area;
A search target partial image specifying unit that compares each peripheral region with a background region of the search target image using an image specific comparison method, and determines a partial image region that is different from the background region among the peripheral regions;
Inquiry means for inquiring the index DB for the character area and a document in which the partial image area around the character area appears or a position in the document;
Search result output means for outputting the result of the inquiry;
The search device according to claim 1, further comprising: The inquiry means includes
The search device according to claim 3, further comprising means for accessing a database in which the position and content of each document are associated in advance and obtaining the content associated with the document position as a search result as the search result. A page break or a line feed position has been determined, and a search request for acquiring a document in which the area appears and a position in the document using a partial area including the character string as a search query in the document including the character string is satisfied. A search method for a device that creates a search index and performs a search,
A document group input step in which the document group input means accepts input of a document group to be indexed;
A character region specifying means for specifying a character region in the document using a character region extraction method;
A peripheral area specifying step for specifying a peripheral area that is an area existing around each of the character areas;
A partial image specifying unit compares each peripheral area with the background area of the document using an image feature comparison method, and determines a partial image area that is different from the background area as the partial image area. Steps,
An index output step, wherein the index output means associates the character area, the partial image area around the character area, and the appearance position in the document in which the character string appears, and outputs the associated position to the index DB;
The search method characterized by performing. The document also includes an image,
The search method according to claim 5, wherein a partial area including an image and a character string in the document is used as a search query. A search target image input unit for receiving an input of an image obtained by photographing a partial region of a document including an image and a character string;
A search target character region specifying unit, wherein a character region in the search target image is specified using a character region extraction method;
A search target peripheral region specifying unit for specifying a peripheral region that is a region existing around each character region;
A search target partial image specifying unit uses an image specific comparison method to compare each peripheral region with a background region of the search target image, and to determine a peripheral image different from the background region as a partial image region A target partial image identification step;
Inquiry means for inquiring the index DB about the character area and a document in which the partial image area around the character area appears or a position in the document,
A search result output means for outputting a result of the inquiry;
The search method according to claim 5 or 6, further comprising: In the inquiry step,
8. The search method according to claim 7, further comprising the step of accessing a database in which the position and content of each document are associated in advance and obtaining the content associated with the document position as a search result as the search result.   The search program for functioning a computer as each means which comprises the search device of any one of Claims 1 thru | or 4.

Images