JP5315935B2 - Image search apparatus, image search method, control program, and recording medium - Google Patents

Description

  The present invention relates to an image search device, an image search method, a control program, and a recording medium, and more particularly to a search method in image search.

  Search technology for electronic data or search result display technology has become an increasingly important technology because of the increase in the number of search results due to an increase in the amount of information to be searched. This is because the information that is sought is buried in a large amount of search results, making it difficult to find. As such a search technique, an image search method for inputting an image as a search image and extracting an image similar to the input image from an already stored search image has been proposed (for example, Patent Documents). 1).

An image search technique such as that disclosed in Patent Document 1 is used, for example, for searching published patent publications. For example, when searching for a drawing included in a document such as a published patent publication, the drawing is set as a search key, and a document including a drawing similar to the drawing is extracted. In addition, in the case of searching for image information using image information as a search key in a database storing a plurality of image information such as trademark search, it can be used in the same manner.
JP 2002-245087 A

  In a conventional image search as disclosed in Patent Document 1, the entire image set as a search key (hereinafter referred to as a key image) and the entire image to be searched (hereinafter referred to as a search target image) are similar. It is assumed that the degree is judged. For this reason, even if the search target image includes an image similar to the key image, the degree of similarity is not calculated high in relation to the whole image.

  On the other hand, the inventor of the present application invented a method for extracting a plurality of regions such as a document region, a photographic region, a table region, and the like from an image and matching the images for each region, and filed an application (special feature). Application No. 2007-022108). In the above method, numerical sequence data such as a histogram is generated for each extracted region, and image matching is performed based on the numerical sequence data. Thereby, an image search can be performed for a part of the key image and the search target image. Such matching by a part of an image is particularly effective in trademark search or the like.

  However, in the image search, even images having similar shapes may not be extracted as similar images if they are rotated. In order to solve such a problem, it is performed to calculate the similarity after determining the orientation of the image to be searched and extracted, but depending on the shape of the image, it is difficult to determine the orientation, There are cases where extraction of similar images cannot be performed accurately.

  The present invention has been made in consideration of the above circumstances, and an object thereof is to provide an image search system that can accurately extract a rotated image as a similar image.

In order to solve the above-described problem, the invention according to claim 1 is an image search device that performs an image search by calculating a similarity between an image to be searched and an image of the search condition. an image acquisition unit for acquiring condition image is, the a contour extraction unit for extracting a contour of the obtained condition image, and the central calculation unit for calculating a center of the extracted contour, from the calculated center Based on the distance calculation unit that calculates the distance to the contour every time from 0 ° to 359 ° over the entire circumference of the center and generates the numerical value string information of the distance, and the generated numerical value string information The similarity is obtained by performing a character string search based on the character string information generating unit that generates character string information, and the generated character string information and the search target character string information generated for the search target image. Similarity calculation unit to calculate and Have a, the similarity calculation unit, the reverse character string information corresponding to the periphery of the center numerical sequence information calculated for each one time of 0 ° from 359 ° counterclockwise, is calculated clockwise Character string information corresponding to numerical string information, and performing the character string search based on,

In the image search device according to claim 1, the character string information generation unit associates a numerical value corresponding to the numerical value string information with a character constituting the character string information. The character string information is generated by converting the numerical string information into character string information with reference to the converted correspondence information.

  The invention described in claim 3 is the image search device according to claim 1 or 2, further comprising a contour emphasizing unit that emphasizes the contour of the acquired condition image, wherein the contour extracting unit includes the contour extracting unit, The contour emphasized by the contour emphasizing unit is extracted.

  According to a fourth aspect of the present invention, in the image search device according to any one of the first to third aspects, the contour extracting unit extracts an outermost contour from the contours of the condition image. To do.

The invention according to claim 5 is the image search device according to any one of claims 1 to 4 , wherein the character string information generation unit is smoothed based on the generated numerical string information. The character string information is generated based on numeric string information.

According to a sixth aspect of the present invention, in the image search device according to any one of the first to fifth aspects, the character string information generation unit is normalized based on the generated numerical string information The character string information is generated based on numeric string information.

The invention described in claim 7 is the image search device according to any one of claims 1 to 6 , wherein the similarity calculation unit executes the character string search by N-gram.

According to an eighth aspect of the present invention, in the image search device according to any one of the first to sixth aspects, the similarity calculation unit executes the character string search using a suffix array.

The invention according to claim 9 is the image search device according to any one of claims 1 to 8 , further comprising a search target image storage unit storing the search target image, and the search target image. The storage unit stores search target character string information generated for the search target image.

The invention according to claim 10 is an image search method for executing an image search by calculating a similarity between an image to be searched and an image of the search condition, wherein the image acquisition unit includes an image of the search condition. get the condition image is, the contour extraction unit extracts the outline of the obtained condition image, the center calculation unit, a center of the extracted contour is calculated, the distance calculation unit were the calculated The distance from the center to the contour is calculated every time from 0 ° to 359 ° around the circumference of the center to generate the numerical value string information of the distance, and the character string information generation unit Character string information is generated based on the numerical string information, and the similarity calculation unit performs a character string search based on the generated character string information and the search target character string information generated for the search target image. To calculate the similarity and calculate the similarity Is a reverse character string information corresponding to numerical value string information calculated every time from 359 ° to 0 ° counterclockwise around the center, and a character string corresponding to the numerical value string information calculated clockwise. The character string search is executed based on the information .

The invention according to claim 11 is a control program for causing an information processing apparatus to operate as an image apparatus that calculates a similarity between an image to be searched and an image of a search condition and executes an image search. , acquiring a condition image is an image of the search condition, extracting the contour of the obtained condition image, calculating a center of the extracted contour, said from the calculated center A step of calculating a distance to the contour every time from 0 ° to 359 ° around the circumference of the center to generate numerical sequence information of the distance; and a character string based on the generated numerical sequence information The similarity is calculated by performing a character string search based on the step of generating information and the generated character string information and the search target character string information generated for the search target image. With that, the opposite character string information corresponding to the periphery of the center numerical sequence information calculated for each one time of 0 ° from 359 ° counterclockwise, the string corresponding to the numeric string information calculated clockwise And causing the information processing apparatus to execute the step of executing the character string search based on the information.

The invention according to claim 12 is a recording medium, wherein the control program according to claim 11 is recorded in a format readable by the information processing apparatus.

  According to one embodiment of the present invention, it is possible to provide an image search system that can accurately extract even a rotated image as a similar image.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the present embodiment, as an example of an image search apparatus, an apparatus that searches search target image information stored in advance based on an input image and extracts an image similar to the input image will be described as an example. .

  FIG. 1 is a diagram illustrating an example of an operation mode of an image search system according to the present embodiment. As shown in FIG. 1, the image search system according to the present embodiment includes an image search device 1, a client device 2, and an image information DB 200. The image search device 1, the client device 2, and the image information DB 200 are connected via the network 3.

  The client device 2 is configured by a general information processing device such as a PC (Personal Computer). The client device 2 can be configured by a mobile communication terminal such as a mobile phone, an MFP (Multi Function Peripheral), or the like. The image search device 1 is connected to the client device 2 via a network, and searches for image information stored in the image information DB 200 in response to a search request from the client device 2. The image information DB 200 stores a plurality of pieces of image information as search target information. That is, the image information DB 200 is a search target image storage unit. The image information DB 200 is configured by a nonvolatile storage medium such as an HDD (Hard Disk Drive).

  As shown in FIG. 1, in this embodiment, an example in which the image information DB 200 is provided separately from the image search device 1 will be described. However, the image information DB 200 may be configured inside the image search device 1. Is possible. Furthermore, the image information DB 200, the image search device 1, and the client device 2 may be configured as a single device.

  Next, a hardware configuration of an information processing terminal such as the image search device 1 or the client device 2 according to the present embodiment will be described. FIG. 2 is a block diagram illustrating a hardware configuration of the information processing terminal according to the present embodiment. In the following description, the hardware configuration of the image search apparatus 1 will be described. As shown in FIG. 2, the image search apparatus 1 according to the present embodiment has the same configuration as an information processing terminal such as a general server or a PC (Personal Computer).

  That is, in the image search apparatus 1 according to the present embodiment, a CPU (Central Processing Unit) 10, a RAM (Random Access Memory) 20, a ROM (Read Only Memory) 30, an HDD 40, and an I / F 50 are connected via a bus 80. ing. Further, an LCD (Liquid Crystal Display) 60 and an operation unit 70 are connected to the I / F 50.

  The CPU 10 is a calculation means and controls the operation of the entire image search apparatus 1. The RAM 20 is a volatile storage medium capable of reading and writing information at high speed, and is used as a work area when the CPU 10 processes information. The ROM 30 is a read-only nonvolatile storage medium and stores a program such as firmware. The HDD 40 is a non-volatile storage medium that can read and write information, and stores an OS (Operating System), various control programs, application programs, and the like.

  The I / F 50 connects and controls the bus 80 and various hardware and networks. The LCD 60 is a visual user interface for the user to check the state of the image search device 1. The operation unit 70 is a user interface for the user to input information to the image search apparatus 1 such as a keyboard and a mouse.

  As described with reference to FIG. 1, the image search apparatus 1 according to the present embodiment is operated as a server. Therefore, user interfaces such as the LCD 60 and the operation unit 70 can be omitted. In addition, as long as the image search device 1 is a terminal having an information processing function, any realization mode may be used. For example, a general PC, MFP, etc. can be considered.

  In such a hardware configuration, a program stored in a storage medium such as the ROM 30, the HDD 40, or an optical disk (not shown) is read into the RAM 20, and operates according to the control of the CPU 10, thereby configuring a software control unit. A functional block for realizing the functions of the image search apparatus 1 according to the present embodiment is configured by a combination of the software control unit configured in this way and hardware.

  Next, functional blocks of the image search device 1 according to the present embodiment will be described with reference to FIG. FIG. 3 is a block diagram showing functional blocks of the image search apparatus 1 according to the present embodiment. As illustrated in FIG. 3, the image search apparatus 1 according to the present embodiment includes a search control unit 100, an information input unit 110, a network I / F 120, and a display unit 130.

  The information input unit 110 is configured to allow a user to operate the image search apparatus 1 and input information to the search control unit 100, and is realized by the I / F 50 and the operation unit 70 illustrated in FIG. The network I / F 120 is an interface for the image search apparatus 1 to acquire information via the network or transmit information via the network, and is realized by the I / F 50 illustrated in FIG. As a specific example, the network I / F 120 is realized by an Ethernet (registered trademark) connection interface or a USB (Universal Serial Bus) connection interface.

  The display unit 130 is configured to display the operation state of the image search apparatus 1, search results, and the like, and is realized by the I / F 50 and the LCD 60 shown in FIG. The search control unit 100 is an image search execution unit that performs a search function of the image search apparatus 1 according to the present embodiment, and includes an input image information acquisition unit 101, an image processing unit 102, a character information conversion unit 103, and a similarity calculation unit 104. And an extraction result processing unit 105.

  The input image information acquisition unit 101 uses the image information input by the user via the information input unit 110 or the image information input via the network via the network I / F 120 as input image information, that is, a condition image that serves as a search condition. Functions as an image acquisition unit. The image acquisition unit is configured by a program loaded in the RAM 20 shown in FIG.

  The image processing unit 102 performs predetermined processing on the condition image acquired by the input image information acquisition unit 101, and generates numerical string information. The processing executed by the image processing unit 102 will be described in detail later. The character information conversion unit 103 converts the numerical string information generated by the image processing unit 102 into character string information and generates character string information.

  The similarity calculation unit 104 calculates the similarity between the condition image and the search target image based on the character information generated by the character information conversion unit 103. Then, the similarity calculation unit 104 generates information (hereinafter referred to as similarity list information) in which the search target image and the similarity of the image to the condition image are associated with each other. The similarity calculation unit 104 is configured by a program loaded in the RAM 20 illustrated in FIG. 2 operating according to the control of the CPU 10. An existing image search technique can be used as the similarity calculation method by the similarity calculation unit 104.

  The extraction result processing unit 105 displays the similarity list information generated by the similarity calculation unit 104 on the display unit 130 or generates and outputs display information for display on the display unit of the client device 2. That is, the extraction result processing unit 105 functions as a display information generation unit that generates display information for displaying the calculation result of the similarity. The display information generation unit is configured by a program loaded in the RAM 20 shown in FIG.

  Next, information stored in the image information DB 200 will be described. FIG. 4 shows an example of information stored in the image information DB 200. As described above, the image information DB 200 stores image information to be searched. As shown in FIG. 4, the image information DB 200 stores the ID, image title, and character string information of the search target image. The image ID is information for uniquely identifying an image to be searched. The image title is the name of the image to be searched. The character string information is character string information generated by processing similar to the processing by the image processing unit 102 and the character information conversion unit 103 described above for each image information. This character string information is search target character string information used in image search.

  In such an image search apparatus 1, the main point of the present embodiment is processing by the image processing unit 102 and the character information conversion unit 103. Hereinafter, the operation of the image search apparatus 1 according to the present embodiment will be described with reference to the drawings. FIG. 5 is a sequence diagram showing a search operation in the image search system according to the present embodiment.

  As shown in FIG. 5, when searching for image information registered in the image information DB 200, first, the user operates the client device 2 to display information for displaying a search image input screen in the image search device 1. A request is made (S501). In response to a request from the client device 2, the image search device 1 transmits information for displaying a search image input screen (S502).

  Upon acquiring the search image input screen from the image search device 1, the client device 2 displays the search image input screen (S503). Here, the search image input screen is a screen for inputting search image information. Hereinafter, in the description of the present embodiment, a case where the user operates the client device 2 and uses the function of the image search device 1 will be described as an example.

  The client device 2 transmits information on the input image (hereinafter referred to as input image information) input via the search image input screen to the image search device 1 according to the operation by the user (S504). An example of input image information input in the present embodiment is shown in FIG. The input image information transmitted to the image search device 1 is input from the network I / F 120 to the image search device 1 and acquired by the input image information acquisition unit 101 of the search control unit 100 (S505). The input image information acquired by the image search device 1 in S505 is used as a condition image. That is, the input image information acquisition unit 101 functions as an image acquisition unit that acquires a condition image. The input image information acquisition unit 101 inputs the acquired input image information to the image processing unit 102.

  When acquiring the input image information as the condition image, the image processing unit 102 executes predetermined image processing on the acquired image information (S506). The processing by the image processing unit 102 in S506 will be described in detail with reference to FIG. FIG. 7 is a flowchart showing the processing of the image processing unit 102 in S506.

  As shown in FIG. 7, when the image processing unit 102 acquires the condition image (S701), the image processing unit 102 executes a process of enhancing the contour of the condition image (S702). That is, the image processing unit 102 functions as an outline emphasizing unit. For the contour enhancement processing in S702, an existing method can be used, and detailed description thereof is omitted. An example of an image generated as a result of the contour enhancement processing in S702 is shown in FIG. As shown in FIG. 8A, as a result of the contour enhancement processing in S702, an image in which the contour of the image in FIG. 6 is enhanced is generated.

  Then, the image processing unit 102 extracts the outermost contour (hereinafter referred to as the outermost contour) of the image whose contour is emphasized (S703). That is, the image processing unit 102 functions as a contour extraction unit. For the outermost contour extraction process of S703, an existing method can be used, and detailed description thereof is omitted. FIG. 8B shows an example of an image generated as a result of the outermost contour extraction process in S703.

  As shown in FIG. 8B, the outermost contour in the image of FIG. 8A is extracted as a result of the processing of S703. In the outermost contour extraction process of S703, there are cases where the contour cannot be accurately extracted, for example, when the contour of the original image is faint. On the other hand, by executing the contour emphasis process of S702, the outermost contour extraction process can be accurately executed even for a drawing with a faint outline.

  Then, the image processing unit 102 extracts the center point of the image generated as the outermost contour and the image direction (hereinafter referred to as image direction) by calculation as shown in FIG. 8B (S704). That is, the image processing unit 102 functions as a center calculation unit. In the processing of S704 according to the present embodiment, the image processing unit 102 extracts the center point by calculating the center of gravity of the image shown in FIG. Since a known method can be used for the calculation method of the center of gravity, detailed description is omitted.

  Further, the image processing unit 102 extracts the direction of the image with the widest image direction shown in FIG. 8B as the vertical direction and the direction orthogonal to the vertical direction as the horizontal direction (S704). As a result of the processing of S704, the center point and the image direction are extracted as shown in FIG.

  When the center point and the image direction are extracted, the image processing unit 102 sets the distance from the extracted center point to the outermost contour (hereinafter referred to as the contour distance) over the entire circumference of the center point, that is, 360 degrees. (S705). That is, the image processing unit 102 functions as a distance calculation unit. FIG. 8D shows how the contour distance is calculated in S705. As indicated by a broken line in FIG. 8D, in S705, the distance between the center point and the outermost contour is calculated in an arbitrary direction from the center point. When calculating the contour distance, the distance between the coordinates of the point where the straight line drawn in an arbitrary direction from the center point and the outermost contour intersect with the coordinates of the center point is calculated.

  In S705 according to the present embodiment, as illustrated in FIG. 8D, the image processing unit 102 sets the vertical direction of the image to 0 ° and 180 °, and contours every 0 ° from 0 ° to 359 °. Calculate the distance. When the calculation of the contour distance is completed for all angles, the image processing unit 102 generates numerical value string information in which the angle and the contour distance are associated as shown in FIG. 9 (S706).

  In the example of FIG. 9, the information in which the angle and the contour distance are associated is the numerical string information. However, “XX”, “▽▽”, “◇◇”... In this way, information in which only the contour distances are arranged in the order of angles may be used as the numerical string information. This numeric string information can be said to be information obtained by digitizing the shape of the image shown in FIG. 6, that is, the condition image.

  When the numerical sequence information is generated by the processing of S706, the image processing unit 102 removes, that is, down-samples, high-frequency components of the numerical sequence information (S707). In other words, the image processing unit 102 smoothes the numerical value string information. In the present embodiment, as described above, the numerical value string information generated as a result of S706 includes 360 numerical values from 0 ° to 359 °, and thus has one value per 1 °. In the present embodiment, the numerical value string information is down-sampled so as to have one value per 4 ° and to be composed of a total of 90 values by the processing of S707.

  FIG. 10 shows a mode of processing by the image processing unit 102 in S707. As shown in FIG. 10, in the processing of S707, the image processing unit 102 groups each numerical value included in the numerical value string information every four, calculates an average value of the numerical values, and sets new numerical value string information and To do. In downsampling, various known methods can be used in addition to the embodiment shown in FIG. By such processing, the removal of the high-frequency component is completed, and down-sampled numerical sequence information (hereinafter referred to as down-sampling information) is generated. The down-sampling information can also be referred to as smoothed numerical value string information.

  When the removal of the high frequency component is completed, the image processing unit 102 normalizes the numerical value of the generated down-sampling information (S708). In the processing of S708 according to the present embodiment, the image processing unit 102 divides each numerical value included in the down-sampling information generated as shown in FIG. 10 by the largest numerical value among the included numerical values, and the result Normalization is performed by quantizing the calculated numerical value. In addition, various known methods can be used in the normalization process of S708. In the present embodiment, the numerical values included in the down-sampling information are converted into 8-level numerical values by the quantization. The effect of normalization will be described later.

  By the processing of S708, each numerical value included in the down-sampling information becomes a normalized normalized numerical value. Information generated in this way (hereinafter referred to as normalized numerical sequence information) can be visually displayed as a histogram. FIG. 11 is a diagram showing a histogram of normalized numerical sequence information generated as a result of the processing of FIG. The histogram shown in FIG. 11 is a histogram showing the shape of the image shown in FIG. 6, that is, the condition image. Since the histogram shown in FIG. 11 is normalized, it is dimensionless and is a histogram showing the shape of the condition image regardless of the size of the condition image. By such processing, the image processing in S506 of FIG. 5 is completed.

  When the image processing is completed in S506, the character information conversion unit 103 converts the generated normalized numerical string information to generate character string information (S507). That is, the character information conversion unit 103 functions as a character string information generation unit. In step S507, the character information conversion unit 103 performs character information conversion with reference to information (hereinafter referred to as conversion correspondence information) in which the normalized numerical value and the character are associated with each other. FIG. 12 shows an example of conversion correspondence information.

  As described above, the normalized numerical value string information according to the present embodiment is configured by numerical values in eight stages from “1” to “8”. As shown in FIG. 12, the conversion correspondence information is information in which the numerical values “1” to “8” are associated with the characters “a” to “h”, respectively. The character information conversion unit 103 refers to the conversion correspondence information as shown in FIG. 12, converts the normalized numerical value into a corresponding character, and generates character string information from the numerical string information. In the present embodiment, the conversion correspondence information is held by the character information conversion unit 103.

  An example of character string information generated by the processing of S507 is shown in FIG. As shown in FIG. 13, the pieces of character information are connected by the processing of S507, and continuous character string information is generated.

  Note that the character string information described in FIG. 4 (hereinafter referred to as accumulated character string information) is information corresponding to the character string information illustrated in FIG. 13 (hereinafter referred to as conditional character string information). That is, the stored character string information is also character string information generated through the processing described in FIG. 7 based on each image, similarly to the conditional character string information.

  FIG. 14A to FIG. 14D are diagrams showing how the accumulated character string information is generated, and correspond to FIG. In the example of FIG. 14A to FIG. 14D, an image displayed by tilting the graphic displayed in the image of FIG. 6 is taken as an example.

  As shown in FIG. 14 (a), contour emphasis is first performed when the accumulated character string information is generated. Then, as shown in FIG. 14B, the outermost contour is extracted. Then, as shown in FIG. 14C, the center point and the orientation of the image are extracted. Then, as shown in FIG. 14D, the contour distance is calculated.

  Numerical value string information is generated by the process of FIG. 14D, and normalized numerical value string information is generated through down-sampling and numerical value normalization in the same manner as the process described in FIG. Further, based on the conversion correspondence information described in FIG. 12, character information as shown in FIG. 15 is generated as accumulated character string information.

  When the process of S507 is completed, the similarity calculation unit 104 calculates the similarity between the condition image and the image stored in the image information DB 200 based on the character string information generated in the process of S507 (S508). In the processing of S508, the similarity calculation unit 104 acquires the stored character string information of the stored image from the image information DB 200 via the network I / F 120, and calculates the degree of coincidence with the conditional character string information.

  When calculating the degree of coincidence between the conditional character string information and the accumulated character string information, the similarity calculation unit 104 according to the present embodiment performs a character search by N-gram. FIG. 16 shows a character search mode by N-gram according to the present embodiment. FIG. 16 is a diagram illustrating a case where a character search by 3 (tri) -gram is executed as a character search by N-gram in the character string of FIG. 13 and the character string of FIG.

  3-gram is a character indicating the shape of the outermost contour corresponding to three normalized numerical values, that is, the shape of the outermost contour of 12 ° from the predetermined angle of the center point shown in FIG. Is a column. That is, when a character string search by 3-gram is executed, as shown by a thick line in FIG. 17, the outermost contour shape of 12 ° from a predetermined angle of the center point is shown in FIGS. Matching with the shape of the entire outermost contour of the image to be searched as shown in FIG.

  Note that, as described above, since the down-sampled numerical string information is used in the conversion to the character string information, N-gram indicates the shape of the input image itself. As a result, image noise can be removed, and an image whose shape roughly matches can be extracted as a similar image.

  As described above, the original images in FIGS. 14A to 14D are images obtained by rotating the image shown in FIG. 6 and have the same shape. Therefore, as shown in FIG. 16, the character string of FIG. 13 and the character string of FIG. 15 match many character strings, and a high similarity is calculated. In addition, in the similarity calculation processing in S508, various known character searches such as a character search method by lcp (longest common prefix) using a suffix array (suffix array) in addition to character search by N-gram. It is possible to use a method.

  The similarity calculation unit 104 performs a character search as shown in FIG. 16 for all the image information stored in the image information DB 200, and the image ID shown in FIG. 4 and the similarity calculated for the image are obtained. Generate related similarity list information.

  When the similarity calculation unit 104 generates the similarity list information through the process of S508, the similarity calculation unit 104 extracts an image having a similarity exceeding a predetermined threshold as a similar image (S509). The threshold value is a value set to determine whether the image to be searched, that is, the accumulated image stored in the image information DB 200, is a similar image based on the calculated similarity. is there. That is, this is information for determining whether or not an image stored in the image information DB 200 corresponds to a similar image. In other words, the value is set to exclude an image whose characteristics do not match the condition image from the extraction result.

  The threshold is set according to the similarity calculation method in S508. For example, in the similarity calculation processing in step S508, an image whose shape does not match is calculated as “0%”, and an image whose shape matches even a little is calculated as “0%”. When the calculation mode is used, the threshold value is set as “0%”.

  In addition, the threshold value may be set to a value other than “0%” depending on the similarity calculation method in S508. Further, the threshold value may be set in order to extract an image having a high degree of coincidence between the condition image and the shape, in addition to setting an image whose shape does not match the condition image from the extraction result. In this case, the above threshold is set to a higher value. The similarity calculation unit 104 inputs the image extracted by the processing of S509, that is, similarity list information including similar images (hereinafter referred to as extraction result information) to the extraction result processing unit 105.

  When obtaining the extraction result information, the extraction result processing unit 105 determines the display order of the extracted similar images. Then, the extraction result processing unit 105 generates display information for displaying the extraction result, and transmits the display information to the client device 2 via the network I / F 120 (S510). In S510, the image search device 1 determines the order in which similar images are displayed in the order of the calculated similarity. The client device 2 displays the extraction result on the display unit based on the display information received from the image search device 1 (S511), and ends the process.

  As described above, the image search device 1 according to the present embodiment generates the numerical sequence information by obtaining the distance from the center point of the image to the outermost contour in all directions, and the generated numerical sequence information is normalized. The character string information is converted into character string information, and a similar image is searched by performing a character search based on the converted character string information. Therefore, as described with reference to FIG. 17, by performing a character search using N-gram or lcp, it is possible to determine the similarity of the shape for each portion of the contour shape of the image. As a result, it is possible to calculate an appropriate similarity based on the shape of the image even when the image direction extraction shown in FIGS. 8C and 14C is incorrect. That is, it is possible to provide an image search device that can extract a rotated image accurately as a similar image.

  In the above-described embodiment, as described with reference to FIG. 9, numerical sequence information is generated by arranging the contour distances in the order of 0 ° to 359 °, and after down sampling processing, normalization processing, and character conversion processing, FIG. An example of generating conditional character string information as shown in FIG. In other words, in the above-described embodiment, the example in which the character search is executed based on the character string information corresponding to the numerical value string information calculated clockwise around the calculated center has been described.

  In addition, it is also preferable to generate conditional character string information and use it as a search condition for numerical value string information generated by arranging contour distances in the order of 359 ° to 0 °. In other words, it is preferable to perform a character search based on character string information corresponding to numerical value string information calculated counterclockwise around the calculated center. As a result, a high similarity is also calculated for an image that is asymmetric and that is horizontally reversed, and can be easily extracted as a similar image.

  The conditional character string information for the numerical string information generated by arranging the contour distances in the order of 359 ° to 0 ° described above is obtained by reversing the character order of the generated character string information as shown in FIG. Can also be generated. Thus, the character string information generated by reversing the character order can be referred to as reverse character string information. In addition, character string information is generated based on the numerical string information generated in S706 and the numerical string information obtained by reversing the numerical order of the numerical string information (hereinafter referred to as inverse numerical string information). Also good.

  Next, a screen displayed on the client device 2 in S511, that is, a GUI (Graphical User Interface) of the image search system according to the present embodiment will be described. FIG. 18 is a diagram showing a search result display screen in the GUI displayed on the display unit such as the LCD 60 of the client apparatus 2 in S511. As shown in FIG. 18, the search result display screen according to the present embodiment includes a search target designation unit 181, a condition image display unit 182, a re-search instruction unit 183, an image editing screen display instruction unit 184, and a similar image display unit 185. Have. The similar image display unit 185 displays a plurality of similar images, and a feedback image designating unit 186 is provided for each image.

  The search target designation unit 181 is a part that designates whether each of the accumulated images stored in the image information DB 200 is a search target or a part of the stored image is a search target. The user operates the user I / F such as a mouse, and switches between the entire search and the partial search by operating the toggle displayed next to the characters “overall search” and “partial search”.

  The condition image display unit 182 is a display unit that displays the input image information input in S504 as a condition image. In the example of FIG. 18, the image shown in FIG. 6 is displayed. The check box displayed on the upper left of the condition image will be described later. The re-search instruction unit 183 is a part for instructing execution of another search after changing the search condition, and is mainly used for executing the feedback search. The user operates the mouse or the like and clicks the re-search instruction unit 183 to execute the re-search.

  The image editing screen display instruction unit 184 is a part for the user to request display of an image editing screen which is a GUI for editing a condition image. The image editing screen will be described in detail later. The similar image display unit 185 is a part where a similar image extracted as a result of the processing of S509 is displayed. As described above, in the present embodiment, the similar images are displayed in the order of the similarity calculated in S508. In the similar image display unit 185, the similarity of each similar image is displayed by a number and a bar length.

  The feedback image designating unit 186 is a check box for designating a feedback image when the re-search instruction unit 183 is clicked to perform a feedback search. When an image extracted as a similar image is added to the search condition and further search is performed, a feedback search is performed by checking the feedback image specifying unit 168 of the image to be added to the search condition and clicking the re-search instruction unit 183. Is executed.

  For example, FIG. 19 shows a search result display screen when the images “B” and “F” shown in FIG. 18 are designated as feedback images and a feedback search is executed. As shown in FIG. 19, when a feedback search is executed, an image designated as a feedback image is added to the condition image display unit 182. In the screen shown in FIG. 19, by clearing the check box displayed at the upper left of each condition image and executing a re-search, the image with the check removed can be deleted from the condition image.

  When executing the feedback search, the client device 2 first transmits the image ID designated as the feedback image to the image search device 1 via the network. When the ID of the image specified as the feedback image is acquired, the image information of the specified ID is acquired as a condition image from the image information DB 200. Then, the processing after S506 in FIG. 5 is executed using the original image and the feedback image as condition images. In addition, various known methods can be used for executing the feedback search.

  Next, FIG. 20 shows an example of a search result display screen when a partial image search is executed by specifying the search target specifying unit 181. As shown in FIG. 20, when a partial image search is executed, a portion determined to be similar (hereinafter referred to as a similar portion) among images extracted as similar images is highlighted and displayed. In the present embodiment, the similar part is shown surrounded by a broken line, but other methods such as a red line frame may be used. It should be noted that various known methods can be used when executing a similar image search, and detailed description thereof is omitted. As described above, by highlighting and displaying the similar portion, the user can easily grasp the portion determined as the portion similar to the condition image.

  Next, an example in which the image editing screen is displayed by operating the image editing screen display instruction unit 184 is shown in FIG. The image editing screen shown in FIG. 21 is a diagram illustrating an example in which display of the image editing screen is instructed in the state of FIG. As shown in FIG. 21, the image editing screen includes a condition image display unit 211 and an edited image display unit 212, and a new image is displayed on the edited image display unit 212 using the image displayed on the condition image display unit 211. It is a screen to create.

  The condition image display unit 211 displays an image selected as a condition image as shown in FIG. As shown in FIG. 21, when the image editing screen display instruction unit 184 is operated with the feedback image designation unit 186 checked even before being displayed on the condition image display unit 182 as a condition image. You may display on the condition image display part 211. FIG.

  The edited image display unit 212 is a display unit that displays an image generated by user editing. As shown in FIG. 21, when the user selects any part of the condition image and clicks an arrow displayed beside it, the selected part is displayed on the edited image display unit 212. Further, on the image editing screen, an operation button for cutting out a portion surrounded as a similar portion as shown in FIG. 20 may be provided. Thereby, it is possible to assist a user's operation suitably.

  The user can edit the image displayed on the edited image display unit 212 by operating with a mouse or the like. As an editing mode of an image displayed on the edited image display unit 212, for example, enlargement, reduction, rotation, inversion, and the like can be performed.

  On the image editing screen, the user can also generate a new image by combining a plurality of images displayed on the condition image display unit 211. An example of such a case is shown in FIG. FIG. 22 shows an example in which an image combining the image displayed at the top and a part of the image displayed at the bottom among the images displayed on the condition image display unit 211 is generated. is there.

  As shown in FIG. 22, the user operates the user I / F such as a mouse on the image editing screen to select all or part of the image displayed on the condition image display unit 211 to display the edited image. This is displayed on the unit 212. By performing the same operation on a plurality of images displayed on the condition image display unit 211, it is possible to display a new image by combining a plurality of images as shown in FIG.

  Then, the user operates the “OK” button displayed at the lower right of the image editing screen, and the image generated in the edited image display unit 212 by the method described in FIGS. 21 and 22 is displayed on the search result display screen. The condition image display unit 182 can be added.

  As described above, in the GUI of the image search system according to the present embodiment, a similar image as a result of the search such as the image input by the user as shown in FIG. 6 or the images “B” and “F” shown in FIG. And a function of generating a new image by combining the extracted images (hereinafter referred to as an image combination editing function). Thereby, the image search system which can assist the input of the image for a search can be provided.

  Further, in the GUI of the image search system according to the present embodiment, as described with reference to FIG. 21, a function of cutting a part of each image displayed on the condition image display unit 211 and inserting it into the edited image display unit 212 ( Hereinafter, the image cutting function is used. Thereby, it is possible to further improve the convenience of the difference in which the user generates the search image.

  Further, in the GUI of the image search system according to the present embodiment, when a partial search is executed, as described with reference to FIG. 20, a function that explicitly displays a frame by enclosing a similar part determined to be similar to the condition image. (Hereinafter, similar part highlight function). As a result, the user can easily grasp the similar part. Further, the similar part highlight function can assist the user's operation when cutting out a part of the image on the image editing screen as shown in FIG. That is, the convenience of the user can be further improved by combining the similar part highlight function with the image cropping function or the image combination function.

  In addition, as described above, the image search system according to the present embodiment has a function of executing a feedback search (hereinafter referred to as a feedback search function). Thereby, it is possible to improve the fitness of the search result. In addition, when executing the feedback search, by using the above-described image combination editing function, the image cropping function, and the similar partial highlight function, it becomes easy to add a condition image in the feedback search, thereby improving user convenience. Is possible.

  Note that the screens shown in FIGS. 18 to 22 are displayed based on the display information acquired by the client device 2 from the image search device 1 as described above. In the image search device 1, as described above, the extraction result processing unit 105 generates display information based on the extraction result information acquired from the similarity calculation unit 104. That is, the extraction result processing unit 105 generates display information for displaying a GUI having the functions described with reference to FIGS.

  Further, the search image input screen displayed in S503 of FIG. 5 and the GUI of the image search system such as the GUI described in FIGS. 18 to 22 are provided by, for example, a web browser. That is, the user browses the GUI of the image search system via the web browser on the client device 2. Therefore, the extraction result processing unit 105 generates information for browsing with a web browser as display information for displaying the screen as described with reference to FIGS. Thus, the client device 2 does not need to have a special function, and if there is an information processing device such as a PC in which a general web browser is installed, the image search system according to the present embodiment can be used. Is possible.

  With reference to FIG. 23, the functional configuration of the GUI of the image search system according to the present embodiment will be described. As illustrated in FIG. 23, the GUI according to the present embodiment includes an operation information acquisition unit 231, an image information acquisition unit 232, a display information output unit 233, a search execution instruction unit 234, and an image information editing unit 235. The functional configuration shown in FIG. 23 is configured in the client device 2 by a program loaded in the RAM 20 operating under the control of the CPU 10. The program loaded into the RAM 20 is an application program constituting the web browser and the display information described above.

  The operation information acquisition unit 231 acquires operation information input by the user operating the operation unit 70 of the client device 2. For example, the operation information acquisition unit 231 operates the search target designation unit 181, the re-search instruction unit 183, the image edit screen display instruction unit 184, and the feedback image designation unit 186 in FIG. 18, and the image on the image edit screen in FIG. 21. Get editing operation information.

  The image information acquisition unit 232 acquires image information input by the user or image information extracted as a similar image. For example, the image information acquisition unit 232 acquires the image input by the user in S504 of FIG. 5 as a condition image. Further, the image information acquisition unit 232 acquires information on an image extracted as a similar image as a result of the processing of S509 in FIG. Further, the image acquired by the image information acquisition unit 232 is an image displayed on the condition image display unit 211 in the image editing screen shown in FIG. That is, the image information acquisition unit 232 acquires a plurality of images that are edited or combined on the image editing screen.

  The display information output unit 233 outputs the search image input screen displayed in S503 of FIG. 5 and information for displaying the GUI as described in FIGS. 18 to 22 (hereinafter referred to as screen drawing information). . This screen drawing information is input to the LCD 60 via the I / 50, and each GUI is displayed on the LCD 60.

  The display information output unit 233 switches display according to the operation information acquired by the operation information acquisition unit 231. For example, when the operation information acquisition unit 231 acquires information on an operation for switching the search target designation unit 181 in FIG. 18 from “overall search” to “partial search”, the display information output unit 233 displays the search target designation unit 181. Output display information to toggle toggle.

  The search execution instructing unit 234 8 transmits a command for instructing the image search apparatus 1 to execute an image search according to the operation by the user, that is, the information acquired by the operation information acquiring unit 231. For example, the search execution instruction unit 234 outputs a command for executing a search using the input image as a condition image in S504 of FIG. When the operation information acquisition unit 231 acquires information on an operation of pressing the re-search instruction unit 183, the search execution instruction unit 234 selects an image specified as a search condition by a check of the feedback image specification unit 186 or the like. A command for executing a search as a condition image is output. That is, the search execution instruction unit 234 functions as an image search execution instruction unit that instructs execution of an image search.

  The image information editing unit 235 is configured to perform the functions described with reference to FIGS. 21 and 22. The image information editing unit 235 edits the image information in accordance with the user operation, that is, the information acquired by the operation information acquisition unit 231. Generate image information. That is, the image information editing unit 235 functions as a condition image generation unit that edits the image acquired by the image information acquisition unit 232 or generates a condition image by combining a plurality of images.

  In addition, the same effect as described above can be obtained by installing dedicated application software in the client device 2. Even in this case, the extraction result processing unit 105 uses the extraction result information to display information for displaying the GUI shown in FIGS. 18 to 22 by the dedicated application software installed in the client device 2. Transmit to device 2. In this case, the application software installed in the client device 2 and the extraction result processing unit 105 work together to generate the GUI shown in FIGS.

  In this case, the information transmitted by the extraction result processing unit 105 is, for example, information on an image extracted as a similar image or information indicating a storage location in which an image extracted as a similar image in the image information DB 200 is stored. . The information indicating the storage location is, for example, URL (Uniform Resource Locator) information. On the other hand, the application software installed in the client device 2 has the respective functions described with reference to FIGS.

  The image search system according to the present embodiment can be used as a trademark search system, for example. That is, the image information DB 200 is a database that stores trademark information. In particular, many images disclosed as trademarks have a white background and a clear outline. Therefore, a high effect can be obtained by applying the present embodiment for extracting the contour of an image to trademark search. In addition, as an example of the image information stored in the image information DB 200 to be searched, information on a drawing included in a patent document such as a published patent publication can be considered.

  In the above description, as illustrated in FIG. 4, the example in which the character string information of the search target image is stored in the image information DB 200 has been described. Thereby, it becomes possible to reduce the processing amount required in the response to the search request from the user. In addition, the character string information of the search target image may be calculated by the image processing unit 102 and the character information conversion unit 103 in the process shown in S508 of FIG. Thereby, the amount of information stored in the image information DB 200 can be reduced.

It is a figure which shows the operation | use form of the image search system which concerns on embodiment of this invention. It is a block diagram which shows typically the hardware constitutions of the image search device which concerns on embodiment of this invention. It is a block diagram which shows the function structure of the image search device which concerns on embodiment of this invention. It is a figure which shows the example of the information memorize | stored in image information DB which concerns on embodiment of this invention. It is a sequence diagram which shows operation | movement of the image search system which concerns on embodiment of this invention. It is a figure which shows the example of a condition image. It is a figure which shows the image as a search condition which concerns on embodiment of this invention. It is a flowchart which shows operation | movement of the image process part which concerns on embodiment of this invention. It is a figure which shows the aspect of the image process by the image process part which concerns on embodiment of this invention. It is a figure which shows the example of the information produced | generated by the image process part which concerns on embodiment of this invention. It is a figure which shows the example of the down sampling which concerns on embodiment of this invention. It is a figure which shows the numerical sequence information produced | generated by the image processing part which concerns on embodiment of this invention as a histogram. It is a figure which shows the example of the conversion corresponding | compatible information which concerns on embodiment of this invention. It is a figure which shows the example of the character string information produced | generated by the character information conversion part 103 which concerns on embodiment of this invention. It is a figure which shows the example of the image processing aspect of the image of the search object which concerns on embodiment of this invention. It is a figure which shows the example of the character string information of the image of the search object which concerns on embodiment of this invention. It is a figure which shows the execution aspect of the character search in the similarity calculation process which concerns on embodiment of this invention. It is a figure which shows the example of the part of the image corresponding to N-gram which concerns on embodiment of this invention. It is a figure which shows the example of GUI of the image search system which concerns on embodiment of this invention. It is a figure which shows the example of GUI of the image search system which concerns on embodiment of this invention. It is a figure which shows the example of GUI of the image search system which concerns on embodiment of this invention. It is a figure which shows the example of GUI of the image search system which concerns on embodiment of this invention. It is a figure which shows the example of GUI of the image search system which concerns on embodiment of this invention. It is a figure which shows the function structure of GUI of the image search system which concerns on embodiment of this invention.

Explanation of symbols

1 Image Retrieval Device 2 Client Device 10 CPU
20 RAM
30 ROM
40 HDD
50 I / F
60 LCD
70 Operation unit 80 Bus 100 Search control unit 101 Input image information acquisition unit 102 Image processing unit 103 Character information conversion unit 104 Similarity calculation unit 105 Extraction result processing unit 110 Information input unit 120 Network I / F
130 Display Unit 181 Search Target Specification Unit 182 Condition Image Display Unit 183 Re-Search Instruction Unit 184 Image Editing Screen Display Instruction Unit 185 Similar Image Display Unit 200 Image Information DB
211 Conditional image display unit 212 Edited image display unit 231 Operation information acquisition unit 232 Image information acquisition unit 233 Display information output unit 234 Search execution instruction unit 235 Image information editing unit

Claims (12)

An image search device that calculates a similarity between an image to be searched and an image of a search condition and executes an image search,
An image acquisition unit that acquires a condition image that is an image of the search condition;
An outline extraction unit for extracting an outline of the acquired condition image;
A center calculator for calculating the center of the extracted contour;
A distance calculating unit that generates a numeric sequence information of the calculated the distance to the contour from the center, the distance is calculated for each one time of 359 ° from 0 ° over the periphery around the center,
A character string information generating unit that generates character string information based on the generated numerical string information;
Possess a similarity calculation section for calculating the degree of similarity by performing a text search on the basis of the search object character string information generated for the search target image and the generated character string information,
The similarity calculation unit includes:
Reverse character string information corresponding to numerical value string information calculated every time from 359 ° to 0 ° counterclockwise around the center, character string information corresponding to numerical value string information calculated clockwise, An image search apparatus that performs the character string search based on The character string information generating unit refers to conversion correspondence information in which a numerical value corresponding to the numerical string information and a character constituting the character string information are associated with each other, and converts the numerical string information into character string information. The image search apparatus according to claim 1, wherein the character string information is generated. A contour emphasizing unit for emphasizing the contour of the acquired condition image;
The image search device according to claim 1, wherein the contour extraction unit extracts a contour emphasized by the contour enhancement unit.   The image search device according to claim 1, wherein the contour extracting unit extracts an outermost contour from the contours of the condition image. The said character string information generation part produces | generates the said character string information based on the smoothed numerical sequence information smoothed based on the said generated numerical sequence information, The one of the Claims 1 thru | or 4 characterized by the above-mentioned. The image search device described in 1. The said character string information generation part produces | generates the said character string information based on the normalized numerical sequence information normalized based on the produced | generated numerical sequence information, The one of Claims 1 thru | or 5 characterized by the above-mentioned. The image search device described in 1. The similarity calculation unit, and executes the character string retrieved by the N-gram, the image retrieval apparatus according to any one of claims 1 to 6. The similarity calculation unit, and executes the character string retrieved by the suffix array, the image retrieval apparatus according to any one of claims 1 to 6. A search target image storage unit storing the search target image;
The search target image storing unit is characterized by storing a search object character string information the generated search target image, the image retrieval apparatus according to any one of claims 1 to 8. An image search method for performing image search by calculating a similarity between an image to be searched and an image of a search condition,
The image acquisition unit acquires a condition image that is an image of the search condition,
The contour extraction unit extracts the contour of the acquired condition image,
A center calculation unit calculates the center of the extracted contour,
Distance calculating unit, the distance from the calculated center to the contour, calculates and generates a numerical sequence information of the distance to each one time of 359 ° from 0 ° over the periphery around the center,
A character string information generation unit generates character string information based on the generated numerical string information,
The similarity calculation unit calculates the similarity by executing a character string search based on the generated character string information and the search target character string information generated for the search target image ,
The similarity calculation unit converts the reverse character string information corresponding to the numerical string information calculated every time from 359 ° to 0 ° counterclockwise around the center, and the numerical string information calculated clockwise. An image search method for executing the character string search based on corresponding character string information . A control program for causing an information processing apparatus to operate as an image apparatus that calculates a similarity between an image to be searched and an image of a search condition, and executes an image search,
Obtaining a condition image that is an image of the search condition;
Extracting an outline of the acquired condition image;
Calculating the center of the extracted contour;
Calculating the distance from the calculated center to the contour every time from 0 ° to 359 ° around the circumference of the center to generate numerical sequence information of the distance;
Generating character string information based on the generated numerical string information;
The similarity is calculated by performing a character string search based on the generated character string information and the search target character string information generated for the search target image, and the periphery of the center is counterclockwise. The character string search is executed based on the reverse character string information corresponding to the numerical value string information calculated every time from 359 ° to 0 ° and the character string information corresponding to the numerical value string information calculated clockwise. A control program that causes the information processing apparatus to execute the step of: 12. A recording medium in which the control program according to claim 11 is recorded in a format readable by an information processing apparatus.

Images