JP2010250633A - Image server, image retrieval system, image retrieval method, and index creation method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve the precision of image retrieval without increasing a time required for image retrieval. <P>SOLUTION: An image server 10 is provided with a storage device 13 for storing a plurality of image data. The plurality of image data stored in a storage device 13 are classified into a plurality of clusters as image indexes by a hierarchial clustering method. Representative values as the attributes of the clusters are associated with the respective clusters. Thus, it is possible to improve image retrieval precision without deteriorating an image retrieval speed. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image search system for searching for an image.

  A technique for searching for a desired image via a network has been put into practical use. A server computer (image server) that stores image data stores a lot of image data in association with metadata that describes the characteristics of the image data. A client computer connected to the image server via a network transmits a search request representing an image desired to be searched with a keyword to the image server, and receives one or more image data as a search result from the image server. .

JP 2004-361987 A

  However, in a conventional search using metadata, a search request must be input by a keyword even though the search target is an image. It is not easy to appropriately specify a desired image with a keyword, and there is a case where appropriate metadata is not attached to image data stored on the image server. As a result, there is a problem that an image desired by the user cannot be obtained accurately and quickly. In addition, when an image (image data) is used as a search request, there is a problem that an enormous time is required for the image search and a realistic image search system cannot be provided.

  The present invention has been made to solve at least a part of the above-described problems, and an object thereof is to improve the accuracy of image search without increasing the time required for image search.

  In order to solve at least a part of the above problems, the present invention adopts the following various aspects.

A first aspect provides an image server. An image server according to a first aspect includes: a storage device that stores a plurality of images; a feature amount acquisition unit that acquires a plurality of feature amounts representing image features for each of the plurality of stored images; A normalization unit that normalizes the plurality of feature amounts acquired from each of the images, and a calculation load at the time of image search using the plurality of feature amounts acquired from the images and normalized by a hierarchical clustering technique A cluster processing execution unit that creates a plurality of clusters including images equal to or less than a predetermined number of images determined by the above, and an index creation unit that determines the attribute of each created cluster and associates it with each cluster.

  According to the image server according to the first aspect, by using a plurality of feature amounts acquired from the previous image and normalized by the hierarchical clustering technique, images having a predetermined number of images or less determined by the calculation load at the time of image search are included. A plurality of clusters can be created, attributes of each created cluster can be determined, and an image index can be created in association with each cluster. Therefore, the accuracy of the image search can be improved without increasing the time required for the image search.

  In the image server according to the first aspect, the normalization unit may normalize the plurality of feature amounts using a standard deviation. In this case, it becomes possible to increase the deviation (dispersion, variation) of the feature amount, and the clustering accuracy can be improved.

  In the image server according to the first aspect, the feature amount may include at least two of luminance, hue, texture, and face size. In this case, the feature of the image can be expressed well, and the accuracy of clustering can be improved.

  In the image server according to the first aspect, the predetermined number of images may be 100 to 300. In this case, the image search can be realized without increasing the calculation load on the image server (without increasing the search time).

  A second aspect provides an index creation method in an image server. The index creation method according to the second aspect includes obtaining a plurality of feature quantities representing image features for each of a plurality of images, normalizing the plurality of feature quantities obtained from the images, and a hierarchical clustering method. By using the plurality of feature quantities acquired and normalized from each image, a plurality of clusters to which an image equal to or less than a predetermined number of images determined by the calculation load at the time of image search belongs, and the attribute of each created cluster is Determining and associating with each of the clusters.

  According to the index creation method according to the second aspect, it is possible to obtain the same operational effects as the image server according to the first aspect, and in various aspects similarly to the image server according to the first aspect. Can be realized.

  A third aspect provides an image server. According to the third aspect, the warehouse server uses a hierarchical clustering method to store a plurality of image data clustered in advance into a plurality of clusters, and a search for acquiring a search image that is an image to be searched An image acquisition unit; a plurality of clusters that approximate the search image from the plurality of clusters; a search unit that selects a plurality of images that approximate the search image from a plurality of images belonging to each of the selected clusters; An output unit that outputs a plurality of images searched by the search unit as a search result.

  According to the image server according to the third aspect, a plurality of clusters that approximate the search image are selected and selected from a storage device that stores a plurality of image data clustered in advance into a plurality of clusters by a hierarchical clustering technique. Since a plurality of images that approximate the search image are selected from the plurality of images that belong to each cluster, the accuracy of the image search can be improved without increasing the time required for the image search.

  In the image server according to the third aspect, the cluster is associated with an attribute representing a feature of the cluster, and the search unit acquires the feature amount of the search image, and the acquired feature amount and the attribute are obtained. The plurality of approximate clusters may be selected by using them. In this case, the search time can be shortened without reducing the cluster search accuracy.

  In the image server according to the third aspect, the search unit acquires the feature amount of the image included in the selected cluster, the feature amount of the image included in the selected cluster, and the acquired search The plurality of approximate images may be selected using image feature amounts. In this case, the search image can be searched from a plurality of images included in the cluster, so that the search time can be shortened.

  The image server according to the third aspect further includes, for each of a plurality of images, a feature amount acquisition unit that acquires a plurality of feature amounts representing image features, and the plurality of feature amounts acquired from each of the images. A normalization unit for normalization, and a cluster including images equal to or less than a predetermined number of images determined by a calculation load at the time of image search, using the plurality of feature quantities acquired and normalized from each image by a hierarchical clustering technique. There may be provided a plurality of cluster execution units to be created, and an index creation unit for determining attributes of each created cluster and associating the obtained attributes with the clusters. In this case, it is possible to create an image index that can improve the accuracy of the image search without increasing the time required for the image search.

  A fourth aspect provides an image search system. An image search system according to a fourth aspect is an image server according to the third aspect and a printer that is accessible to the image server via a network, and transmits the search image to the image server. And a printer that includes a receiving unit that receives a plurality of images as the search results output from the image server, and a display unit that outputs the received images.

  According to the image search system according to the fourth aspect, it is possible to improve the accuracy of the image search without increasing the time required for the image search.

  The fifth aspect provides an image search method. The image search method according to the fifth aspect obtains a search image, which is an image to be searched, and uses a plurality of image data stored in a storage device and clustered in advance by a hierarchical clustering technique. Selecting a plurality of clusters approximating the search image from the plurality of clusters, selecting a plurality of images approximating the search image from a plurality of images belonging to each of the selected clusters, and retrieving the selected plurality of images as a search result As output.

  According to the image search method according to the fifth aspect, it is possible to obtain the same operational effects as the image server according to the third aspect, and in various aspects similarly to the image server according to the third aspect. Can be realized.

  The methods according to the second and fifth aspects can be realized as an index creation program and an image search program, respectively, and can also be realized as a computer-readable medium on which the index creation program and the image search program are recorded.

It is explanatory drawing which shows schematic structure of the image search system which concerns on a present Example. It is explanatory drawing which shows typically the internal structure of the image server which concerns on a present Example with a functional block diagram. It is explanatory drawing which shows the various programs and modules stored in the memory with which an image server is provided. FIG. 3 is an explanatory diagram schematically illustrating an internal configuration of the printer according to the embodiment in a functional block diagram. It is explanatory drawing which shows the various programs and modules stored in the memory with which an image printer is provided. It is a flowchart which shows the processing routine performed in order to produce an image index using a hierarchical clustering method as a 1st clustering method. FIG. 4 is an explanatory diagram showing a concept of clusters generated by the first clustering method. It is explanatory drawing which shows an example of the image management table in an image database. It is explanatory drawing which shows an example of the cluster table which manages the attribute of a cluster. It is a flowchart which shows the processing routine performed in order to produce an image index using a non-hierarchical clustering method as a 2nd clustering method. It is explanatory drawing which shows the concept of the cluster produced | generated by the 2nd clustering method. It is a flowchart which shows the image search process routine performed in the image server which concerns on a present Example. 6 is a flowchart illustrating an image search request processing routine executed in the printer according to the embodiment. It is explanatory drawing which shows typically the concept of the search process of the image data using the cluster formed by the 2nd clustering method.

  Hereinafter, an image search system, an image server, an index creation method, an image search method, and a printer according to the present invention will be described based on embodiments with reference to the drawings.

  FIG. 1 is an explanatory diagram illustrating a schematic configuration of an image search system according to the present embodiment. The image search system ISS according to the present embodiment includes a server computer 10, a printer 20, and a personal computer 30. The server computer 10 is connected to the printer 20 and the personal computer 30 through a network NE so that bidirectional communication is possible. The network may be the Internet or an intranet.

  The server computer 10 stores a large number of image data to be searched and executes an image search in response to a search request from a client computer. Therefore, it can be said that the server computer is an image server. The printer 20 and the personal computer 30 can be called a client for the image server 10 or an image search terminal device. The personal computer 30 includes a display 31 and an input device 32 such as a keyboard and a mouse.

Image server configuration:
FIG. 2 is an explanatory diagram schematically showing the internal configuration of the image server according to the present embodiment in a functional block diagram. FIG. 3 is an explanatory diagram showing various programs and modules stored in a memory included in the image server. The image server 10 includes a central processing unit (CPU) 11, a memory 12, a storage device (also referred to as an image database) 13 in which an image database is constructed, and an input / output interface 14 that are communicably connected to each other. . The CPU 11 executes various programs and modules stored in the memory 12. The memory 12 stores programs and modules executed by the CPU 11 in a nonvolatile manner, and has a volatile work area where the programs and modules are expanded when the CPU 11 executes processing. As the memory 12, for example, a semiconductor storage device such as a read only memory that stores a program or the like in a non-volatile manner, or a random access memory that provides a volatile work area when the program is executed can be used.

  The storage device 13 is configured by one or a plurality of mass storage devices such as a hard disk drive and a flash memory drive. In the storage device 13, an image database including a plurality of image data and image data management information (attribute information) such as the feature amount of each image data, the cluster to which the image data belongs, and the feature amount of the image data is constructed. In addition, although the aspect in which the image data is output (displayed or printed) is an image, it is obvious to those skilled in the art that the image in the claims means the image data. The storage device 13 may function as an image data storage unit that stores only a plurality of image data. In this case, the feature amount of each image data stored in the storage device 13 and the management information (attribute information) of the image data are stored in the memory 12. In this embodiment, when viewed from the client, the image server 10 itself functions as an image database. In the present embodiment, the storage device 13 is described as an example of a database that simply stores image data and image management information. However, the storage device 13 includes a control unit having a search function, and searches from the outside. It may be an independent database system (file server) that outputs search results in response to requests. In this case, an image database system is arranged outside the image server 10, and search requests and search results are transmitted and received between the two via the input / output interface 14. In either case, the database search program is stored in the memory 12 and executed by the CPU 11.

  The input / output interface 14 transmits / receives a search request and a search result to / from an external device, for example, a client such as the printer 20 or the personal computer 30 according to a known communication protocol.

  Various programs and modules stored in the memory 12 will be described with reference to FIG. The memory 12 stores an image index creation program SP1 that is executed to create an image index, and an image search program SP2 that is executed to search for an image from the storage device 13. The image index creation program SP1 includes an image data acquisition module SM11, a feature amount acquisition module SM12, a normalization module SM13, a clustering module SM14, and an index creation module SM15. The image data acquisition module SM11 is a module executed to sequentially acquire image data stored in the storage device (image database) 13 without duplication. The feature amount acquisition module SM12 is a module executed for analyzing the image data acquired by the image data acquisition module SM11 and acquiring the feature amount. Here, the feature amount is, for example, values such as average luminance, minimum luminance, maximum luminance, hue, and face size and position included in the image data. Since these feature quantities can be obtained by methods well known to those skilled in the art, description thereof will be omitted.

  The normalization module SM13 is a module executed to normalize each feature amount of all image data included in the storage device 13 acquired by the feature amount acquisition module SM12. The normalization may be executed using, for example, a standard deviation of feature amounts, or may be executed such that the maximum value of each acquired feature amount is 1 and the minimum value is 0. The clustering module SM14 is a module executed to classify the image data in the image database 13 into a plurality of clusters using each normalized feature amount. Hierarchical clustering methods and non-hierarchical clustering methods are known as clustering methods. In this embodiment, a hierarchical clustering method is used as the first clustering method, and a K-means method, which is one of the non-hierarchical clustering methods, is used as the second clustering method. The index creation module SM15 is a module executed to create an index used for image search by assigning a representative value to each cluster formed by clustering. Note that the image index creation program SP1, the image data acquisition module SM11, the feature amount acquisition module SM12, the normalization module SM13, the clustering module SM14, and the index creation module SM15 are executed by the CPU 11, respectively, so that the image index creation unit, the image It functions as a data acquisition unit, a feature amount acquisition unit, a normalization unit, a clustering unit, and an index creation unit.

  The image search program SP2 includes a search image data acquisition module SM21 and an image data search module SM22. The image data search module SM22 includes a search image data feature amount acquisition module SM23 and a similarity calculation module SM24 as submodules. The search image data acquisition module SM21 is a module executed to acquire search image data transmitted by the printer 20 and the personal computer 30 that are clients. The image data search module SM22 is a module that is executed to search the storage device 13 for image data that is similar to or matches the search image data, using the acquired search image data. The search image data feature amount acquisition module SM23 is executed to acquire a plurality of feature amounts of the acquired search image data, and the similarity calculation module SM24 includes the acquired feature amount of the search image data, the representative value of each cluster, and This is a module executed to calculate the similarity between the search image data and the searched image data using the feature amount of the image data belonging to each cluster. Note that the image search program SP2, the search image data acquisition module SM21, the image data search module SM22, the search image data feature amount acquisition module SM23, and the similarity calculation module SM24 are executed by the CPU 11, respectively, so that the image search unit, the search It functions as an image acquisition unit, an image data feature amount acquisition unit, and a similarity calculation unit.

Printer configuration:
FIG. 4 is an explanatory diagram schematically showing the internal configuration of the printer according to the present embodiment in a functional block diagram. FIG. 5 is an explanatory diagram showing various programs and modules stored in a memory included in the image printer. In this embodiment, the printer 20 is described as an example of the image search terminal device, but it goes without saying that the personal computer 30 can also be used as an image search terminal device. The printer 20 includes a control circuit 21, an input operation unit 22, a display unit 23, a printing unit 24, and an external input / output interface 25 that are connected to each other by signal lines. The control circuit 21 includes a central processing unit (CPU) 210, a memory 211, and an input / output (I / O) interface 212 that are communicably connected to each other. The CPU 210 executes various programs and modules stored in the memory 211. The memory 211 stores programs and modules executed by the CPU 210 in a nonvolatile manner, and has a volatile work area where the programs and modules are expanded when the CPU 210 executes processes. As the memory 211, for example, a semiconductor storage device such as a read-only memory that stores a program or the like in a nonvolatile manner, or a random access memory that provides a volatile work area when the program is executed can be used. The input / output interface 212 transmits and receives commands and data between the control circuit 21 and the input operation unit 22, the display unit 23, the printing unit 24, and the external input / output interface 25. The input operation unit 22 is an operation unit for a user to input an instruction to the printer 20 and can be realized by, for example, a button or a wheel. The display unit 23 is a display screen capable of color display for displaying an image based on image data searched for the user and displaying various information for the user. The printing unit 24 is a printing execution unit that forms an image on a printing medium in accordance with a printing instruction from a user (control circuit 21). The external input / output interface 25 transmits / receives a search request and a search result to / from an external device, for example, the image server 10 according to a known communication protocol.

  Various programs and modules stored in the memory 211 will be described with reference to FIG. The memory 211 includes an image search request program CP1 for requesting an image search to the image server 10, and the image search request program CP1 includes an image data identification module CM11, a search request module CM12, a search result acquisition module CM13, and a search. A result display module CM14 is provided. The image data specifying module CM11 is a module that is executed to specify an image (image data) to be searched (search key). The search request module CM12 is a module for transmitting the specified image data and a search request to the image server 10. The search result acquisition module CM13 is a module for acquiring one or a plurality of image data as a search result from the image server 10. The search result display module CM14 is a program for displaying an image as a search result on the display unit 23 using one or a plurality of image data acquired as a search result. The image search request program CP1, the image data identification module CM11, the search request module CM12, the search result acquisition module CM13, and the search result display module CM14 are executed by the CPU 210, respectively, so that an image search request unit, a search image specification unit , Functions as a search request unit, a search result acquisition unit, and a search result display unit.

First clustering method:
FIG. 6 is a flowchart showing a processing routine executed for creating an image index using a hierarchical clustering technique as the first clustering technique. FIG. 7 is an explanatory diagram showing the concept of clusters generated by the first clustering method. FIG. 8 is an explanatory diagram showing an example of an image management table in the image database. FIG. 9 is an explanatory diagram showing an example of a cluster table for managing representative values of clusters.

  This processing routine is executed by the image server 10 at, for example, the first timing of creating an image index for the image database, or at the timing when a predetermined amount of new image data is added to the image database. The image server 10 (CPU 11) executes the image data acquisition module SM11 and acquires image data from the image database 13 (step S100). The acquired image data is temporarily stored in the memory 12.

  The CPU 11 executes the feature amount acquisition module SM12 and acquires a plurality of feature amounts from the image data stored in the memory 12 (step S102). The feature amount used in this embodiment is, for example, the brightness of the image data or the brightness histogram in a partial area of the image data, texture, presence / absence of face, face size, hue, and a plurality of feature amounts are used. It is done. From the luminance histogram, the average luminance value, the maximum luminance value, and the minimum luminance value are acquired, and the hue is the average value of each of R, G, and B components in the image data, or the total number of pixels constituting the image data. The ratio of the number of R, G, and B component pixels that occupy a predetermined value or more is acquired. Since a method of acquiring a luminance histogram, hue, and texture as a feature amount is a method well known to those skilled in the art, a description thereof will be omitted. The presence / absence of a face, that is, face detection and face size detection, for example, includes a pattern matching method using a model face shape, a method for specifying a skin color hue region, and a method for detecting a face organ. Or they can be used in combination.

  The CPU 11 repeatedly executes steps S100 and S102 until the feature amount is acquired from all the image data stored in the image database (step S104: No). When the acquisition of feature amounts for all image data is completed (step S104: Yes), the CPU 11 normalizes the feature amounts acquired by executing the normalization module SM13 for each type (step S106). In this embodiment, the standard deviation σ of the feature amount of each image data is obtained for each feature amount, and normalization is performed using the obtained standard deviation σ and the following formula so that the average is 0 and the standard deviation is 1. Is called. Normalization value = (x−μ) / σ (x is a feature value to be normalized, and μ is an average value of the feature values).

  The CPU 11 executes the clustering module SM14 and executes a clustering process for classifying the image data in the image database 13 into a plurality of clusters by a hierarchical clustering method using each feature amount after the normalization process as a parameter (step) S108). In this embodiment, as a hierarchical clustering method, as shown in FIG. 7, a divided clustering method is used in which the whole is sequentially divided using a predetermined feature amount. In the present embodiment, the same predetermined feature amount is used a plurality of times for the divided cluster, but may be used only once. In the example of FIG. 7, first, image data is divided based on whether or not the image data (image) includes a face based on the normalized feature amount. Subsequently, for a cluster including a face, the face size (a normalized feature amount indicating the face size) is greater than or equal to a predetermined value (large face size) or less than a predetermined value (small face size). It is further divided based on whether there is any. A cluster having a large face size is further divided based on whether the face is a child's face or an adult's face (age) (for example, based on a feature amount related to the organ position of the face). A cluster having a small face size is further classified based on whether or not it is a landscape scene (whether the normalized feature amount indicating the hue of blue or green is equal to or greater than a predetermined value). The scene scene cluster is further divided based on whether the hue indicates blue, and the cluster where the hue is not blue is further divided based on whether the hue indicates green. Clusters that do not include a face are classified based on whether or not they are landscapes. For clusters that are not landscape scenes, the saturation is greater than or equal to a predetermined value (high saturation) or less than a predetermined value (saturation low). It is further divided based on whether there is any.

  The division of the clusters is repeated until, for example, the number of image data included in each cluster is equal to or less than a predetermined number of image data determined by a calculation load at the time of image data search. Here, the calculation load at the time of image data search is a calculation load (calculation time) of similarity between search image data to be described later and image data belonging to a cluster, and the predetermined number of image data is, for example, calculation time Is the number of image data that is 2 seconds or less. The number of predetermined image data that satisfies this condition based on the current general server computing capacity is, for example, 100 to 300. However, it is clear that the number of predetermined image data increases as the computing capability of the server increases. In addition to this, an agglomerative clustering method that repeatedly collects similar image data may be used as the hierarchical clustering method. Also in this case, the number of image data belonging to each cluster can be controlled to a desired number. Since the hierarchical clustering method is well known to those skilled in the art, further explanation is omitted.

  Each image data is associated with a cluster by the management table shown in FIG. 8, thereby realizing clustering. In the example of FIG. 8, the management table describes the file name of the image data, the cluster to which the image data belongs, and the unnormalized feature quantity. Instead of the management table, the belonging cluster and the unnormalized feature quantity may be described in a common format in the header of each image data.

  The CPU 11 acquires a representative value (attribute) of each cluster obtained by clustering, associates it with each cluster (step S110), and ends this processing routine. That is, an image index for quickly identifying similar image data is created by associating attributes with each cluster. The representative value of each cluster is acquired by, for example, obtaining an average value of each feature amount that is not normalized of the image data belonging to each cluster. The acquired representative value of each cluster and each cluster are associated with each other by, for example, the cluster table shown in FIG. In the example of FIG. 9, luminance components, R, G, and B components, and the average face size are associated with each cluster as attributes (representative values). This cluster table may be stored in the memory 12 or may be stored in a predetermined area of the image database (storage device 12).

Second clustering method:
FIG. 10 is a flowchart showing a processing routine executed for creating an image index using a non-hierarchical clustering technique as the second clustering technique. FIG. 11 is an explanatory diagram showing the concept of clusters generated by the second clustering method.

  This processing routine is executed by the image server 10 at, for example, the first timing of creating an image index for the image database, or at the timing when a predetermined amount of new image data is added to the image database. Since the image index generation method using the first clustering method is the same except that the clustering method is the second clustering method, the same steps (processes) are the same as the first clustering method. The same step number as the step number used in the description of FIG. The image server 10 (CPU 11) executes the image data acquisition module SM11 and acquires image data from the image database 13 (step S100). The acquired image data is temporarily stored in the memory 12.

  The CPU 11 executes the feature amount acquisition module SM12 and acquires a plurality of feature amounts from the image data stored in the memory 12 (step S102). The feature amount used in this embodiment is, for example, the brightness of the image data or the brightness histogram in a partial area of the image data, texture, presence / absence of face, face size, hue, and a plurality of feature amounts are used. It is done. From the luminance histogram, the average luminance value, the maximum luminance value, and the minimum luminance value are acquired, and the hue is the average value of each of R, G, and B components in the image data, or the total number of pixels constituting the image data. The ratio of the number of R, G, and B component pixels that occupy a predetermined value or more is acquired. Since a method of acquiring a luminance histogram, hue, and texture as a feature amount is a method well known to those skilled in the art, a description thereof will be omitted. The presence / absence of a face, that is, face detection and face size detection, for example, includes a pattern matching method using a model face shape, a method for specifying a skin color hue region, and a method for detecting a face organ. Or they can be used in combination.

  The CPU 11 repeatedly executes steps S100 and S102 until the feature amount is acquired from all the image data stored in the image database (step S104: No). When the acquisition of feature amounts for all image data is completed (step S104: Yes), the CPU 11 normalizes the feature amounts acquired by executing the normalization module SM13 for each type (step S106). In this embodiment, the standard deviation σ of the feature amount of each image data is obtained for each feature amount, and normalization is performed using the obtained standard deviation σ and the following formula so that the average is 0 and the standard deviation is 1. Is called. Normalization value = (x−μ) / σ (x is a feature value to be normalized, and μ is an average value of the feature values).

  The CPU 11 executes the clustering module SM14 and executes a clustering process for classifying the image data in the image database 13 into a plurality of clusters by a non-hierarchical clustering method using each feature amount after the normalization process as a parameter ( Step S109). In this embodiment, as a non-hierarchical clustering method, a K-average method is used as shown in FIG. In the example of FIG. 11, the vertical axis and the horizontal axis take values of 0 to 1, and two normalized feature quantities (feature quantity A and feature quantity B) are plotted. First, the number K of clusters to be generated is determined, and K arbitrary cluster center points (positions) CP are determined (K temporary clusters are generated. Subsequently, the center of gravity of each cluster, for example, each cluster is determined. The respective averages of the feature amounts A and B of the image data to which it belongs are calculated, the distance between each image data (coordinate position) and the center of gravity (coordinate position) of each cluster is obtained, and each image data is set to the nearest cluster. The image data is reassigned repeatedly until each image data does not move, or until the center-of-gravity distance between each image data and each cluster is equal to or less than a predetermined distance. The distance from the coordinates of the center of gravity can be determined by, for example, the Euclidean distance, Mahalanobis distance, and Manhattan distance known to those skilled in the art. As, In addition to this, it is possible to use a self-organizing map (SOM) and the like.

  Each image data is associated with a cluster by the management table shown in FIG. 8, thereby realizing clustering. The CPU 11 acquires a representative value (attribute) of each cluster obtained by clustering, associates it with each cluster (step S110), and ends this processing routine. That is, an image index for quickly identifying similar image data is created by associating attributes with each cluster. The representative value of each cluster is obtained by obtaining an average value of unnormalized feature amounts of image data belonging to each cluster. The acquired representative value of each cluster and each cluster are associated with each other by, for example, the cluster table shown in FIG.

Search for image data:
FIG. 12 is a flowchart showing an image search processing routine executed in the image server according to the present embodiment. FIG. 13 is a flowchart illustrating an image search request processing routine executed in the printer according to the present embodiment. FIG. 14 is an explanatory diagram schematically showing the concept of image data search processing using clusters formed by the second clustering method. Hereinafter, description will be made with reference to FIGS. 12 and 13 as appropriate.

  This will be described with reference to FIG. The printer 20 (CPU 210) executes the image data specifying module CM11 and specifies the search image data corresponding to the image (search image) specified by the user from among the images displayed on the display unit 23 (step). S300). The image displayed on the display unit 23 is, for example, image data stored in a portable semiconductor memory device, and is selected via the input operation unit 22. The CPU 210 executes the search request module CM12 and transmits the search image data to the image server 10 (step S302). Specifically, the image data is transmitted to the image server 10 via the external input / output interface 25 and the network NE.

  This will be described with reference to FIG. The image server 10 (CPU 11) executes the image data acquisition module SM21 to acquire the search image data transmitted from the printer 20 (step S200). Specifically, the search image data is taken into the image server 10 via the input / output interface 14 and stored in the memory 12. The CPU 11 executes the search image data feature amount acquisition module SM23 to acquire a plurality of feature amounts of the search image data (step S202). As the feature amount, in order to improve the search accuracy, for example, a plurality of feature amounts used by the image server 10 for clustering are acquired. The method for acquiring the feature amount is as described in the creation of the image index.

  The CPU 11 executes the similarity calculation module SM24 and searches for a plurality of clusters that are similar to or match the search image data (step S204). Specifically, the similarity is calculated using the value of each feature value of the acquired search image data and the attribute (representative value of each feature value) associated with each cluster. The similarity is, for example, the distance between the feature amount of the search image data calculated by a distance calculation method such as the Euclidean distance or the Mahalanobis distance, and the representative value associated with each cluster, that is, the search image data Judgment is made using the distance between the multi-element vectors represented by the feature quantity and the representative value of each cluster, and the shorter the obtained distance, the more similar it is judged. Alternatively, the determination may be made by obtaining the inner product of the multi-vector of each feature quantity of the search image data and the multi-vector of the representative value corresponding to each cluster. In this case, the closer the inner product value is to 1, the higher the similarity can be determined. It should be noted that the calculation method of each distance and the calculation formula of the inner product for calculating the similarity are well known to those skilled in the art, and thus description thereof is omitted. For example, when the image index of the image database shown in FIG. 14 is formed by the K-average method, a plurality of clusters having representative values close to the search image data TD are selected. In FIG. 14, the vertical axis and the horizontal axis are unnormalized actual values of the feature amounts. Even when the image index of the image database is formed by the hierarchical clustering method, a plurality of clusters having representative values close to the search image data, generally a plurality of clusters divided from the same branch, are selected. The In selecting a plurality of clusters, a distance as a threshold value may be determined in advance, or the number of clusters to be selected may be determined.

  The CPU 11 executes the similarity calculation module SM24 to search for a plurality of image data that matches or is similar to the search image data from the plurality of image data belonging to the plurality of clusters obtained by the search (step S206). Similarity or coincidence is determined by calculating the above-described similarity using each feature amount of the search image data and each feature amount of each image data belonging to each cluster. In selecting a plurality of image data, a distance as a threshold value may be determined in advance, or the number of image data may be determined.

  The CPU 11 transmits the detected plurality of image data to the printer 20 that is the transmission source of the search request (step S208), and ends this processing routine. Note that the printer 20 from which the search request is transmitted can be identified by, for example, identification information (for example, a MAC address) of the external input / output interface 25 provided in the printer 20, and the search is performed using the identification information. A plurality of pieces of image data can be transmitted to the printer 20 that is the transmission source of the request.

  This will be described with reference to FIG. When the printer 20 receives a plurality of image data (step S304), the printer 20 executes the search result acquisition module CM13 and stores it in the memory 211. The CPU 210 executes the search result display module CM14 to display the image data stored in the memory 211 on the display unit 23 (step S306), and ends this processing routine.

  According to the image index creation method, the image server, the image search method, and the printer according to the present embodiment described above, desired image data can be inspected using the image index. Even when searching, it is possible to improve the image data search speed. In addition, since clustering is performed using a plurality of feature amounts of image data when an image index is created, the image data search speed can be improved without reducing the image data search accuracy.

  In addition, the image data is searched based on the feature amount of the image data without using the metadata describing the contents of the image data associated with the image data, so it is necessary to specify the image data using words (keywords). This makes it easy to search for image data. Furthermore, since it does not depend on subjective metadata, the objectivity of image data search can be improved, and the image data search accuracy can be improved.

  When the hierarchical clustering method is used as the first clustering method, the number of image data belonging to each cluster can be adjusted to an arbitrary number, so that the image data according to the search processing capability of each image server. By generating each cluster so as to include a number, it is possible to suppress a delay in the search time for image data.

  In addition, since the division type first clustering method has one feature amount used for one division, it is easy to specify the attribute of each cluster, for example, a landscape or a face image. Accordingly, it is possible to provide a text description for each cluster in an image data providing service or the like, and there is an advantage that it is easy to provide image data desired by the user.

  When the K-means method (non-hierarchical clustering method) is used as the second clustering method, clustering is performed using a plurality of normalized feature amounts. The image data can be searched based on the image index more reflecting the characteristics of each image data.

Other examples:
(1) In the above embodiment, a server computer that searches an image database in response to a request from a client has been described as an example of the image server 10, but for example, a local computer stored in a storage device of the personal computer 30 is used. The above-described image index may be created for the image database. In other words, the image server may be realized as a part of the functions of a stand-alone personal computer or printer that is not connected to a network, as well as a computer program and a computer-readable medium storing the computer program. In this case, it is possible to realize convenience of image data search in personal use, that is, improvement in search speed, improvement in search accuracy, and ease of search. Furthermore, when the printer 20 is provided with a large-capacity storage device, the above-described image index creation and image search method may be applied to local image data search in the printer 20. As the computer-readable medium, various recording media such as a CD, a DVD, a hard disk drive, and a flash memory can be used.

(2) In the creation of an image index according to the present embodiment, brightness, hue, texture, face size, and presence / absence of a face are used as the feature amount of image data, but other feature amounts may be used. Needless to say. For example, it is possible to use feature amounts such as contrast of image data, saturation, arrangement positions (patterns) of organs such as face eyes, nose, mouth, exposure correction amount, and shooting scene information. By using these feature amounts, an image index suitable for the purpose can be created, and a search result of image data desired by the user can be obtained.

(3) In the above embodiment, the image server including both the image index creation program and the image search program for creating the image index has been described. However, the image server may be either an image index creation program or an image search program. Either may be provided. For example, when an external image database that has already been constructed is used, the image server only needs to have an image search program. When the image search program is provided to the client, the image server has an image search program. It is sufficient if only an index creation program is provided. Which aspect is taken may be determined depending on what kind of system configuration is used.

(4) In the above embodiment, the description of the clusters by text is not performed. However, for example, in the case of the split type first clustering method, each cluster is described by text, and the user An environment in which desired image data can be easily selected may be prepared. In this case, in addition to the feature amount of the image data, it is possible to provide a search result of the image data more suitable for the user's needs by prompting selection of a cluster having the attribute desired by the user.

(5) In the above embodiment, when the image data included in the image database increases, the processing for forming the image index is executed again in the first and second clustering methods and included in the image database. When the image data is reduced, only the representative values of the clusters need be recalculated in the first and second clustering methods. Generally, in the second clustering method, only a representative value of each cluster is obtained, and the affiliation of the image data to each cluster is not questioned. However, if managed by the table described with reference to FIG. The cluster to which the image data belongs can be clarified.

(6) In the above-described embodiment, a plurality of clusters and a plurality of image data are searched. However, when time is not required for the narrowing search, a single cluster and a single image data may be searched. . In this case, further selection by the user can be eliminated.

  As mentioned above, although this invention was demonstrated based on the Example and the modification, Embodiment mentioned above is for making an understanding of this invention easy, and does not limit this invention. The present invention can be changed and improved without departing from the spirit and scope of the claims, and equivalents thereof are included in the present invention.

ISS ... Image search system 10 ... Server computer (image server)
11 ... CPU
12 ... Memory 13 ... Storage device (image database)
DESCRIPTION OF SYMBOLS 14 ... Input / output interface 20 ... Printer 21 ... Control circuit 22 ... Input operation part 23 ... Display part 24 ... Printing part 25 ... External input / output interface 210 ... CPU
211 ... Memory 212 ... Input / output interface 30 ... Personal computer 31 ... Display display 32 ... Input device CP1 ... Image search request program CM11 ... Image data identification module CM12 ... Search request module CM13 ... Search result acquisition module CM14 ... Search result display module SP1 Image index creation program SM11 Image data acquisition module SM12 Feature quantity acquisition module SM13 Normalization module SM14 Clustering module SM15 Index creation module SP2 Image search program SM21 Image search data acquisition module SM22 Image data search module SM23 ... Search image data feature acquisition module SM24 ... Similarity calculation module TD ... Search image data Data NE ... network

Claims (11)

An image server,
A storage device for storing a plurality of images;
For each of the plurality of stored images, a feature amount acquisition unit that acquires a plurality of feature amounts representing image features;
A normalization unit that normalizes the plurality of feature amounts acquired from each of the images;
A cluster processing execution unit that creates a plurality of clusters including images equal to or less than a predetermined number of images determined by a calculation load at the time of image search using the plurality of feature amounts acquired and normalized from each image by a hierarchical clustering method; ,
An image server comprising an index creation unit that determines an attribute of each created cluster and associates the attribute with each cluster. The image server according to claim 1,
The normalization unit is an image server that normalizes the plurality of feature amounts using a standard deviation. The image server according to claim 1 or 2,
The feature server includes at least two of brightness, hue, texture, and face size. The image server according to any one of claims 1 to 3,
An image server in which the predetermined number of images is 100 to 300. An indexing method in an image server,
For each of a plurality of images, obtain a plurality of feature amounts representing the features of the image,
Normalizing the plurality of feature quantities acquired from each of the images;
Using a plurality of feature quantities acquired and normalized from each image by a hierarchical clustering method, a plurality of clusters to which images of a predetermined number of images or less determined by a calculation load at the time of image search belong,
An index creation method for determining an attribute of each created cluster and associating it with the cluster. An image server,
A storage device for storing a plurality of image data clustered in advance into a plurality of clusters by a hierarchical clustering method;
A search image acquisition unit for acquiring a search image that is an image to be searched;
A search unit that selects a plurality of clusters that approximate the search image from the plurality of clusters, and that selects a plurality of images that approximate the search image from a plurality of images belonging to each of the selected clusters;
An image server comprising: an output unit that outputs a plurality of images searched by the search unit as a search result. The image server according to claim 6,
The cluster is associated with an attribute representing the characteristics of the cluster,
The search unit is an image server that acquires a feature amount of the search image and selects the plurality of approximate clusters using the acquired feature amount and the attribute. The image server according to claim 7, wherein
The search unit acquires a feature amount of an image included in the selected cluster, and uses the feature amount of an image included in the selected cluster and the acquired feature amount of the search image to perform the approximation. An image server that selects multiple images to perform. The image server according to any one of claims 6 to 8, further comprising:
For each of the plurality of images, a feature amount acquisition unit that acquires a plurality of feature amounts representing the features of the image;
A normalization unit that normalizes the plurality of feature amounts acquired from each of the images;
A cluster execution unit that creates a plurality of clusters including images equal to or less than a predetermined number of images determined by a calculation load at the time of image search using the plurality of feature amounts acquired and normalized from each image by a hierarchical clustering method;
An image server comprising: an index creating unit that determines an attribute of each created cluster and associates the obtained attribute with the cluster. An image search system,
An image server according to any one of claims 6 to 9,
A printer capable of accessing the image server via a network,
A transmission unit for transmitting the search image to the image server;
A receiving unit for receiving a plurality of images as the search results output from the image server;
A printer comprising a display unit for outputting the received image.
Image search system. An image search method,
Retrieve the search image that is the target of the search,
Using a plurality of image data stored in the storage device, selecting a plurality of clusters that approximate the search image from a plurality of clusters clustered in advance by a hierarchical clustering method,
Select a plurality of images that approximate the search image from a plurality of images belonging to each of the selected clusters,
An image search method for outputting the selected plurality of images as a search result.

Images