JP2006155340A - Image managing device, image management method, and image management program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image managing device for improving the retrieval efficiency of images. <P>SOLUTION: As a result of initial learning, an image having the corresponding image feature quantity is connected to a keyword (Old keyword) in a default recognition dictionary. The relationship between the feature quantity (f) of the image and the keyword (k) is shown by a distribution function P(k¾f) of initial posterior probability. When other images are shown along with images used for initial learning in images, such as a picture photographed by a user, the keyword of the other image is allowed to correspond to that of an image used in the initial learning. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image management apparatus, an image management method, and an image management program, and more particularly to an image management apparatus, an image management method, and an image management program that manage images based on image feature amounts.

  There is a high demand for efficiently searching for a target image from a large number of images. For this reason, a technique has been developed in which an image is automatically given meaning by a computer and a search is performed based on the content.

  In a personal photo database, it is not possible to meet various search needs only by giving general meaning to images. Therefore, in a personal photo database, meaning that reflects the individual matters of the user is essential.

  Also, in order to more actively utilize meanings reflecting individual user matters, it is desirable to perform automatic processing according to the meaning content.

  In relation to such a technique, Patent Document 1 below discloses an image database system that, when a keyword is input, finds a corresponding image from a keyword-image dictionary and searches for a similar image using the image. ing.

  Non-Patent Document 1 below discloses a technique for associating a keyword attached to an image as a search result as a similar word of a keyword of a search image when an example image is searched from an image group to which a keyword is attached. .

Non-Patent Document 2 below discloses a technique for searching for an image corresponding to an input keyword using a dictionary in which the keyword is associated with an image feature value for each divided region.
JP 09-265472 A XS Zhou, TS Huang, “Unifying Keywords and Visual Contents in Image Retrieval”, IEEE MultiMedia, 9 (2), 2002. Pinar Duygulu et al., “Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary”, Seventh European Conference on Computer Vision, pp IV: 97-112, 2002.

  An object of the present invention is to provide an image management apparatus, an image management method, and an image management program that can improve the search efficiency of images.

  In order to achieve the above object, according to one aspect of the present invention, an image management apparatus stores a map that associates a distribution of predetermined image feature amounts with predetermined key information, and a plurality of given images. An association means for associating the image group with the key information by referring to the stored map based on the image feature amount obtained by analyzing the image, the distribution of the image feature amount existing in the image group, and the key information Creating means for newly creating a map in which

  Preferably, the creation means creates a new map in consideration of stored map information.

  Preferably, the image management device further includes output means for referring to a newly created map and outputting an image group associated with the key information from a plurality of given image groups.

  Preferably, the storage means stores a map group consisting of a plurality of sets of image feature quantity distribution and key information, and the creating means creates a map group consisting of a plurality of sets of image feature quantity distribution and key information. Then, the image management apparatus outputs key information related to the given image by referring to the map group.

  Preferably, the key information is any one of a keyword, an image, a sound, a symbol, a telephone number, an e-mail address, or a combination thereof.

  According to another aspect of the present invention, the image management apparatus includes a first storage unit that stores a first map in which a distribution of a predetermined image feature amount and predetermined key information are associated with each other, and another predetermined image. Based on the second storage means for storing the second map in which the distribution of the feature quantity and another predetermined key information are associated, and the image feature quantity obtained by analyzing the given plurality of images, With reference to the first map, based on the first association means for associating the first image group and the first key information, and the image feature amount obtained by analyzing a plurality of given images, With reference to the second map, based on the second association means for associating the second image group and the second key information, and the degree of overlap between the first image group and the second image group, And third association means for associating the first key information with the second key information.

  Preferably, when the operation is performed on the first key information, the image management apparatus automatically executes an operation associated with the second key information.

  Preferably, the operation is any one of sending an image using electronic mail, printing it out, copying the image to another medium, or a combination thereof.

  Preferably, the first and second storage units store maps relating to image feature amount distributions of images of different persons.

  According to still another aspect of the present invention, an image management method is obtained by storing a map in which a distribution of predetermined image feature values and predetermined key information are associated with each other and analyzing a plurality of given images. By referring to the stored map based on the image feature amount, an association step for associating the image group with the key information, a distribution of the image feature amount existing in the image group, and a map in which the key information is associated is newly added. And a creation step for creating.

  According to still another aspect of the present invention, an image management method stores a first map in which a predetermined image feature amount distribution and predetermined key information are associated with each other, and another predetermined image feature amount distribution; The second map associated with another predetermined key information is stored, and the first map is referred to based on the image feature amount obtained by analyzing the given plurality of images. A first association step for associating the image group and the first key information, and a second map with reference to the second map on the basis of the image feature amount obtained by analyzing the given plurality of images. The first key information and the second key information based on the second association step for associating the image group and the second key information, and the degree of overlap between the first image group and the second image group. A third associating step.

  According to still another aspect of the present invention, the image management program stores a map in which a predetermined image feature amount distribution and predetermined key information are associated with each other, and is obtained by analyzing a plurality of given images. By referring to the stored map based on the image feature amount, an association step for associating the image group with the key information, a distribution of the image feature amount existing in the image group, and a map in which the key information is associated is newly added. And causing the computer to execute a creation step.

  According to still another aspect of the present invention, the image management program stores a first map in which a predetermined image feature amount distribution and predetermined key information are associated with each other, and another predetermined image feature amount distribution; The second map associated with another predetermined key information is stored, and the first map is referred to based on the image feature amount obtained by analyzing the given plurality of images. A first association step for associating the image group and the first key information, and a second map with reference to the second map on the basis of the image feature amount obtained by analyzing the given plurality of images. The first key information and the second key information based on the second association step for associating the image group and the second key information, and the degree of overlap between the first image group and the second image group. Causing the computer to execute a third associating step

[Principle of image search]
The main image search methods are roughly classified into the following three.

  (1) A method of manually adding a keyword to an image in a database in advance and searching for this keyword by a text search method.

  (2) A method of searching for an image having a similar image feature amount from a database by exemplifying an image very similar to an image to be searched.

  (3) A method of learning an image recognition classifier (image recognition dictionary) using an image to which a keyword as a learning sample is assigned in advance. The image recognition discriminator receives an image and outputs a keyword-like index. Alternatively, a keyword is input, and an image with a high keyword-likeness index is output.

  The technique of Non-Patent Document 1 described in the above-mentioned background art uses the technique (2). The technique of Patent Document 1 is a modification of the method (2). Moreover, the technique of nonpatent literature 2 uses the method of (3).

[First Embodiment]
In the image management apparatus and the image management program in the present embodiment, image search using the method (3) is performed. By using this type of technique, it is possible to select a keyword having a high index indicating the likelihood of a keyword for a given image and assign it to the image (automatic annotation). Thereby, an image corresponding to the input keyword can be output, or an image can be input and a title suitable for the image can be output (a keyword is assigned).

  The image recognition discriminator inputs image feature amounts such as a normalized color histogram and an edge orientation histogram obtained by image processing.

  As the learning of the image recognition classifier, for example, the technique described in Non-Patent Document 2 can be employed. That is, for a given image feature amount, a posterior probability distribution in which a certain keyword is attached to the image is defined and obtained using an EM algorithm or the like. The posterior probability is a kind of index of keyword likeness for a given image feature amount.

  In Non-Patent Document 2, learning is performed on the feature quantities of the divided areas. However, in this embodiment, the area division is not particularly performed for the sake of brevity. For learning that does not perform area division, a technique in which the result of area division is only one of the entire image in the method described in the cited document 2 can be applied. Of course, region division may also be performed in this embodiment.

  FIG. 1 is a diagram showing a configuration of an image management system according to the first embodiment of the present invention.

  Referring to the figure, the image management system includes an image forming apparatus 1 such as an MFP (Multi Function Peripherals), client PCs 2a, 2b,. The image forming apparatus 1, client PCs 2a, 2b,... And the server 6 are connected via a network.

  The image forming apparatus 1 is an apparatus that forms a copy image of a scanned original image, client PCs 2a, 2b,... And an image generated from print data transmitted from the server 6 on a sheet.

  FIG. 2 is a block diagram illustrating a hardware configuration of the image forming apparatus 1 of FIG.

  Referring to the figure, an image forming apparatus 1 includes a control unit 106 that controls the entire apparatus, an image reader unit 101 that reads image data from a document, a printer unit 102 that prints an image on a sheet, and a short-range wireless communication. A communication unit 103 for performing communication and connecting a printing apparatus to a network or a telephone line, a storage unit 104 for storing job data, an operation panel 105 serving as an interface with a user, and a remaining consumable And a sensor unit 107 that detects the amount and the like.

  FIG. 3 is a block diagram showing a hardware configuration of the client PC or one server in FIG.

  Referring to the figure, a client PC (or server) includes a CPU 601 for controlling the entire apparatus, a display 605, and a LAN (local area network) card 607 (or a modem for connecting to a network or communicating with the outside). Card), an input device 609 including a keyboard and a mouse, a flexible disk drive 611, a CD-ROM drive 613, a hard disk drive 615, a RAM 617, and a ROM 619.

  Data such as a program recorded on the flexible disk F can be read by the flexible disk drive 611, and data such as a program recorded on the CD-ROM 613a can be read by the CD-ROM drive 613. .

  In the present embodiment, managed images are stored in the image forming apparatus 1, the client PCs 2 a, 2 b,.

  FIG. 4 is a flowchart showing operations performed by the computer of the image management system.

  The process of each step will be described.

Step S101 (initial learning (default recognition dictionary learning procedure))
In this step, learning sample images to which appropriate (plural) keywords are attached are prepared by the user.

  The image management system inputs a plurality of learning sample images to which keywords are attached as initial learning. An image feature amount f (a normalized color histogram, an edge orientation histogram, etc.) of the sample image is obtained. According to the method shown in Non-Patent Document 2, the distribution function P (k | f) of the posterior probability of the keyword k with respect to a predetermined image feature amount f is obtained for each keyword k using an EM algorithm or the like.

  The distribution function P (k | f) of the posterior probability is the probability that the keyword k is attached to the image when there is the image feature amount f in all the learning samples. Describes the range.

  The obtained posterior probability distribution function is stored for each keyword.

  It should be noted that the initial learning process may be omitted by introducing the distribution function P (k | f) of the posterior probability of the keyword k with respect to the image feature quantity f, which is calculated and prepared in advance, to the image management system.

Step S103
If there is an input of a photographic image taken by the user with a digital camera or an image captured from a scanner or an external device (an image to be managed and also referred to as a “user image”), the process proceeds to step S105.

Step S105
The input image (s) are registered in an image database in the image management system.

Step S107 (initial internal annotation)
The computer obtains and stores a predetermined image feature amount fi for each image i in the image database in which user images are stored.

  The computer refers to the stored posterior probability distribution function P (k | f) for each keyword for each image i, so that the posterior probability P (k | fi) of each keyword with respect to the image feature quantity fi. Ask for. N keywords with the highest posterior probabilities are obtained and associated as keywords of the corresponding image.

Step S109 (learning related feature values)
For each keyword k, the computer collects images associated with the keyword. For each keyword k, the computer tentatively attaches the relevant keyword only to the collected images, and uses the data of the entire image database (or a part that is appropriately sampled) to use the distribution function Q (k of the posterior probability in the user image. | F) is obtained. At this time, in order to eliminate noisy image features, portions below a predetermined threshold are cut off to “0”.

  For each keyword, the computer fuses the posterior probability distribution function Q (k | f) for the user image with the original posterior probability distribution function P (k | f). Fusion is performed by synthesizing the larger probability of each feature quantity of the posterior probability distribution function as the probability of the feature quantity.

  The posterior probability distribution function P ′ (k | f) updated by the fusion is overwritten on the original P (k | f) and stored.

Step S111 (update internal annotation)
For each image, the computer refers to the updated posterior probability distribution function P ′ (k | f) for each keyword, and for each stored image feature quantity fi, the posterior probability P ′ (k) of each keyword. | Fi). Then, select N keywords with the highest posterior probability.

  The computer updates and stores the association for each image from the selected keyword.

  Note that the processing in steps S105 to S111 constitutes a user recognition dictionary learning procedure.

Step S113
When an image search instruction is received from the user, the process proceeds to step S115.

Step S115 (search procedure)
The user inputs a search keyword and instructs to search for an image related to the computer.

Step S117
The computer reads an image associated with the same keyword as the search keyword and displays it on the screen.

  As described above, in the present embodiment, in step S109, a plurality of image feature quantities that frequently appear in images searched with the same keyword can be associated with each other. This has the effect of automatically customizing the dictionary for personal use.

  Next, operations performed by the computer of the image management system will be described with reference to FIG.

  Referring to FIG. 5, as a result of initial learning, in the default recognition dictionary, a typical soccer-related image feature such as a keyword “Old keyword”, an image feature amount class corresponding to a soccer ball, and the like. A quantity class is tied. That is, the relationship between the feature quantity f of the image and the keyword k is indicated by the distribution function P (k | f) of the initial posterior probability.

  It is assumed that a user image such as a photograph taken by a user includes many images of members of a soccer team to which the son of the user kicking the soccer ball belongs. At this time, the class of the image feature corresponding to the person wearing the uniform of the soccer team is newly associated with the keyword “soccer” and registered in the dictionary by the above-described procedure for learning related features.

  As a result, when a user's image database is searched using “soccer” as a search keyword, it is possible to find an image that does not include a soccer ball but includes a person wearing the uniform of the soccer team.

[Second Embodiment]
The initial learning performed by the image management system and the image management program and the processing in the user recognition dictionary learning procedure in the second embodiment of the present invention are the same as those in the first embodiment, and therefore will be described here. Do not repeat.

  In this embodiment, an image search based on a keyword is not performed, and an appropriate title (related keyword) is displayed (automatic annotation) for an arbitrary user image.

  That is, the following title display process is performed instead of the image search process.

  FIG. 6 is a flowchart showing title display processing executed by the computer.

  In step S201, the user selects an image on which a title is displayed.

  In step S203, the computer displays a keyword associated with the selected image on the screen together with the image.

  Thus, in the example of FIG. 5, the keyword “soccer” is automatically annotated even for an image including a person wearing the uniform of the soccer team, although the soccer ball is not included.

[Third Embodiment]
In the above description, the keyword for searching for an image is stored in association with the image. However, the following may be used.

  In the first and second embodiments, the keyword is used as a label for identifying an image. Therefore, if it can be used as a label, it can be used instead of a keyword.

  For example, instead of a keyword, another image, voice (such as a human voice or music song), symbol (such as a pictograph), telephone number, e-mail address, or the like can be associated.

[Fourth Embodiment]
FIG. 7 is a flowchart showing operations performed by the computer of the image management system according to the fourth embodiment of the present invention.

  Hereinafter, each step shown in FIG. 7 will be described in order.

Step S301 (user recognition dictionary learning procedure)
In this step, the user recognition learning dictionary is learned by performing the following processing.

  (1) The user inputs the name of person A and its e-mail address. Then, a plurality of photographs showing the person A are designated from the image database in which a plurality of photographs are accumulated.

  (2) The computer stores the name of the person A in association with the e-mail address.

  (3) The computer uses the entire image database as a learning sample and distributes the posterior probability of the keyword with respect to a predetermined image feature amount f for each keyword (here, the name of a person) by the method described in the paper of Non-Patent Document 2 above. A function P (k | f) is obtained using an EM algorithm or the like.

  In order to describe the relationship between the image feature quantity and the specific person-likeness, other classifiers for identifying individuals may be used without using the posterior probability distribution function. In this case, it is necessary to learn the parameters of the classifier according to the person. Classifier parameters are associated with keywords instead of posterior probability distribution functions.

  (4) The computer stores the obtained posterior probability distribution function for each keyword.

  (5) The user further inputs the names of persons B, C, D,... And the e-mail address, and repeats the processes (1) to (3).

Step S303 (keyword association)
In this step, the following processing is performed to associate the keywords with each other.

  (1) The computer obtains and stores a predetermined image feature amount fi for each image i in the image database in which the user's photographs are accumulated.

  (2) The computer refers to the stored posterior probability distribution function P (k | f) for each keyword for each image i, and determines the posterior probability P of each keyword for the obtained image feature quantity. Find (k | fi). Then, a keyword whose posterior probability is equal to or higher than a predetermined threshold is selected.

  (3) The computer stores the selected keyword in association with each image.

  (4) The computer analyzes a keyword associated with an image in the image database and detects a set of keywords whose frequency associated with the same image is equal to or higher than a predetermined threshold.

  (5) For each keyword set detected as having a high frequency of being associated together, the computer adds an e-mail address stored in association with each keyword to other keywords in the same set. To associate and memorize.

  However, the additional e-mail address is stored separately from the original associated e-mail address.

Step S305
When the image is transmitted by e-mail, the processing after step S307 is performed.

Step S307 (image sending procedure)
In this step, the following processing is performed.

  (1) The user selects an image (or person) and instructs the computer to send the image.

  (2) The computer sends the image to the e-mail address stored in association with the image, and also sends the same image to the e-mail address stored in association with the image.

  In this way, in the present embodiment, when the image group searched for by keyword A and the image group searched for by keyword B largely overlap, keywords A and B can be associated with each other.

  When the image group searched for person A and the image group searched for person B overlap each other, person A and B are associated with each other, and when an image of person A (or to person A) is sent by mail, person B Are also sent together (or sent to person B). In this way, operations on related images can be performed collectively.

  An e-mail address may be used directly as a keyword.

  In addition to the e-mail transmission process, when an image is printed out for distribution (or an image CD-R is burned (an image is copied on a medium)), it is also printed on an associated person (or The present invention can be carried out when CD-R creation) is additionally performed.

  The processing shown in the flowchart may be performed by a PC computer, an image forming apparatus computer, or a server computer.

  Note that the processing in the above-described embodiment may be performed by software or by using a hardware circuit.

  Moreover, it is good also as providing the apparatus which combined some of the above-mentioned embodiment arbitrarily.

  In addition, a program for executing the processing of the flowchart in the above-described embodiment can be provided, and the program can be recorded on a recording medium such as a CD-ROM, a flexible disk, a hard disk, a ROM, a RAM, a memory card, and the like. You may decide to provide it. The program may be downloaded to the apparatus via a communication line such as the Internet.

  Further, the present invention can be applied to a system connected to a network and a system not connected to a network environment.

  In addition, it should be thought that the said embodiment is an illustration and restrictive at no points. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

It is a figure which shows the structure of the image management system in the 1st Embodiment of this invention. FIG. 2 is a block diagram illustrating a hardware configuration of the image forming apparatus 1 in FIG. 1. FIG. 2 is a block diagram showing a hardware configuration of one client PC or server in FIG. 1. It is a flowchart which shows the operation | movement which the computer of an image management system performs. It is a figure for demonstrating the operation | movement which the computer of an image management system performs. It is a flowchart which shows the title display process which a computer performs. It is a flowchart which shows the operation | movement which the computer of the image management system in the 4th Embodiment of this invention performs.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Image forming apparatus, 2a, 2b Client PC, 6 server, 102 Printer part, 103 Communication part, 104 Storage part, 105 Operation panel, 106 Control part, 601 CPU.

Claims (13)

Storage means for storing a map in which a distribution of a predetermined image feature amount and predetermined key information are associated;
An associating means for associating a group of images with key information by referring to the stored map based on an image feature obtained by analyzing a plurality of given images;
An image management apparatus comprising: a creation unit that newly creates a map in which a distribution of image feature quantities existing in the image group and key information are associated with each other.   The image management apparatus according to claim 1, wherein the creating unit creates a new map in consideration of the stored map information.   The image management apparatus according to claim 1, further comprising an output unit that refers to the newly created map and outputs an image group associated with the key information from a plurality of given image groups. The storage means stores a map group consisting of a plurality of sets of image feature quantity distribution and key information,
The creating means creates a map group consisting of a plurality of sets of image feature quantity distribution and key information,
The image management apparatus according to claim 1, wherein key information related to a given image is output by referring to a map group.   The image management apparatus according to claim 1, wherein the key information is any one of a keyword, an image, a sound, a symbol, a telephone number, an e-mail address, or a combination thereof. First storage means for storing a first map in which a distribution of a predetermined image feature amount and predetermined key information are associated;
Second storage means for storing a second map that associates another predetermined image feature amount distribution with another predetermined key information;
First associating means for associating a first image group and first key information with reference to the first map based on an image feature obtained by analyzing a plurality of given images; ,
Second associating means for associating a second image group and second key information with reference to the second map based on an image feature obtained by analyzing a plurality of given images; ,
An image management apparatus comprising: a third association unit that associates the first key information with the second key information based on a degree of overlap between the first image group and the second image group. .   The image management apparatus according to claim 6, wherein when an operation is performed on the first key information, an operation associated with the second key information is automatically performed on the second key information.   The image management apparatus according to claim 7, wherein the operation is any one of sending an image using electronic mail, printing out, copying an image to another medium, or a combination thereof.   9. The image management apparatus according to claim 6, wherein the first storage unit and the second storage unit store maps relating to distributions of image feature amounts of images of different persons. Storing a map in which a distribution of a predetermined image feature amount is associated with predetermined key information;
Associating the image group with the key information by referring to the stored map based on the image feature amount obtained by analyzing the plurality of given images;
An image management method comprising: a creation step of newly creating a map in which a distribution of image feature amounts existing in the image group is associated with key information. Storing a first map associating a predetermined image feature amount distribution with predetermined key information;
Storing a second map associating another predetermined image feature amount distribution with another predetermined key information;
A first associating step of associating a first image group and first key information with reference to the first map based on an image feature obtained by analyzing a plurality of given images; ,
A second associating step of associating a second image group and second key information with reference to the second map based on an image feature obtained by analyzing a plurality of given images; ,
An image management method comprising: a third association step for associating the first key information and the second key information based on a degree of overlap between the first image group and the second image group. . Storing a map in which a distribution of a predetermined image feature amount is associated with predetermined key information;
Associating an image group with key information by referring to the stored map based on an image feature obtained by analyzing a plurality of given images;
An image management program for causing a computer to execute a creation step of newly creating a map in which a distribution of image feature amounts existing in the image group and key information are associated with each other. Storing a first map associating a predetermined image feature amount distribution with predetermined key information;
Storing a second map associating another predetermined image feature amount distribution with another predetermined key information;
A first associating step of associating a first image group and first key information with reference to the first map based on an image feature obtained by analyzing a plurality of given images; ,
A second associating step of associating a second image group and second key information with reference to the second map based on an image feature obtained by analyzing a plurality of given images; ,
An image that causes a computer to execute a third association step of associating the first key information and the second key information based on a degree of overlap between the first image group and the second image group. Management program.

Images