JP5650628B2 - Image dictionary generation device, image dictionary generation method, and image dictionary generation program - Google Patents

Description

  The present invention relates to an image dictionary generation apparatus, an image dictionary generation method, and an image dictionary generation program that generate an image dictionary that is necessary when recognizing what concept an image or video has.

  In order to realize image database construction and image / video similarity search, it is necessary to generate an image dictionary related to a semantic concept, which is a word representing a concept. However, in many cases, a sufficient amount of learning image data for generating an image dictionary that matches each semantic concept cannot be obtained. For example, when learning image data is collected manually, it is difficult to collect a sufficient amount because it requires a lot of cost and time for visual selection. In order to make up for the shortage of learning image data, one solution is to use image data of a domain (hereinafter referred to as an original domain) different from a domain (hereinafter referred to as a target domain) to which the learning data belongs. For example, when creating an image dictionary for Japanese broadcast video, if there is insufficient learning data (target domain data) on semantic concepts, data with the same semantic concept in other domains such as American broadcast video and web images If (original domain data) is used, learning data can be supplemented.

  As a conventional technology, TRECVID (an international joint research project sponsored by the National Institute of Standards and Technology (NIST) and Disruptive Technology Office (NIST) aiming at the advancement of video search and video analysis technology for that purpose; TRECVID learning keyframe images related to semantic concepts (target domain) when generating an image dictionary for video provided by http://www-nlpir.nist.gov/projects/tv2011/tv2011.html#data Data) and a web image (original domain data) collected from the web using the semantic concept as a query key are simply mixed to obtain learning data (for example, see Non-Patent Document 1).

  Also, when generating an image dictionary for TRECVID video, first analyze the feature distribution of the web image collected in the same way as in Non-Patent Document 1 in the feature amount space and the learning key frame image related to the semantic concept of TRECVID. In addition, it is also known that weighting is performed for each web image and learning data is combined with the web image and the learning key frame image (for example, see Non-Patent Document 2).

IVA-NLPR-IA-CAS TRECVID 2009: High Level Features Extraction www-nlpir.nist.gov/projects/tvpubs/tv9.papers/iva-nlpr-ia-cas.pdf Columbia University / VIREO-CityU / IRIT TRECVID2008 High-Level Feature Extraction and Interactive Video Search http://www-nlpir.nist.gov/projects/tvpubs/tv8.papers/columbia.pdf

  However, even if images from different domains represent the same semantic concept, they often have different distributions in the feature space. For example, a key frame image of a video relating to “telephone” is an image in which a small telephone is placed on a desk in an office, whereas a web image is clearly large at a central position. It is copied. That is, if the image of the “phone” is expressed in the color and size feature space, the feature distribution of the video key frame image and the web image will be different.

  As described above, Non-Patent Document 1 has a problem that the accuracy of the generated image dictionary is extremely low because image data having different feature distributions are simply combined and used as learning data. Further, Non-Patent Document 2 considers different feature distributions of image data of different domains and weights each web image so that the feature distribution of the web image (original domain data) is represented in the key frame image of the video in the feature section. I am trying to fit the distribution of (target domain data), but I do not consider which part of the feature distribution is different in the web image, so I have to add learning data that combines the web image and the keyframe image of the video. There is a problem that the accuracy of the image dictionary generated by using the image dictionary is low.

  For example, as shown in FIG. 4, the meaning concept of “telephone” is composed of parts such as “telephone”, “hand”, and “desk”. When the semantic concept is expressed in the feature space of color and size, even if the “telephone” in the TRECVID domain and the web image domain have different feature distributions, the portion such as “desk” has a similar feature distribution. In this case, there is a problem that even if a uniform weight is given to each image, the image distribution of the original domain cannot be fitted to the distribution of the target domain.

  The present invention has been made in view of such circumstances, and an image dictionary generation capable of accurately generating an image dictionary necessary for recognizing what kind of concept an image or video content is. An object is to provide an apparatus, an image dictionary generation method, and an image dictionary generation program.

  The present invention is an image dictionary generation device that generates an image dictionary used when recognizing the contents of an image, and includes collecting an image of a target domain that is a domain to which learning data related to a semantic concept that is a word representing a concept belongs, Image collection means for collecting an image of an original domain that is a domain to be referred to, which is different from the target domain, and calculates each topic in the image of the target domain and the original domain, information on the calculated topic and an image belonging to the topic A topic calculation unit that outputs information and a topic of the original domain output by the topic calculation unit are associated with a topic of the target domain, and information on the associated topic and a topic that outputs image information belonging to the topic Associating means and for each topic related to the topic of the original domain Image information combining means for fitting image information to a feature distribution of an image related to the topic of the target domain and outputting the target domain, topic information of the original domain, and image information for each topic, the target domain, and the target domain It is characterized by comprising dictionary generating means for generating an image dictionary representing a semantic concept from the topic information of the original domain and the image information for each topic.

  The present invention is an image dictionary generation method for generating an image dictionary used when recognizing the contents of an image, the image collection of a target domain that is a domain to which learning data related to a semantic concept that is a word representing a concept belongs, An image collection step for collecting an image of an original domain that is a domain to be referred to, which is different from the target domain, and calculates each topic in the image of the target domain and the original domain, and information on the calculated topic and an image belonging to the topic A topic calculation step for outputting information, and a topic in the original domain output in the topic calculation step in association with a topic in the target domain, and information on the associated topic and a topic for outputting image information belonging to the topic Mapping step, and for each topic, the original domain An image information combining step for fitting image information related to a pick to a feature distribution of an image related to a topic of the target domain and combining the target domain, topic information of the original domain, and image information for each topic, and the target domain And a dictionary generation step of generating an image dictionary representing a semantic concept from the topic information of the original domain and the image information for each topic.

  The present invention is an image dictionary generation program that causes a computer to function as the image dictionary generation device.

  According to the present invention, it is possible to obtain an effect that an image dictionary necessary for recognizing what concept the contents of an image or video have can be generated with high accuracy.

It is a block diagram which shows the structure of one Embodiment of this invention. It is a flowchart which shows operation | movement of the apparatus shown in FIG. It is explanatory drawing which shows an example of a distance matrix. It is explanatory drawing which shows an example of a semantic concept.

  Hereinafter, an image dictionary generating apparatus according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the embodiment. As shown in this figure, the image dictionary generation device is composed of an image storage unit 1, an image collection unit 2, a topic calculation unit 3, an association unit 4, an image information synthesis unit 5, a dictionary generation unit 6, and a dictionary storage unit 7. The

  The image storage unit 1 stores and manages images prepared in advance for each semantic concept for each domain. When the image storage unit 1 receives the designation of the semantic concept and the domain from the image collection unit 2, the image storage unit 1 outputs an image of the target domain and the original domain related to the semantic concept to the image collection unit 2. When outputting the semantic concept and domain designation to the image storage unit 1, the image collection unit 2 receives the images of the target domain and the original domain related to the stored semantic concept, and outputs the received image to the topic calculation unit 3.

  The topic calculation unit 3 receives the image of the target domain and the image of the original domain from the image collection unit 2, and performs respective topic calculations on the image of the target domain and the image of the original domain. Then, the topic calculation unit 3 outputs the individual topic information related to the target domain image and the original domain image and the image information belonging to the topic to the association unit 4. The association unit 4 receives the topic information of each domain from the topic calculation unit 3, and associates the topic of the original domain with the topic of the target domain. Then, the associating unit 4 outputs the associated topic information and the image information belonging to the topic to the image information combining unit 5.

  The image information synthesizing unit 5 receives the topic information associated with both domains and the image information related to the topic from the associating unit 4, and uses the associated topic information for each topic in the image of the original domain. The feature distribution is fitted to the feature distribution of the target domain image, and the image information of both domains is matched. Then, the image information synthesis unit 5 outputs the combined topic information and the image information related to the topic to the dictionary generation unit 6.

  The dictionary generation unit 6 receives topic information of both domains combined from the image information synthesis unit 5 and image information related to the topic, and generates a topic model using a learning method. The dictionary generation unit 6 combines the topic models and stores topic information and an image dictionary in the dictionary storage unit 7 as an image dictionary representing a semantic concept.

  The basic unit that expresses the meaning of an image can be considered as the topic of the image. Here, a topic is associated with a region or a set of pixels obtained by a method such as image region division or clustering. If a highly accurate image region segmentation method is used, it can be considered that an image region can correspond to a real-world object (eg, sky, wheel, human face, torso, etc.).

  Next, the processing operation of the image dictionary generation device shown in FIG. 1 will be described with reference to FIG. FIG. 2 is a flowchart showing the processing operation of the image dictionary generating apparatus shown in FIG. First, the image collection unit 2 reads the images of the target domain and the original domain related to the semantic concept from the image storage unit 1 and outputs them to the topic calculation unit 3 (step S1). Receiving this, the topic calculation unit 3 performs topic extraction on the target domain image and the original domain image, and outputs the extracted topic information to the association unit 4 (step S2).

  Here, topic extraction processing will be described using an image of the original domain as an example. First, feature amounts are extracted from individual images in the original domain. For example, a SIFT feature point of an image may be extracted as a feature amount. Then, clustering is performed on all SIFT feature points in the original domain image. Finally, a cluster having a certain feature point or more is set as a topic of the original domain. In addition, using an image region dividing method, region division is performed on individual images in the original domain, and clustering is performed on the obtained region group. A cluster having a certain area or more may be used as a topic of the original domain. This same process can also extract the topic of the target domain. Topic information about domain images is the center of the individual clusters obtained. That is, the average value of all feature quantities in the cluster. Image information belonging to a topic is a cluster.

  Next, the associating unit 4 associates the topic of the original domain with the topic of the target domain using the topic information of both domains output from the topic calculating unit 3 (step S3). The association process is performed by the processes of steps S31 to S36. The associating unit 4 reads the topic information of the original domain (M) and the topic information of the target domain (N) (step S31). Then, the associating unit 4 determines the distance dmn, m = 1. . . M, n = 1. . . N is calculated (step S32). Here, dmn may be calculated as the Euclidean distance between the mth topic in the original domain and the nth topic in the target domain. The distance between topics in both domains can be expressed using the distance matrix shown in FIG.

  Next, the associating unit 4 sets 1 to the variable m (step S33). Then, the associating unit 4 obtains the minimum value dmj of the distance in the m-th row and determines whether or not the minimum value dmj is smaller than a certain value (D1) (step S34). If the minimum value dmj is not smaller than the constant value (D1) as a result of this determination, 1 is added to the variable m and the process is repeated until the minimum value dmj is smaller than the constant value (D1).

  Next, when the minimum value dmj becomes smaller than the fixed value (D1), the associating unit 4 associates the mth topic in the original domain with the jth topic in the target domain (step S35). Then, the associating unit 4 determines whether m = M is satisfied (step S36). If m = M is not satisfied, 1 is added to the variable m, the process returns to step S34, and the process is repeated. The association unit 4 outputs the topic information associated with both domains and the image information belonging to the topic to the image information synthesis unit 5 when m = M.

  By the processing of steps S31 to S36, the topic of the target domain having a large distance, that is, the topic of the unrelated original domain is filtered, and only the topic of the original domain that is near the topic of the target domain is selected and associated. Good quality learning data can be selected from the original domain according to the semantic concept of the target domain. Using these topics and the image information related to the topics, the accuracy of the image dictionary in the target domain is improved.

  Next, the image information composition unit 5 fits the feature distribution of the image information of both domains for each topic using the topic information associated with both domains output from the association unit 5 and the image information belonging to the topic. , And outputs the result to the dictionary generator 6 (step S4). There are the following synthesis methods (1) to (3).

(1) When there is no topic in the original domain associated with topic i in the target domain, the learning data related to the i-th topic is a cluster belonging to the i-th topic. For example, when the SIFT feature point is an image feature amount, the learning data regarding the i-th topic is a cluster of SIFT feature points belonging to the i-th topic.

(2) If there is one topic (originally j) in the original domain associated with topic i in the target domain, it is first determined whether dij is below a certain level. If it becomes below a certain level, the learning data related to the i-th topic is obtained by combining the cluster belonging to the i-th topic in the target domain and the cluster belonging to the j-th topic in the original domain. If it exceeds a certain level, the probability distribution of the image information belonging to both topics is analyzed, the cluster belonging to the jth topic in the original domain is fitted to the cluster belonging to the ith topic in the target domain, and then the ith topic By combining with the clusters belonging to, it becomes learning data regarding the i-th topic. For example, after obtaining a cul divergence (KL divergence) representing a difference in probability distribution between both topics, a cluster belonging to the jth topic in the original domain is fit to a cluster belonging to the ith topic in the target domain. The learning data related to the i-th topic may be obtained.

(3) When there are a plurality of topics (j, l) in the original domain associated with the topic i in the target domain, it is first determined whether or not dij and dir are all below a certain level. If it falls below a certain level, the cluster belongs to the i-th topic in the target domain and the clusters belonging to the j and l-th topics in the original domain are used as learning data for the i-th topic. If it becomes a certain value or more, as in the case of No in the above (2), the clusters for the i-th topic and the j-th topic are combined in order, and the cluster for the new i-th topic and the l-th topic Align clusters on topics.

  Next, the dictionary generation unit 6 generates N topic models using the learning method for the topic domain topic information output from the image information synthesis unit 5 and the clusters related to the topics. As a learning method used here, a known method can be applied, and for example, SVM can be used. Then, the dictionary generation unit 6 sets the N topic models as an image dictionary related to the semantic concept model (step S5). The dictionary generation unit 6 stores the image dictionary and N pieces of topic information in the dictionary storage unit 7. When a semantic concept is identified for an unknown image, first, N topics are generated in the image by similarity matching between the feature amount of the image and topic information. Then, an individual identification result is obtained for the topic of the generated image using an image dictionary. Finally, the final identification of the image is performed by combining the N identification results. For example, the average of N identification results may be the final identification result.

  As explained above, after analyzing the feature distribution of the image data of different domains for each topic, it is associated with the topic of the image data of the original domain and the topic of the image data of the target domain. By fitting the feature distribution to the feature distribution of the image information of the target domain and combining them, high-quality learning data can be collected in the target domain, and an image dictionary generated using the learning data can be realized with high accuracy.

  Note that an image dictionary generation process is performed by recording a program for realizing the function of the processing unit in FIG. 1 on a computer-readable recording medium, causing the computer system to read and execute the program recorded on the recording medium. May be performed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer system” includes a WWW system having a homepage providing environment (or display environment). The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” refers to a volatile memory (RAM) in a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, those holding programs for a certain period of time are also included.

  The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

  As mentioned above, although embodiment of this invention has been described with reference to drawings, the said embodiment is only the illustration of this invention, and it is clear that this invention is not limited to the said embodiment. is there. Accordingly, additions, omissions, substitutions, and other modifications of components may be made without departing from the spirit and scope of the present invention.

  The present invention can be applied to applications where it is indispensable to generate an image dictionary necessary for recognizing what kind of concept the contents of an image or video are.

  DESCRIPTION OF SYMBOLS 1 ... Image storage part, 2 ... Image collection part, 3 ... Topic calculation part, 4 ... Association part, 5 ... Image information synthetic | combination part, 6 ... Dictionary generation part, 7 ... Dictionary storage

Claims (3)

An image dictionary generation device that generates an image dictionary used when recognizing the contents of an image,
Image collection means for collecting an image of a target domain that is a domain to which learning data related to a semantic concept that is a term representing a concept belongs, and collecting an image of an original domain that is a domain to be referred to different from the target domain;
Topic calculation means for calculating each topic in the images of the target domain and the original domain, and outputting the calculated topic information and image information belonging to the topic;
Associating the topic of the original domain output by the topic calculation means with the topic of the target domain, information on the associated topic, and topic association means for outputting image information belonging to the topic;
An image for outputting image information about the target domain, the topic information of the original domain, and image information for each topic by combining the image information related to the topic of the original domain with the feature distribution of the image related to the topic of the target domain for each topic Information synthesis means;
An image dictionary generating apparatus comprising: dictionary generating means for generating an image dictionary representing a semantic concept from the topic information of the target domain and the original domain, and image information for each topic. An image dictionary generation method for generating an image dictionary used when recognizing the contents of an image,
An image collecting step of collecting an image of a target domain that is a domain to which learning data related to a semantic concept that is a word representing a concept belongs, and collecting an image of an original domain that is a domain to be referred to different from the target domain;
A topic calculation step of calculating respective topics in the images of the target domain and the original domain, and outputting the calculated topic information and image information belonging to the topic;
Associating the topic of the original domain output by the topic calculation step with the topic of the target domain, the topic association step of outputting information of the associated topic and image information belonging to the topic;
An image for outputting image information about the target domain, the topic information of the original domain, and image information for each topic by combining the image information related to the topic of the original domain with the feature distribution of the image related to the topic of the target domain for each topic An information synthesis step;
An image dictionary generation method comprising: a dictionary generation step of generating an image dictionary representing a semantic concept from the topic information of the target domain and the original domain, and the image information for each topic.   An image dictionary generation program for causing a computer to function as the image dictionary generation apparatus according to claim 1.

Images