JP2010067014A - Image classification device and image classification method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a customizable image classification device and method for classifying images by using whole images while suppressing the labor of a user to the minimum. <P>SOLUTION: The image classification device 10 includes an image registration device 20. The image registration device 20 includes: a first similarity calculation means 21 for calculating the similarity of images with respect to an input image in the local region of an image; a first image DB22 for storing the data of an image to which a keyword which is concrete for a user is attached as a tag; a first similarity decision means 23 for making first similarity decision; a second similarity calculation means 24 for calculating the similarity of the whole images for the input image; a second image DB25 for storing the data of the image to which the tag of an ambiguous keyword is attached; and a second similarity decision means 26 for making second similarity decision. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image classification device and an image classification method for comparing and classifying images when accumulating images taken with, for example, a digital still camera.

  In recent years, with the explosive spread of digital still cameras and the increase in storage capacity mounted on digital still cameras, there has been a significant change in general photography methods. In other words, the conventional silver halide camera can only shoot about 20 times with a single film change, and the photograph cannot be retaken, so the user examines the scene, person, and timing to shoot. By pressing the shutter at the scene, only the carefully selected photos were acquired.

  On the other hand, with a digital still camera, it is possible to shoot hundreds of images, check the captured images on the LCD monitor, and easily delete unnecessary images. Shooting methods are often used in which the shutter is released many times in the scene, the images are examined and selected afterwards, and the number of images held by the user is constantly increasing. Furthermore, the capacity of storage devices such as PC hard disks and optical discs that hold images taken with digital still cameras has been increasing year by year, and now users can hold astronomical numbers of images. ing. As a result, there are many users who hold a large amount of images in the storage device without selecting them.

  On the other hand, when such an enormous number of images are held in the storage area of the PC, it takes a long time to search for the necessary images. For example, as an example of a case where a necessary image has to be searched, there is a case where a photograph of a child photographed with a friend is selected and printed on the day of an athletic meet and distributed to the friend. In such a case, a user's general method of searching for an image is as follows.

  First, the image is narrowed down by the information added to the image (first narrowing down), the narrowed down images are displayed in a reduced size and arranged, and the image that seems to be a friend is narrowed down (second narrowing down). Finally, each image is enlarged and displayed, and a desired image is searched for (3rd narrowing) while confirming. Looking back at the past while checking the images one by one in this way is the best part of photography, but as mentioned earlier, when searching for images to distribute to others, efficient work is required. .

  At this time, if the image can be narrowed down sufficiently by narrowing down the image, particularly the first narrowing down, the burden on the user is not so great. However, among the additional information used in the first narrowing down, additional information that is automatically added without human intervention is generally indirect with respect to an image. For example, although the most common additional information is the shooting date and time, there are few users who accurately store the date and time of the event, and for such a user, the shooting date and time is only indirect information connecting the event and the image. . Further, as other general additional information, there is a shooting mode or the like, which is more indirect information. For example, even if the fact that the flash was used at the time of shooting was held as additional information, it was estimated from the information whether the user was shooting indoors, shooting at night, or the weather was bad There is a problem that it is necessary to do. As a result, in order to obtain effective additional information, the user must rely on manual classification. The classification means is roughly classified into two types: classification based on a directory structure, and annotation for tagging an image by some means.

  For the above reasons, various techniques have been proposed in order to automate classification by manual operation of the user (see, for example, Patent Document 1). Patent Document 1 proposes a technique for calculating the feature amount of the entire image and automatically classifying the image according to the feature amount. By using this technique, images are automatically classified, so that the user's manual work can be greatly reduced.

  Here, in order to classify a photographic image using this technique, it is necessary to determine an identification rule as to which feature quantity is classified into which category. Various means can be considered as a method of determining the identification rule. For example, a method of classifying an image into an image category having the highest similarity to the input image among images classified in advance can be considered. However, a photograph is often composed of various elements such as a background, a person, and a specific object. Even if the positional relationship of each element changes, the image feature amount changes greatly. As a result, for example, it is necessary to prepare images of various patterns just by classifying landscape pictures and portraits. Here, for example, instead of determining the degree of similarity between images one by one, a category / category such as SVM (Support Vector Machine) shown in Non-Patent Document 1 has a high generalization performance. If the tendency is calculated, the number of images to be prepared can be reduced, but it still requires a large number of images. Providing the user with such a large amount of images is heavy on the user.

  On the other hand, for example, when a service is provided by software in advance, the above-mentioned problem can be avoided by preparing various images at the time of software shipment. become unable. For example, even if it is a portrait, a picture of itself or its relatives has a special meaning for the user, and it can be easily imagined that the user wants to classify such an image. On the other hand, it is impossible to register the user's face when the software is shipped. As described above, it is difficult to classify images flexibly using the entire image.

On the other hand, a method of classifying by focusing on the local part of an image has been proposed (for example, see Patent Document 2). According to the technique described in Patent Document 2, when a specific object is extracted from an image and an object name is once added to the extracted object, when an object similar to the object with the object name is subsequently photographed, Similarly, the object name is added to the image. Compared with the above-described method of comparing the entire images, individual objects are extracted, so that their combination patterns are dramatically reduced and classification can be performed with a relatively small number of images. However, there is a high possibility that a user's own face or a relative's face is generally included in a large amount in a photograph held by the user, and the method described in Patent Document 2 cannot be sufficiently narrowed down. there were.
Japanese Patent No. 4036009 JP 2006-333443 A C. Cortes and VN Vapnik, "Support vector Networks," Machine Learning, vol.20, pp.273-297, 1995

  The present invention has been made in view of the above-described circumstances, and is an image classification device and an image classification that can be customized and can be classified using the entire image while minimizing the burden on the user. It aims to provide a method.

  The image classification device according to the present invention is an image classification device that classifies the input image by comparing the similarity between the image to which the keyword is assigned and the input image, and includes the first and the first registered with the keyword. First and second image data storage means for storing data of the second image, local area extraction means for extracting a local area of a predetermined size from the input image and the first image, and extraction First similarity calculating means for calculating the similarity between the input image and the first image in the local region, and the similarity between the entire image of the input image and the entire image of the second image. A second similarity calculating unit for calculating; and a keyword adding unit for adding a keyword to the input image based on the similarity calculated by the first and second similarity calculating units. ing.

  With this configuration, the image classification apparatus according to the present invention can perform detailed image classification by calculating the similarity in the local region and the entire image, and can be customized while minimizing the burden on the user. Classification using the whole can be performed.

  In the image classification device according to the present invention, the first image data storage unit stores image data registered by a user by assigning a specific keyword as data of the first image. have.

  With this configuration, the image classification device according to the present invention allows the user to assign a specific keyword to the input image, so that the input image can be customized.

  The image classification apparatus of the present invention has a configuration in which the second image data storage means stores image data to which a predetermined ambiguous keyword is attached as data of the second image. is doing.

  With this configuration, the image classification device of the present invention can calculate the similarity of the entire image using an ambiguous keyword. Further, if the data of the second image is registered at the time of factory shipment of the apparatus, the user can save time for registering the second image, and the burden on the user can be suppressed.

  Further, in the image classification device of the present invention, the local region extraction unit includes an extreme value pixel detection unit that detects an extreme value pixel whose feature value of the image indicates an extreme value, and each of the input image and the first image. A feature amount calculating means for calculating an image feature amount in the vicinity of the extreme pixel, a feature amount comparing unit for comparing the calculated feature amount, and a positional relationship of the extreme pixel in each of the input image and the first image An extreme pixel selection unit that selects an extreme pixel based on the image, and an image area corresponding to the first image is extracted from the input image based on the number of extreme pixels selected by the extreme pixel selection unit. And a corresponding area extraction unit.

  With this configuration, the image classification device of the present invention can extract an image region corresponding to the first image from the input image based on the number of extreme pixels selected by the extreme pixel selection unit.

  In the image classification device of the present invention, the first similarity calculation unit compares the first image with the image of the image area extracted by the corresponding area extraction unit, and calculates the similarity. It has the structure provided with the comparison means.

  With this configuration, the image classification device of the present invention can improve the similarity calculation accuracy by calculating the similarity based on the feature amount in the extracted image region.

  In the image classification device of the present invention, the extreme pixel detection unit detects an extreme pixel of an image when a user registers as the first image, and the extreme pixel detection unit The system includes a warning unit that gives a warning to the user when the number of detected extreme value pixels is equal to or less than a predetermined number.

  With this configuration, the image classification device of the present invention can prevent the same keyword from being assigned to a large number of registered images, and thus can improve user convenience.

  The image classification apparatus according to the present invention further includes an area designating unit for designating a specific area of the input image, and the first image data storage unit stores image data of the area designated by the area designating unit. It has the structure which is thing.

  With this configuration, the image classification apparatus of the present invention can specify only a partial area of the image, not the entire image, when registering the first image, so that the user can capture an image showing only a subject to which a keyword is to be assigned. There is no need to prepare, and the convenience of the user can be improved.

  The image classification apparatus according to the present invention further includes similarity determination means for determining whether or not the input image is similar to the first image based on the similarity calculated by the first similarity calculation means. And the second similarity calculation means calculates a similarity related to the entire image only when the input image and the first image are not similar.

  With this configuration, the image classification apparatus of the present invention uses the second similarity calculation unit when one keyword is given to one image and the input image and the first image are not similar. Since the processing can be omitted, the waiting time of the user can be shortened at the time of image registration, and the convenience for the user can be improved.

  The image classification method of the present invention is an image classification method for classifying the input image by comparing the similarity between the image to which the keyword is assigned and the input image, wherein the first and the first registered with the keyword are registered. Storing each of the second image data; extracting a local area of a predetermined size from the input image and the first image; and extracting the input image and the first in the extracted local area Calculating the similarity between the first image and the second image, calculating the similarity between the entire image of the input image and the entire image of the second image, and the similarity calculated by the first and second images Adding a keyword to the input image based on the degree.

  With this configuration, the image classification method of the present invention can perform detailed image classification by calculating the similarity in the local region and the entire image, and can be customized while minimizing the burden on the user. Classification using the whole can be performed.

  The present invention can provide an image classification apparatus and an image classification method that can be customized and can be classified using the entire image while minimizing the burden on the user. .

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the embodiment, an example in which the image classification device according to the present invention is applied to a web service system capable of uploading and managing image data will be described. This web service system includes a server PC and a plurality of client PCs. In the following description, an image to be uploaded from the client PC to the server PC is referred to as an input image.

(First embodiment)
First, the configuration of the image classification apparatus according to the first embodiment of the present invention will be described.

  As shown in FIG. 1, the image classification device 10 according to the present embodiment includes an image registration device 20 that registers input images, and a keyword registration device 30 that mainly registers keywords.

  The image registration apparatus 20 includes a first similarity calculation unit 21 that calculates a first similarity, a first image database (DB) 22 that stores data of a first search target image, and a first similarity A first similarity determination unit 23 that performs a determination, a second similarity calculation unit 24 that calculates a second similarity, a second image DB 25 that stores data of a second search target image, and a second And second similarity determination means 26 for performing the similarity determination. Note that the first image DB 22 and the second image DB 25 constitute first and second image data storage means according to the present invention, respectively. The first similarity determination unit 23 and the second similarity determination unit 26 constitute a keyword assignment unit according to the present invention.

  The keyword registration device 30 includes a region designation unit 31 for designating a specific region in the input image, an extreme pixel detection unit 32 for detecting extreme pixels, a registration unit 33 for registering keywords and images, and a warning to the user Warning means 34 for giving.

  The first search target image is an image searched from the first image DB 22 in order to calculate the similarity of the image to the input image in the local region of the image, and is an image registered and accumulated by the user. For example, the first search target image is an image to which a specific keyword is attached as a tag for a user such as “the face of the eldest son” or “the face of the child friend A”. The second search target image is an image searched from the second image DB 25 to calculate the similarity of the entire image with respect to the input image. For example, the image is registered and stored on the manufacturer side at the time of shipping the system. It is an image. For example, the second search target image is tagged with an ambiguous (in other words, abstract) keyword such as “portrait”, “landscape”, “animal”, “group photo”. It is an image.

  As for the first search target image, some sample image may be registered at the time of system shipment. Further, regarding the second search target image, the user may prepare and store an image for tagging an ambiguous keyword.

  As shown in FIG. 2, the first similarity calculation unit 21 includes a local region extraction unit 40 that extracts a local region, and an image comparison unit 50 that compares images. The local region extraction unit 40 includes a SIFT calculation unit 41 that calculates a SIFT (Scale-Invariant Feature Transform) of the input image, a SIFT calculation unit 42 that calculates a SIFT of the first search target image, and a SIFT comparison unit 43 that compares the SIFTs. , A corresponding point selecting unit 44 for selecting corresponding points, and a corresponding region extracting unit 45 for extracting corresponding regions. The SIFT calculation units 41 and 42 constitute an extreme pixel detection unit and a feature amount calculation unit according to the present invention. The SIFT comparison unit 43 constitutes a feature amount comparison unit according to the present invention. The corresponding point selection unit 44 constitutes an extreme pixel selection unit according to the present invention. In addition, the corresponding region extraction unit 45 constitutes a corresponding region extraction unit according to the present invention.

  FIG. 3 is a block diagram of a computer 60 used as a server PC and a client PC constituting the web service system in the present embodiment. The image classification apparatus 10 in the present embodiment is loaded on the computer 60 and the computer 60. It is realized by the program.

  In FIG. 3, a CPU 61 is a central processing unit, and performs overall control and arithmetic processing of the computer 60. The ROM 62 is a read-only memory and has a storage area for information such as system startup programs. The RAM 63 is a random access memory and has a data storage area. An operating system, a device driver, an application such as a web browser, and a program such as communication control are loaded on the RAM 63 and executed by the CPU 61. The input / output unit 64 includes input / output devices such as a keyboard and a mouse, and transmits information input by the user to the input / output device to the CPU 61. The display unit 65 includes, for example, a liquid crystal display, a display control unit, and the like. The HDD 66 is a hard disk device, and stores search target image data, a web browser program file, and the like. The communication unit 67 performs network communication control, and can communicate with other computers and peripheral devices connected to the network. The data bus 68 serves as a data path between the aforementioned components. In the present embodiment, it is assumed that the first search target image and the second search target image are stored in the hard disk device of the server PC.

  With the configuration described above, the user can upload an image from each client PC to the server PC and view the uploaded image from the client PC. Each image is accompanied by a keyword automatically assigned by the server PC or client PC as tag information. Each image file may be accompanied by a plurality of keywords.

  Next, the operation of the image classification device 10 in this embodiment will be described.

(System operation as seen from the user)
First, the operation of the system as seen from the user will be described.

  First, the user interface will be described. The user accesses the server PC from the client PC via, for example, a web browser. Specifically, when the user starts up a web browser and inputs address information of the server PC, a dialog screen 70 as shown in FIG. 4 is displayed on the display. The dialog screen 70 illustrated in FIG. 4 includes a keyword input box 71, a search button 72, an image registration button 73, a keyword registration button 74, and an image display area 75.

  When the user inputs a favorite keyword in the keyword input box 71 and presses the search button 72, thumbnails of image files that hold the keyword as a tag among the image files held in the client PC are aligned in the image display area 75. Displayed.

  When the user presses the image registration button 73, a file selection dialog screen is displayed. When an image file held in the client PC is designated on the file selection dialog screen, the designated image file is uploaded to the server PC. A tag is automatically assigned to an image (input image) uploaded to the server PC. Note that how to add a tag will be described later.

  Here, it is determined whether or not an image area including an image similar to an image already registered in the server PC exists in the input image. If there is an image area, the image area is associated with the image registered in the server PC. The keyword is added to the uploaded image as a tag. Further, the program for the dialog screen includes interface means for adding a new keyword as a tag to the image.

  Next, when the user presses the keyword registration button 74, a dialog screen 80 as shown in FIG. 5 is displayed on the display. The dialog screen 80 includes an image selection button 81, a keyword input box 82, a registration button 83, a display area 84, and a warning display area 85.

  By pressing the image selection button 81, a file selection dialog screen is displayed. When an image file held in the client PC is designated here, an image related to the image file is displayed in the display area 84. When the user drags the mouse on the display area 84, a rectangle as shown in the image of FIG. 5 is drawn.

  The user can input tag information to be added to the image as a keyword in the keyword input box 82. When the registration button 83 is pressed, the keyword input in the keyword input box 82 is registered in the first image DB 22 together with the image of the area surrounded by the rectangle in the display area 84. At this time, the server PC determines whether or not the registered image is suitable for identification. If the image is not suitable for identification, the server PC displays a warning such as “the selected area cannot be identified well” in the warning display area 85. The registration of images shall be cancelled. The processing so far is an operation of determining whether or not an image to be input thereafter is similar to a registered image, and creating a template for automatically assigning a keyword if it is similar. However, when a flat image area (an image area where the density change between pixels is small over a wide range of the screen), for example, an area such as the background of the person image shown in FIG. The same keyword is assigned to almost all images. Therefore, in the present embodiment, at the time of image registration, the same keyword is assigned to most images by making a determination on the designation of an area where such a problem is likely to occur and displaying a warning to the user. To solve the problem. How to determine whether it is suitable for identification will be described later.

(System operation seen from server PC)
Next, the operation of the system viewed from the server PC will be described.

  First, the flow at the time of image registration will be described with reference to FIGS. FIG. 6 is a flowchart at the time of image registration.

  The first similarity calculation unit 21 inputs the input image and the data of the first search target image (steps S11 and S12). The first similarity calculating unit 21 determines whether an image area similar to the image area included in the first search target image exists in the input image, and calculates the similarity when it exists. (Step S <b> 13), data indicating the degree of similarity is transferred to the first similarity determination unit 23. If there is no similar area, a similarity of 0 is calculated and transferred to the first similarity determination means.

  Here, it is assumed that the first search target image is an image registered by the user as described above, and an associated tag is assigned to the image. Further, as will be described later, the first similarity calculation unit 21 detects various extreme pixels (Keypoints) between the input image and the search target image when determining whether or not a similar region exists, Using the information around each extreme pixel, the corresponding (similar) extreme value between both images is detected, and the number information of the detected extreme pixel is also transferred to the first similarity determining means 23 at the same time. To do.

  The first similarity determination unit 23 determines whether or not the input image and the first search target image are similar (step S14). In step S14, if the input image and the first search target image are similar to each other, the first similarity determination unit 23 outputs a tag given to the search target image (step S18). If the first search target image is not similar, no tag is output. The detailed operation of the first similarity determination unit 23 will be described later.

  Subsequently, the second similarity calculation means 24 calculates the similarity between the entire area of the input image and the entire areas of the various images held in the second search target image, and stores the data indicating the similarity in the first 2 to the similarity determination means 26. The detailed operation of the second similarity calculation unit will be described later.

  The second similarity determination unit 26 determines whether the input image and the second search target image are based on whether the similarity calculated by the second similarity calculation unit 24 is a predetermined threshold (for example, 0.7) or more. It is determined whether the entire image is similar (step S17). In step S17, when there is a second search target image that is equal to or greater than the similarity threshold, the second similarity determination unit 26 outputs the tag that has been assigned to the second search target image (step S18). ). On the other hand, the second similarity determination unit 26 does not output a tag when there is no second search target image that is equal to or greater than the similarity threshold. Note that the similarity threshold is preferably determined based on, for example, data obtained through experiments in advance.

Through the above processing, a plurality of tags are automatically assigned to the input image. As described above, tags such as landscapes and portraits are ambiguous and are composed of a plurality of components such as a person and a background. If the positional relationship of these elements changes on the image, the feature amount of the entire image changes greatly. Therefore, in order to determine these tags from the feature amount of the entire image, a very large amount of images must be prepared. The operation of registering this becomes a very high load for the user. On the other hand, in the case of a specific object such as the user himself / herself, family face, or Mt. Fuji, it is often composed of a single component. Even in the case of a plurality of components, the positional relationship between the components is often fixed on the image. With respect to such specific object images, it is possible to determine the degree of similarity with high accuracy only by registering a small number of images. As in this embodiment, ambiguous tags that must be compared with a large number of images are registered at the time of system shipment, and specific tags that only need to be compared with a small number of images are given to the user. With the configuration of registration, various tags can be attached while reducing the burden on the user, and customization by the user is possible.
(Operation of the first similarity calculation means 21)
Next, how the first similarity calculation unit 21 determines whether an area similar to an image included in the first search target image (hereinafter referred to as a search target image) exists in the input image. 2 and FIG. FIG. 7 is a flowchart showing the detailed operation of the first similarity calculation means 21.

  The SIFT calculation unit 41 inputs data of an input image to be registered (step S21), and calculates the SIFT of the input image (step S22). Also, the SIFT calculation unit 42 receives the data of the first search target image (step S23), and calculates the SIFT of the first search target image (step S24).

SIFT here refers to Document 1 (David G. Lowe, "Distinctive Image Features from Scale-Invariant Keypoints", International Journal of Computer Vision, 2004.
) Is a technique for detecting a plurality of characteristic pixels in an image and calculating a feature amount from information on a peripheral region for each pixel.

  The characteristic pixel referred to here means a pixel that is maximal or minimal, that is, an extreme value with respect to the periphery. However, it is not simply an extreme value in the image. An extreme pixel in SIFT is a process of applying a plurality of continuous Gaussian filters of size (dispersion) to an image, creating a plurality of blurred images, arranging them in the order of size, A difference image (Difference of Gaussian: DoG) is created. A characteristic pixel is not only an extreme value in an image of the same DoG but also an extreme value for a pixel corresponding to a small DoG target pixel and a large DoG target pixel. Are detected as extreme values.

  In this way, it is possible to know which size of Gaussian the peak or valley constituting the extreme value most closely matches. Thereafter, by calculating the feature amount of the peripheral region using DoG obtained by the Gaussian of that size, even if the input image and the search target image are different in size, similar feature amounts can be obtained in corresponding points. It is done. That is, SIFT can calculate an invariant with respect to the scale of the image.

  In Document 1, as the subsequent processing, whether or not the characteristic pixel obtained is a point on the edge, whether or not the contrast of the peripheral pixel is equal to or higher than the threshold, the characteristic pixel detected by the processing Although detailed position estimation of characteristic pixels by selection and parabolic fitting is performed, the description is omitted because it is different from the essence of the present invention.

  Subsequently, a characteristic amount calculation method around a characteristic pixel in SIFT will be described. In SIFT, characteristic pixel orientation estimation is performed so that a corresponding characteristic pixel can be detected even if the search target image exists in a rotated manner in the input image.

  For this purpose, first, data indicating the gradient strength and gradient direction is calculated. Assuming that the pixel value L (u, v) of the image, the gradient intensity m (u, v), and the gradient direction θ (u, v), it can be calculated as follows.

  Thereafter, a histogram is prepared by discretizing the gradient direction by 10 degrees in 36 directions. In the histogram, a value obtained by multiplying the gradient intensity by Gaussian centered on the target pixel is added. The direction of the largest value in the histogram is the characteristic pixel orientation.

  Next, the image is rotated so that the characteristic pixel orientation is directed upward. Thereafter, the peripheral area of the characteristic pixel is divided into 16 blocks, 4 blocks on each side. By creating a gradient histogram in 8 directions at 45 degrees for each block, 4 × 4 × 8 = 128-dimensional feature values can be obtained. After normalizing the pixels around the characteristic pixels in this way so that the estimated orientation of the characteristic pixels is directed upward, the calculated feature values are calculated with respect to the rotation of the image. And invariant features.

  As described above, the SIFT calculation units 41 and 42 use the SIFT feature value to search extremely stably with respect to the difference in size and rotation when searching for the search target image in the input image. can do.

  Subsequently, the SIFT comparison unit 43 determines whether or not characteristic pixels (corresponding points) having characteristic amounts close to characteristic pixels included in the search target image exist in the input image one by one. The hit is examined (step S25). The near feature amount referred to here indicates that the Euclidean distance between characteristic pixels is equal to or less than a predetermined threshold (for example, 300).

  Since SIFT is a feature quantity obtained by referring to only a local region, the obtained corresponding points do not necessarily belong to the same image. Therefore, the corresponding point selection unit 44 selects corresponding points from the positional relationship of the corresponding points (step S26). In order to make the selection, Hough transform is used in Document 1. That is, it is possible to estimate what posture the plane of the search target image takes in the input image from the size and orientation of one corresponding point. Corresponding points are selected by quantizing the estimated values and looking at their distribution. When there are a large number of corresponding points indicating the same posture, they are highly reliable corresponding points, and when there are a small number of corresponding points indicating the same posture, they can be said to be corresponding points with low reliability. In the present embodiment, the number of points indicating the same posture is referred to as the corresponding number of points.

  The corresponding point selection unit 44 determines that the search target image exists in the input image when the number of corresponding points is 3 or more, and when the number of corresponding points is 2 or less, the search target image is included in the input image. The similarity is calculated as 0 because it does not exist (step S27).

  When the search target image exists in the input image, the corresponding region extraction unit 45 extracts a region corresponding to the search target image from the input image (step S28). When the search target image exists in the input image, as described above, it is possible to estimate what posture the plane of the search target image takes in the input image. Extract from

  Subsequently, the image comparison means 50 compares the extracted image with the search target image and calculates the similarity between the two (step S29). The detailed operation of the image comparison unit 50 will be described later.

  As described above, the first similarity calculation unit 21 determines whether or not the search target image exists in the input image, and can calculate the similarity when it is determined that the search target image exists.

(Operation of the image comparison means 50)
The image comparison unit 50 calculates a feature amount indicating the feature of the image from the two images, and calculates the similarity by comparing them. In the present embodiment, it is assumed that three types of feature amounts of color, edge, and pattern are calculated.

  As a premise, the color information of each pixel in the image is assumed to indicate the gradations of the three primary colors R (red), G (green), and B (blue) with 256 gradations of 0 to 255, respectively. When the gradations of the three primary colors are all 0, the color of the pixel is black. When the gradations of the three primary colors are all 255, the color of the pixel is white. As described above, it is assumed that three-dimensional color information of the sRGB color system is assigned to each pixel.

  First, a method for calculating the edge feature amount will be described. First, convolution integration is performed on each pixel of the pixel matrix in the image using a 3 × 3 filtering matrix as shown in FIG. The central pixel value “4” in the illustrated filtering matrix is assigned to the target pixel in the image, and the pixels existing around the central pixel of the filtering matrix are assigned to the pixels existing around the target pixel. Assign a value. Such convolution integration is performed on the entire image to obtain an edge image. Thereafter, the image is binarized using a predetermined threshold (for example, 128). Next, the entire image is equally divided into, for example, 10 × 10 blocks of the same size, and the pixels exceeding the binarization threshold in each block are counted. A 100-dimensional vector is obtained by the above processing. Also, normalization is performed by dividing by the number of all pixels included in the block, and the value of each element of the vector is normalized to 0-1.

  Next, a method for calculating the color feature amount will be described. First, all pixels included in the image are divided by 255 and normalized. The image is converted from the sRGB color system to the Lab color system based on the following equations 3 to 9.

  When a D65 light source is assumed as the light source, Xn = 0.95, Yn = 1.00, and Zn = 1.09. After conversion to the Lab color system in this way, the image is then equally divided into 10 × 10 blocks in the same manner as the calculation of the edge feature amount, and an average Lab is obtained for each block. Further, the obtained Lab is converted into L′ a′b ′ normalized to a value of 0 to 1 by the following formula. As a result, a 100 × 3 = 300-dimensional vector is obtained.

Next, a method for calculating the pattern feature amount will be described. A known density co-occurrence matrix is used for calculating the pattern feature amount. Co-occurrence matrix, the frequency of 1 pixel in the small area from the pixel brightness of the k of the grayscale image shown in FIG. 9 δ (r, θ) in the relative position indicated by that appears p _[delta] (r , Θ). The sRGB color system image is converted to a gray image, and then each pixel is divided by 16 and the remainder is discarded to quantize it to 16 gradations. Then, a 16 × 16-dimensional density co-occurrence matrix is obtained based on Equation 13, where 16 is the number of gradations. The obtained matrix value is divided by the number of pixels included in the image and normalized to a value of 0 to 1.

  In this embodiment, since three types of density co-occurrence matrices of δ (1,0), δ (1,45), and δ (1,90) are obtained, the final result is 256 × 3 = 768 dimensions. A feature vector is obtained.

  Since the density co-occurrence matrix is a feature amount indicating an outline of the frequency information of the image, it can be used for calculating the texture feature amount. In addition, it is possible to use Fourier transformation for acquisition of frequency information. Further, if the image data file stored in the MFP (multifunction machine) is compressed by the JPEG method, the frequency information can be easily obtained by using the Discrete Cosine Transform. As described above, a 100 + 300 + 768 = 1168-dimensional feature vector is finally obtained.

  The above feature quantities are calculated for two images, respectively, and the Euclidean distance between them is calculated to calculate the image similarity. Since the feature amounts are all normalized to 0 to 1, if the Euclidean distance is subtracted from 1, the similarity of 0 to 1 can be obtained.

  As described above, when the image comparison unit 50 calculates the similarity based not only on the corresponding points of SIFT but also on the feature amount of the image, the first similarity calculation unit 21 calculates the similarity with higher accuracy. can do.

(Operation of the first similarity determination means 23)
Next, the operation of the first similarity determination unit 23 will be described. The first similarity determination means 23 attaches a tag given to the first search target image to the input image depending on whether or not the similarity calculated by the first similarity calculation means 21 exceeds a predetermined threshold. Determine whether or not. Here, the normal threshold value is 0.8, but when the number of corresponding points is more than 10, for example, the threshold value is preferably 0.7. In general, when a plurality of corresponding points have the same posture, there is a low possibility of being erroneously determined. Therefore, in step S27 of FIG. 7, when there is a very small number of corresponding points of “3”, it is determined that the search target image exists in the input image. Therefore, when there are a sufficient number of corresponding points, there is a high possibility that the degree of similarity is high. In this case, the threshold value is lowered. As a result, the possibility that an image that is originally similar is erroneously determined to be dissimilar is reduced, and the possibility that correct tagging can be performed is increased.

(Operation of Second Similarity Calculation Unit 24)
The operation of the second similarity calculation means 24 is the same as the processing in step S29 in FIG. 7, and the similarity between the input image and the image included in the second search target image is compared. However, the comparison here does not compare in the local region of the image, but compares the entire images.

(Operation when registering keywords)
Next, the operation of the server PC at the time of keyword registration will be described based on FIG. 1 and FIG. FIG. 10 is a flowchart showing the operation of the server PC at the time of keyword registration.

  The area designating unit 31 inputs the input image data (step S31), and outputs the input image data in which the area is designated to the extreme pixel detection unit 32. The extreme pixel detection means 32 performs the same processing as this SIFT calculation (step S22 in FIG. 7) on this input image (step S32). However, the extreme pixel detection means 32 detects a characteristic pixel in the input image, but does not calculate a feature amount. The extreme pixel detection means 32 calculates the number of characteristic pixels by this processing, and if the number of characteristic pixels is 20 or less, for example, the input image in which the area is designated is not suitable for identification and warns the user. (Step S33 “No”). On the other hand, if the number of characteristic pixels is greater than 20, a keyword is registered together with the image (step S33 “Yes”). In the method of determining whether or not there is a similar region using SIFT, there is a possibility that it can be determined that a similar region exists if there is not a sufficient number of characteristic pixels to select corresponding points. Becomes lower. Here, an image without a sufficient number of characteristic pixels is often a uniform image with few patterns. Since a uniform image can exist in various images as described above, it is not suitable for identification.

  As described above, according to the image classification device 10 of the present embodiment, the first similarity calculation unit 21 calculates the first search target image and the input image that are tagged with a specific keyword for the user. The similarity in the local region is calculated, and the second similarity calculation means 24 is configured to calculate the similarity in the entire image between the second search target image tagged with the ambiguous keyword tag and the input image. Therefore, it is possible to perform customization while minimizing the burden on the user and perform classification using the entire image.

  In the above-described embodiment, the configuration in which the first search target image is stored in the hard disk device of the server PC has been described as an example. However, the present invention is not limited to this, and the first search target image is stored. The image may be stored in the hard disk device of the client PC. Further, for example, a folder for each user may be provided in the hard disk device of the server PC, and each user and the first search target image may be stored in association with each other.

(Second Embodiment)
First, the configuration of the image classification apparatus according to the second embodiment of the present invention will be described.

  As shown in FIG. 11, the image classification device according to the present embodiment includes an image registration device 90 for registering images. In addition, the same code | symbol is attached | subjected to the structure similar to 1st Embodiment, and description of a structure is abbreviate | omitted.

  The image registration apparatus 90 includes a first similarity calculation unit 21 that calculates a first similarity, a first image DB 22 that stores data of a first search target image, and a first similarity determination. 1 similarity determination unit 91, a second similarity calculation unit 92 that calculates a second similarity, a feature amount DB 93 that stores feature amount data, and a second similarity determination that performs a second similarity determination Means 94.

  As shown in FIG. 11, an image registration apparatus 90 in the present embodiment has substantially the same configuration as that of the first embodiment, but differs from the first embodiment in two points.

  First, in the first embodiment, the input image and the first and second search target images are individually compared. However, in the present embodiment, each of the second search target images is Rather than comparing, the feature amount is calculated from various images with the same tag (the feature amount calculation method is the same as that of the image comparison unit 50 described above), and the SVM described in Non-Patent Document 1 is used by using them. It shall be identified by using it. The SVM is a binary discriminator that can classify an unknown input relatively accurately, that is, has high generalization performance. Therefore, the number of images to be prepared in advance can be reduced by using SVM. Furthermore, since the SVM has a noise removal function and ignores similar data, it is more accurate and can be identified at a higher speed than comparing each image. The SVM method will be described later.

  Second, in the present embodiment, the image is not attached with a tag but is classified and stored in a folder associated with a predetermined tag. At this time, a plurality of tags cannot be attached to one image and must be narrowed down to one. In general, it is expected that specific information is given priority over ambiguous information. Therefore, in the present embodiment, when the first search target image is included in the input image, the tag added to the search target image is added to the input image, and the second similarity calculation unit 92 is added. The second similarity determination unit 94 is not processed. Thereby, the amount of calculation processing is reduced, the waiting time of the user is reduced during image registration, and the convenience for the user is improved.

(Method for generating classification rules by SVM)
Next, a method for generating classification rules by SVM will be described. The SVM is a discriminator that outputs y = 1 if the inner product of the input vector and the weight vector ω exceeds a specific threshold, and outputs y = −1 if it does not exceed the output y = Assume that the input image is a document image when 1 and the input image is a photographic image when output y = −1. That is, SVM learning is an operation for determining the weight vector ω and the threshold value h. In addition, although the detailed description about learning of SVM is described in the above-mentioned literature 1, the outline | summary is demonstrated below.

  FIG. 13 shows an outline of the operation of the SVM. First, as a precondition, it is assumed that there are two types of vector groups represented by ○ × as shown on the left side of the figure. SVM can be said to be an algorithm for determining a hyperplane (see the right in FIG. 13) for optimally separating these two classes. In SVM, optimally dividing two vector groups is equivalent to maximizing the ability to respond when an unknown vector is input, that is, the generalization ability. In order to realize this, a vector (Support Vector) existing at the boundary position between two vector groups is found, and the hyperplane is set so that the distance between this vector and the hyperplane is maximized.

  Here, since there is erroneous teacher data in actual operation, it is necessary to determine a parameter for setting an error tolerance (soft margin). The above is a description of the linear SVM, but actual teacher data is not necessarily a vector group that can be linearly discriminated. However, by projecting the feature vector onto a higher-order space (kernel trick) and obtaining a hyperplane in that space, it is possible to cope with nonlinear problems.

In order to realize the above, as a result, a Lagrange multiplier vector α _i that maximizes Expression 16 under the condition of Expression 15 is obtained using the teacher data (x _i , y _i ). Then, a super plane parameter is obtained using a teacher data group S corresponding to a non-zero element of Lagrange multiplier vector elements (this is a support vector) and any one of the teacher data (x ₀ , y ₀ ). ω and h are obtained (Equations 17 and 18).

  In Equation 16, K (x, y) represents a kernel function for realizing a kernel trick. Various kernel functions have been devised. In this embodiment, RBF (Radial Basis Function) is used. RBF is a function expressed by Equation 19, and C is an arbitrary numerical value. As described above, in order to perform the learning using the SVM, it is necessary to set the parameter γ for setting the allowable amount of the soft margin and C for determining the RBF as the kernel function, but Document 2 (Chih− As described in Chung Chang and Chih-Jen Lin, LIBSVM: a library for support vector machines, 2001), the range of C and γ and the step width of those values are determined in advance, and the round-robin identification rate It is preferable to calculate C and γ that obtain the best discrimination rate.

  Next, the operation of the image classification apparatus according to the present embodiment will be described with reference to FIGS. FIG. 12 is a flowchart showing the operation of the image classification apparatus according to this embodiment.

  The first similarity calculation unit 21 inputs the input image and the data of the first search target image (steps S11 and S12). The first similarity calculating unit 21 determines whether an image area similar to the image area included in the first search target image exists in the input image, and calculates the similarity when it exists. (Step S <b> 13), data indicating the degree of similarity is transferred to the first similarity determination unit 23. If there is no similar area, a similarity of 0 is calculated and transferred to the first similarity determination means.

  The first similarity determination means 23 determines whether or not the input image and the first search target image are similar based on a predetermined threshold (step S41). The data of the input image is moved to the folder associated with the tag assigned to one search target image (step S45).

  On the other hand, if the input image and the first search target image are not similar in step S41, the second similarity calculation unit 92 reads predetermined feature amount data from the feature amount DB 93 (step S42), and SVM. Is used to calculate the second similarity (step S43). The calculated second similarity data is sent to the second similarity determination means 94.

  The second similarity determination unit 94 determines whether or not the input image and the second search target image are similar in the entire image based on the similarity calculated by the second similarity calculation unit 92 ( Step S44).

  In step S44, when the input image and the second search target image are similar to each other in the entire image, the second similarity determination unit 94 stores the tag in the folder associated with the tag assigned to the second search target image. The data of the input image is moved (step S45), and if the input image and the second search target image are not similar in the entire image, the process ends.

  As described above, according to the image classification apparatus of the present embodiment, the second similarity calculation unit 92 is configured to calculate the similarity only when the input image and the first search target image are not similar. Therefore, when performing one image classification for one image, unnecessary processing can be reduced and image classification processing can be performed at high speed, reducing the waiting time of the user at the time of image registration, and improving user convenience. Can be improved.

  As described above, the image classification device and the image classification method according to the present invention have the effect that customization is possible and classification using the entire image can be performed while minimizing the burden on the user. It is useful as software for printers, copiers, digital cameras, PCs and servers.

1 is a block diagram of an image classification device according to a first embodiment of the present invention. The block diagram of the 1st similarity calculation means in 1st Embodiment of this invention. 1 is a block diagram of a computer that implements an image classification apparatus according to a first embodiment of the present invention. The figure which shows the dialog screen at the time of starting of a web browser in the image classification device in 1st Embodiment of this invention. The figure which shows the dialog screen at the time of keyword registration in the image classification device in 1st Embodiment of this invention. The flowchart which shows the operation | movement at the time of the image registration of the image classification device in 1st Embodiment of this invention. The flowchart which shows the detailed operation | movement of the 1st similarity calculation means in 1st Embodiment of this invention. The figure which shows an example of the filtering matrix of the image classification device in 1st Embodiment of this invention. Explanatory drawing of the calculation method of the pattern feature-value in 1st Embodiment of this invention. The flowchart which shows operation | movement of server PC at the time of keyword registration in 1st Embodiment of this invention. The block diagram of the image registration apparatus in 2nd Embodiment of this invention. The flowchart which shows operation | movement of the image classification device in 2nd Embodiment of this invention. The figure which shows the outline | summary of operation | movement of SVM in 2nd Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Image classification apparatus 20 Image registration apparatus 21 1st similarity calculation means 22 1st image DB (1st image data storage means)
23, 91 First similarity determination means (similarity determination means, keyword assignment means)
24, 92 Second similarity calculation means 25 Second image DB (second image data storage means)
26, 94 Second similarity determination means (keyword assignment means)
DESCRIPTION OF SYMBOLS 30 Keyword registration apparatus 31 Area | region designation | designated means 32 Extreme value pixel detection means 33 Registration means 34 Warning means 40 Local area extraction means 41, 42 SIFT calculation part (Extreme value pixel detection part, feature-value calculation means)
43 SIFT comparison part (feature quantity comparison part)
44 Corresponding point selection part (extreme pixel selection part)
45 Corresponding region extraction unit (corresponding region extraction unit)
DESCRIPTION OF SYMBOLS 50 Image comparison means 71 Keyword input box 72 Search button 73 Image registration button 74 Keyword registration button 75 Image display area 80 Dialog screen 81 Image selection button 82 Keyword input box 83 Registration button 84 Display area 85 Warning display area 93 Feature-value DB

Claims (9)

An image classification device for classifying the input image by comparing the similarity between the image with the keyword and the input image,
First and second image data storage means for storing first and second image data registered with the keyword assigned thereto, and a local area of a predetermined size as the input image and the first image data. Local area extracting means for extracting from the image, first similarity calculating means for calculating the similarity between the input image and the first image in the extracted local area, the entire image of the input image, and the A second similarity calculating means for calculating a similarity between the second image and the whole image; and a keyword is assigned to the input image based on the similarity calculated by the first and second similarity calculating means. An image classification apparatus comprising: a keyword assigning means for performing The first image data storage unit stores data of an image registered by giving a specific keyword by a user as data of the first image. Image classification device. 3. The second image data storage means stores image data to which a predetermined ambiguous keyword is added as data of the second image. The image classification device described. The local region extraction means includes an extreme value pixel detection unit that detects an extreme value pixel whose feature value is an extreme value, and an image feature value near the extreme value pixel in each of the input image and the first image. A feature amount calculating unit that calculates the feature amount, a feature amount comparison unit that compares the calculated feature amount, and an extreme pixel that selects an extreme pixel based on a positional relationship of the extreme pixel in each of the input image and the first image. A value pixel selection unit, and a corresponding region extraction unit that extracts an image region corresponding to the first image from the input image based on the number of extreme pixels selected by the extreme pixel selection unit. The image classification device according to any one of claims 1 to 3, wherein the image classification device is characterized by the following. The first similarity calculation means includes image comparison means for calculating the similarity by comparing the image of the image area extracted by the corresponding area extraction unit with the first image. The image classification device according to claim 4. The extreme pixel detection unit detects an extreme pixel of an image when a user registers as the first image,
6. The image according to claim 4, further comprising warning means for giving a warning to the user when the number of extreme pixels detected by the extreme pixel detection unit is equal to or less than a predetermined number. Classification device. An area designating unit for designating a specific area of the input image;
The image classification according to any one of claims 1 to 6, wherein the first image data storage means stores image data of an area designated by the area designation means. apparatus. Similarity determination means for determining whether or not the input image and the first image are similar based on the similarity calculated by the first similarity calculation means;
The second similarity calculation means calculates the similarity related to the entire image only when the input image and the first image are not similar to each other. The image classification device according to claim 1. An image classification method for classifying the input image by comparing the similarity between the image with the keyword and the input image,
Storing the data of the first and second images registered with the keyword, extracting a local area of a predetermined size from the input image and the first image, and extracting Calculating the similarity between the input image and the first image in the local region, calculating the similarity between the entire image of the input image and the entire image of the second image, And a step of assigning a keyword to the input image based on the similarity calculated by the first and second images.