JP2007041762A - Image recognizing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image recognizing device for performing proper image recognition. <P>SOLUTION: This image recognizing device is provided with a storage means for associating an image featured value, a keyword, location information and location information relevancy showing how much an object to which the keyword is added is unique to a specific place with each of a plurality of object image data, and for storing the image data as a database, an image data input means for inputting the image data as acquired image data, and for extracting the location information, an image featured value extracting means for extracting the image featured value of the acquired image data, an exclusion deciding means for deciding whether or not the object image data should be excluded from the processing object by using the location information relevancy of the database and the interval of a photographic location calculated from the location information of the acquired image data and the location information of the object image data and a keyword extracting means for comparing the image featured values of the acquired image data with the image featured values of the object image data, and for extracting the keyword satisfying conditions from the database when it is judged that the object image data are the processing object. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a technique for recognizing the contents of image data.

  2. Description of the Related Art Conventionally, systems for searching Web pages by inputting text format keywords have been widely put into practical use. In such a system, a desired Web page can be searched by inputting an appropriate keyword. As a search method using such keywords, there is a technique for searching various information from an arbitrary image. For example, in Patent Document 1 below, a database that stores information related to captured images and the names of the locations where the images are captured is prepared. When arbitrary image data is input, similar image data is retrieved from the database and searched. A technique for extracting a shooting place name associated with similar image data is disclosed.

JP 2002-373168 A

  However, in the technique using such a database, information on the shooting location name of the image data can be extracted, but information other than the shooting location name, such as the name of the subject, cannot be acquired. For this reason, it is difficult to extract an appropriate keyword representing the content of the image data and perform image recognition.

  An object of the present invention is to provide an image recognition apparatus that performs appropriate image recognition in view of the problem that appropriate image recognition is difficult.

  In view of the above problems, the first image recognition apparatus of the present invention employs the following method. That is, an image recognition apparatus that executes processing for extracting a keyword representing the image content of one image data having photographing position information from a predetermined search target and recognizes the image content, and is the search target. For each of the plurality of target image data, an image feature amount, a keyword representing the image content, position information at which an image having the image feature amount is photographed, and an object to which the keyword is assigned is specific to a specific location A storage means for associating a position information relevance degree indicating a degree of image data and storing it as a database; and image data for inputting the one image data as acquired image data and extracting position information where the acquired image data is photographed An input means; an image feature quantity extraction means for extracting an image feature quantity included in the acquired image data; and a position of the target image data stored in the database. Whether or not to exclude the target image data from the keyword extraction processing target using the degree of report relevance, the position information of the acquired image data, and the shooting position distance obtained from the position information of the target image data An exclusion determination means for determining whether or not the target image data satisfies the predetermined condition by comparing the image feature amount of the acquired image data with the image feature amount of the target image data when the target image data is determined to be the processing target. And a keyword extracting means for extracting the keywords from the database.

  Further, an image recognition method corresponding to the first image recognition apparatus of the present invention executes a process of extracting a keyword representing the image content of one image data having shooting position information from a predetermined search target, An image recognition method for recognizing image contents, wherein for each of a plurality of target image data to be searched, an image feature amount, a keyword representing the image content, and a position where an image having the image feature amount is photographed The information and the positional information relevance indicating the degree to which the object to which the keyword is assigned is unique to a specific place are stored as a database, and the one image data is input as acquired image data, Extracting position information where the acquired image data is photographed, extracting image feature amounts included in the acquired image data, and extracting the position information function of the target image data stored in the database. And whether or not to exclude the target image data from the processing target of the keyword extraction process using the position information of the acquired image data and the distance of the photographing position obtained from the position information of the target image data. If it is determined that the image is to be processed, the image feature quantity of the acquired image data is compared with the image feature quantity of the target image data, and the keyword of the target image data satisfying a predetermined condition is determined as the database. The main point is to extract from.

  According to the first image recognition apparatus and image recognition method of the present invention, a database is prepared in which image feature quantities, keywords representing image contents, position information, and position information relevance are stored in association with each target image data. Then, the database is searched, and a keyword indicating the image content of the acquired image data is extracted from the image feature amount of one acquired image data. Therefore, an appropriate keyword representing the image content of the acquired image data can be extracted from the database.

  Further, when extracting a keyword, it is determined whether or not to exclude the target image data from the processing target of the keyword extraction process by using the positional information relevance level and the distance between shooting positions. That is, the search target of the database is narrowed down. Therefore, it is possible to reduce the number of search targets and perform a quick process.

  The positional information relevance of the image recognition apparatus having the above configuration is as follows: A) Type A in which the keyword in the target image data is assigned to a specific place, and B) in the target image data An object to which the keyword is added is of a type B where the degree to which the keyword is present is high and the place where the keyword is present is limited, and C) the object to which the keyword in the target image data is assigned is specified It is classified and set according to three types, ie, type C that is not unique to the place, and the exclusion determination means has a position information relevance level of the target image data of the type A, and the shooting position When the gap is equal to or larger than a predetermined amount, it may be determined that the target image data is excluded from the processing target.

  According to such an image recognition apparatus, when searching the database, target image data whose positional information relevance is type A and whose shooting position gap is a predetermined amount or more is excluded from the processing target. Since such target image data is specific to a specific location, there is a possibility that the image content of the acquired image data is the same as or similar to the image content of this target image data unless it is determined that the shooting location is close. Remarkably low. By excluding such target image data from the processing target, it is possible to perform quick processing and improve the accuracy (recognition accuracy) for extracting an appropriate keyword.

  In the image recognition apparatus having the above-described configuration, the storage unit further sets an area range indicating a geographical range in which an object to which a keyword is assigned in the target image data can be captured for each target image data. Is stored as the database, and the exclusion determination unit is configured to detect the shooting position when the position information of the acquired image data does not fall within an area range of the target image data stored in the database. It is good also as what judges that the gap of this is more than predetermined.

  According to such an image recognition apparatus, an area range is provided in the database, and it is determined whether or not the shooting positions are separated based on the area range. Therefore, the determination can be easily made.

  The keyword extraction unit of the image recognition apparatus having the above configuration uses the image feature amount of the acquired image data and the image feature amount of the target image data to determine whether the acquired image data and the target image data are similar or dissimilar. A reliability coefficient calculation unit that calculates a reliability coefficient indicating the degree, and an extraction unit that extracts a keyword of the target image data based on the calculated reliability coefficient may be used.

  According to such an image recognition apparatus, it is possible to easily extract a keyword by comparing the reliability coefficient for each target image data.

  In the image recognition apparatus having the above-described configuration, the keyword extracting unit further corrects the reliability coefficient by adding the position information relevance to the reliability coefficient calculated using the image feature amount. A correction unit may be provided, and the extraction unit may extract the keyword based on the corrected reliability coefficient.

  According to such an image recognition apparatus, the corrected reliability coefficient is a coefficient that takes into consideration the image feature quantity and the position information relevance, and a keyword is extracted based on the reliability coefficient. Since the degree of similarity between the acquired image data and the target image data is determined based on a plurality of pieces of information, it is possible to easily extract an appropriate keyword.

  The extraction unit of the image recognition apparatus having the above configuration extracts a keyword of the target image data having the highest reliability coefficient from among the corrected reliability coefficients obtained for each target image data to be searched. It is good as a thing.

  According to such an image recognition device, by comparing the corrected reliability coefficients, it is possible to extract one keyword that is determined to be most appropriate from the search targets of the database.

  The second image recognition apparatus according to the present invention includes, for each of a plurality of target image data to be searched, an image feature amount, a keyword representing the image content, and position information at which an image having the image feature amount is captured. An image of one piece of image data having position information of a shooting position using a database that stores and associates a position information relevance level indicating the degree to which the object to which the keyword is assigned is unique to a specific place An image recognition apparatus that executes processing for extracting a keyword representing the content from the database and recognizes the image content, wherein the one image data is input as acquired image data, and the acquired image data is captured Image data input means for extracting information, image feature quantity extraction means for extracting image feature quantities included in the acquired image data, and the target image stored in the database The target image data is excluded from the processing target of the keyword extraction process using the degree of relevance of the position information of the data, the position information of the acquired image data, and the distance of the photographing position obtained from the position information of the target image data. An exclusion determination means for determining whether or not to perform processing, and when it is determined that the image is to be processed, the image feature amount of the acquired image data is compared with the image feature amount of the target image data, and the predetermined condition is satisfied. The gist of the invention is that it includes keyword extraction means for extracting keywords of the target image data from the database.

  According to the second image recognition apparatus of the present invention, when a keyword indicating the image content of the acquired image data is extracted from the image feature amount of one acquired image data using a database prepared in advance, Whether or not the target image data is to be excluded from the processing target of the keyword extraction process is determined using the gap between the shooting positions. Therefore, it is possible to reduce the number of search targets and perform a quick process.

  The present invention can also be implemented as a computer program and a medium recording the computer program. As the recording medium, various computer-readable media such as a flexible disk, a CD-ROM, a DVD-ROM / RAM, a magneto-optical disk, a memory card, and a hard disk can be used.

Hereinafter, embodiments of the present invention will be described in the following order based on examples.
A. Image recognition system:
B. Database construction:
C. Structure of image recognition device:
D. Image recognition processing:
E. Variation:

A. Image recognition system:
FIG. 1 is an explanatory diagram showing an image recognition system including an image recognition apparatus as one embodiment of the present invention. The image recognition system 100 mainly includes a database 20 including information on a plurality of captured images, a database creation device 10 that constructs the database 20, an image recognition device 15 that uses the constructed database 20, and the like.

  The database creation device 10 includes a keyword setting unit 11, an area range setting unit 12, a position information relevance setting unit 13, an image feature amount extraction device 14, and the like, and is connected to the database 20 via a communication line. The database creation apparatus 10 receives a plurality of image data captured by an imaging device such as a digital still camera, and creates various information corresponding to each image data such as a keyword representing a subject of the image data.

  The target image data in this embodiment includes geographical position information of the shooting location where the image data was shot. This position information is longitude and latitude information (referred to as GPS information) using communication in the GPS (Global Positioning Systems) system, which is an aspect of wireless communication. The database creation device 10 extracts GPS information of image data and uses it as a part of the image data information. Note that the position information of the image data to be handled is not limited to that based on the GPS information, but may be other position information such as information on a mobile phone base station.

  The keyword setting unit 11 expresses the place name of the shooting location where the image content (subject) of the image data is taken, the name of the subject, and the like as text, and sets this as a keyword in association with the image data. The area range setting unit 12 sets an area where the subject in the image data can be photographed with a numerical value of longitude and latitude, and sets this as an area range R in association with the image data. The position information relevance setting unit 13 evaluates and digitizes the relevance between the subject in the image data to which the keyword is attached and the GPS information, and sets this as the position information relevance PR in association with the image data. The image feature quantity extraction device 14 inputs each image data, and extracts the image feature quantity of the image data by predetermined image processing.

  Various information relating to image data such as GPS information, keywords, region range R, position information relevance PR, and image feature amount set in association with each image data is output to the database 20. Of the information related to image data, the keyword, region range R, and position information relevance level PR are set by the user confirming the contents of each image data.

  The database 20 includes a storage area having a predetermined capacity, and stores information related to the image data input from the database creation device 10. In the database 20 of this embodiment, each image data is associated with information related to the image data, but the image data itself is not stored. In other words, the database 20 is a database that stores only feature amounts that are various information related to image data. By doing so, the amount of data is reduced and the storage capacity is reduced. The database 20 having such a configuration is connected to the image recognition apparatus 15 via a communication line.

  The image recognition device 15 includes an image feature amount extraction device 16, a discriminator 17, a reliability coefficient corrector 18, etc., and inputs arbitrary image data, and extracts keywords suitable for the arbitrary image data from the database 20. To do. That is, the image recognizing device 15 uses the database 20 as a dictionary to extract a keyword corresponding to the content of arbitrary image data, and thus becomes a device that recognizes the content of arbitrary image data.

  The image feature amount extraction device 16 in the image recognition device 15 extracts the image feature amount of the input arbitrary image data, like the image feature amount extraction device 14 of the database creation device 10. The image feature quantity extraction device 16 is connected to the discriminator 17, and the extracted image feature quantity is output to the discriminator 17.

  The discriminator 17 inputs the image feature amount from the image feature amount extraction device 16 and extracts GPS information included in the input arbitrary image data. Based on these, the image feature amount and keyword satisfying a predetermined condition are extracted from the database 20. get. The discriminator 17 compares the image feature quantity of the input arbitrary image data with the image feature quantity acquired from the database 20, and calculates a reliability coefficient TC that represents the similarity between the two image feature quantities. The discriminator 17 is also connected to the reliability coefficient corrector 18 and outputs the calculated reliability coefficient TC and the keyword acquired from the database 20 to the reliability coefficient corrector 18.

  The reliability coefficient corrector 18 also receives the keyword and reliability coefficient TC from the discriminator 17 and the positional information relevance PR corresponding to the image data obtained by the discriminator 17 and the image data. The reliability coefficient corrector 18 corrects the reliability coefficient TC input from the discriminator 17 based on the positional information relevance PR, and selects a keyword that satisfies a predetermined condition based on the corrected reliability coefficient TC1 as an external device ( For example, the data is output to a display device such as a display.

  In the image recognition system 100 configured as described above, when arbitrary image data is input, the image recognition device 15 uses (searches) the database 20 to automatically extract predetermined keywords and the like. As described above, the extracted keyword is certified to match the contents of the image data, such as the name of the subject in the image data and the name of the shooting location.

  The extracted keywords are stored in association with the input image data, for example, and can be used for image data management, such as image data search at a later date and classification for each keyword.

  Hereinafter, prior to the description of specific modes of the image recognition device 15, the construction of the database 20 will be described first. In the following description, each process in the database creation device 10 and the image recognition device 15 is realized by a software program, and the image recognition system 100 is configured by a computer in which the processing program is installed.

B. Database construction:
FIG. 2 is a flowchart showing the flow of the database 20 construction process. This process is a process executed by a computer (not shown) as the database creation device 10 in which the processing program is installed. This computer includes a CPU, ROM, RAM, hard disk, and the like, and is a general computer connected to a keyboard, a display, and the like, and is connected to a digital still camera having a plurality of photographed images. The database 20 construction process is executed by a user instruction via the keyboard.

  When the process is started, the CPU inputs predetermined image data in the digital still camera, and executes a keyword setting process for the image data (step S200). Specifically, the input image data is displayed on a display, a keyword representing the image contents input by the user via the keyboard is received, and this is associated with the image data and stored in the hard disk.

  Following the keyword setting, the CPU performs a process of extracting the image feature amount of the input image data (step S210). In the present embodiment, the image feature amount is extracted by using a color layout method employed in MPEG-7. Specifically, the luminance color difference signal of the image data is converted into YCbCr information. The CPU stores the extracted image feature amount in the hard disk in association with the image data. Note that the image feature amount extraction processing is executed in the image recognition device 15 and will be described in detail later in the description of the image recognition device 15.

  Subsequently, the CPU extracts GPS information attached to the input image data (step S220). The CPU stores the extracted GPS information in the hard disk in association with the image data as well as the keyword and the image feature amount. In addition, when there is no GPS information for the image data, the user inputs via the keyboard.

  After extracting the GPS information, the CPU performs a process of setting the area range R (step S230). As described above, the area range R represents an area in which the subject in the image data can be photographed using longitude and latitude. Specifically, as shown in FIG. 3, an approximate area where the subject in the image data can be photographed is set around the longitude and latitude according to the GPS information. For example, when the longitude and latitude according to GPS information are (X, Y) and the subject is “Tokyo Tower”, the range in which “Tokyo Tower” can be photographed is (X ± R, Y ± R). Set the area to be represented. R in this case is set as the region range R. The CPU receives the area range R input by the user via the keyboard for each image data, and stores it in the hard disk in association with the image data.

  Subsequently, the CPU performs a process of setting the position information relevance PR of the input image data (step S240). As described above, the position information relevance PR is obtained by evaluating the relevance between the subject in the image data to which the keyword is attached and the GPS information. In this embodiment, a predetermined keyword is attached, and the numerical value indicates the level at which the subject of the image data displayed on the display is present at a specific location. The CPU stores the position information relevance PR input by the user in the hard disk in association with the image data.

  When setting the position information relevance PR, the types of image data are roughly classified into three types (types A to C) from subjects in the image data to which keywords are attached. Type A is a case where the subject having the keyword “shows an object specific to the place”, type C is a case “not an object specific to the place”, and type B is type A To the middle of type C.

  For the image data classified as type A, the positional information relevance PR in the range of 0.9 ≦ PR ≦ 1.0 is set. For example, if the subject in the image data is “Tokyo Tower” and “Tokyo Tower” is set as the keyword, “Tokyo Tower” is “the object unique to the place”, so a high value (for example, The position information relevance PR of PR = 0.95) is set.

  For image data classified as type C, a position information relevance PR of PR = 0 is set. For example, when the subject in the image data is “cat” and “cat” is set as the keyword, “cat” is “not a target unique to the place”, and therefore the position information relevance PR with a low value. (PR = 0) is set.

  For the image data classified as type B, the position information relevance PR in the range of 0 <PR <0.9 is set. For example, if the subject in the image data is “deer” of “Nara Park” and “deer” is set as the keyword, “deer” is not an object specific to the place, but “Nara Park”, etc. Since the position is limited to a specific place to some extent, the position information relevance PR having a predetermined value (for example, PR = 0.7) is set. These targets include “airplanes” at “airfields”, “ships” at “ports”, “pandas” at “zoos”, “alpine plants” at “mountain parks”, although they are not specific to the place. This applies to a certain range of effective areas.

  In this way, the keyword, the image feature amount, the GPS information, the area range R, and the position information relevance PR are set, and the series of database 20 construction processing is completed. As a result, various information relating to the image data shown in FIG. 4 is stored in the hard disk.

  For example, as shown in FIG. 4, when the content of the image data (subject) is Tokyo Tower, the position information relevance PR is PR = 0.95 (type A), and the GPS information is the longitude of the shooting location. The latitude and area range R is set to R = 0.012, YCbCr information as an image feature amount or information obtained by processing this, and “Tokyo Tower” and “Shiba Koen” are set as keywords and stored in the hard disk. That is, the hard disk storing such information becomes the database 20. Through the above processing, the constructed database 20 is used as a dictionary, and the image recognition device 15 recognizes the contents of arbitrary image data.

C. Structure of image recognition device:
FIG. 5 is a schematic structural diagram of an image recognition apparatus 15 provided with the image recognition processing program of the present invention as software. As shown in the figure, the image recognition device 15 captures a predetermined image to be processed, inputs the digital still camera 27 storing the image data, and the image data stored in the digital still camera 27 to perform predetermined processing. Computer 30 (hereinafter referred to as PC 30), a display 40 for displaying images processed by the PC 30, a keyboard 41, a mouse 42, and the like as a user interface.

  The digital still camera 27 includes a CCD as an image sensor, an image processing circuit that performs predetermined image processing on image data captured via the CCD, a memory card 24 that stores image data, and the like, and a digital still using GPS. A GPS function for storing geographical position information of the camera 27 is provided. With this function, GPS information is added to the image data.

  In general, the data structure of an image handled by a digital still camera is a so-called Exif format. Based on image data in JPEG format, predetermined image additional information such as shooting information when shooting image data, thumbnail images, and the like is used. Embedded in a format that conforms to the rules. As the image additional information, various information such as the shooting date and time, the exposure time at the time of shooting, the aperture value, the shutter speed, the ISO sensitivity, and the white balance are stored. In the present embodiment, in addition to these, GPS information is included as one of the image additional information. Note that the GPS information may be written in a maker note that is set as a user's usage range according to the Exif convention.

  If the image data to be handled is provided with position information, instead of obtaining from the digital still camera 27, the digital video camera 21, the camera-equipped mobile phone 23, the memory card 24, the hard disk 25, etc. It may be acquired from an imaging device or a recording medium.

  The PC 30 includes a CPU 31, a ROM 32, a RAM 33, a hard disk 34, an I / F circuit unit 35, and the like, and each device is connected by an internal bus. The I / F circuit unit 35 is connected to the digital still camera 27, the display 40, the keyboard 41, and the mouse 42, and functions as an interface between these external devices and the PC 30.

  A predetermined operation system is stored in the ROM 32, and when the PC 30 is turned on, the CPU 31 reads and starts the operation system OS in the ROM 32.

  Various application programs that operate on the operating system OS are installed in the hard disk 34. In this embodiment, an image recognition processing program is installed as one of application programs. The CPU 31 receives a user operation instruction via the keyboard 41 and the mouse 42, reads out this program, develops it on the RAM 33, and executes it.

D. Image recognition processing:
FIG. 6 is a flowchart of the image recognition process. This process is a process of extracting a keyword corresponding to the content of arbitrary image data, in other words, a process of recognizing the content of input arbitrary image data, and is executed by a user instruction via a keyboard.

  When the process is started, the CPU 31 acquires one piece of image data (step S600). Specifically, simultaneously with the start of processing, image data corresponding to an arbitrary sheet designated by the user is acquired from the images in the digital still camera 27 displayed on the display.

  CPU31 which acquired the one image data extracts the positional information (GPS information) incidental to the image data (step S605). The extracted GPS information is information on the longitude and latitude of the shooting location where one image was taken.

  Subsequently, the CPU 31 extracts an image feature amount of one image data (step S610). In this embodiment, as described above, the image feature amount is extracted by using the color arrangement method. Specifically, the CPU 31 divides the image data into 8 × 8 small blocks, and sets the average value (Rave, Gave, Bave) of the pixel data values (R, G, B) constituting each small block. Used as representative values (Rrep, Grep, Brep) for small blocks. The CPU 31 creates reduced pixel data composed of 8 × 8 pixel data using the obtained representative value. The CPU 31 converts the generated reduced image data (RGB data) into YCbCr data, and further performs discrete cosine transform (DCT) for each component of Y, Cb, and Cr. The image data is converted into frequency components by DCT. Since the discrete cosine transform is a conversion process well known to those skilled in the art, detailed description thereof is omitted.

For example, as shown in FIG. 7, an 8 × 8 coefficient value is calculated for each of Y, Cb, and Cr components. That is,
Yc [8] [8] = {yc00, yc01, yc02 ... yc76, yc77}
Cbc [8] [8] = {cbc00, cbc01, cbc02 ... cbc76, cbc77}
Crc [8] [8] = {crc00, crc01, crc02 ... crc76, crc77}
Thus, the positional relationship with the image data (image) is, for example, as shown in FIG. As shown in FIG. 7, the frequency increases as the coefficient value increases.

  Note that various known methods such as a method using a histogram of each of R, G, and B components, a method using maximum / minimum luminance values, and an average luminance value may be used for acquiring the image feature amount. In this case, the image feature amount may be extracted by the same method on the database 20 side.

  Subsequently, the CPU 31 initializes the reference value of the reliability coefficient (step S620). As described above, the reliability coefficient TC1 is an index representing the degree of similarity of the image feature amount of the image data, and is calculated in a processing step described later. Here, first, the calculated reliability coefficient TC1 is evaluated. The reference value TCmax is initialized to zero.

  The CPU 31 selects one target from the database 20 (step S630). As described above, the database 20 associates various pieces of information such as position information relevance PR, GPS information, region range R, image feature amount, keyword, and the like as a group (information group) with a single piece of image data. Remember the group. This step is a process of selecting one information group from a plurality of information groups in the database 20, and can be said to be a process of selecting image data associated with one information group indirectly. Therefore, in this step, it is assumed that one image data is selected. Hereinafter, the selected image data is referred to as a “database image” in order to distinguish it from the one image data acquired in step S600.

  CPU31 which selected the database image judges whether all the database images in the database 20 were selected (step S635). The CPU 31 sets a flag to execute processing to be described later on the selected database image, and determines whether it has been selected and processed by checking the flag.

  If it is determined in step S635 that all database images have been selected (Yes), the search in the database 20 has been completed, and the series of image recognition processing is terminated.

  On the other hand, if it is determined in step S635 that all database images have not been selected (No), the position information relevance level PR of the selected database image is extracted, and the type (A To C) is determined (step S645).

  If the position information relevance PR corresponds to type B or type C in step S645 (when PR is smaller than 0.9), the process proceeds to step S660. That is, for a database image whose position information relevance PR is not so high (likely type A, the possibility of being an object specific to the place is not high), a keyword for extracting one acquired image data keyword is used. Search target.

  On the other hand, if the position information relevance PR corresponds to type A (when PR is 0.9 or more) in step S645, the GPS information and the area range R of the selected database image are extracted. Then, it is determined whether or not the longitude and latitude of the shooting location based on the GPS information of one image data already extracted (step S605) are within the range of longitude and latitude taking into account the region range R of the database image. (Step S655).

  If it is determined in step S655 that it does not fall within the longitude and latitude ranges of the database image (No), a processed flag is set for the currently selected database image, and the process returns to step S630 to return to the database. The next database image is selected from 20.

  That is, the database image classified as type A includes a subject that is an object specific to the place, so that it is determined that the database image and arbitrary image data have almost the same content. The condition is that the shooting locations of both are close.

  If the longitude and latitude of the shooting location of one image data does not fit within the range (see FIG. 3) that includes the region range R centered on the longitude and latitude of the database image, the keyword of the database image However, the image content of one image data is not appropriately expressed. Therefore, for a database image that is type A and whose longitude and latitude position information does not satisfy the condition, a process of excluding it from the search target is performed.

  On the other hand, if it is determined in step S655 that the database image falls within the longitude and latitude ranges (Yes), the process proceeds to step S660. That is, only database images that are of type A and satisfy the conditions of the position information of longitude and latitude (similar to the shooting location) are set as search targets.

  The CPU 31 that has searched for a database image of type B or C or a database image of type A that satisfies the position information extracts the image feature amount of the database image and already extracts it in step S610. The distance D between the image feature amounts of both images is calculated from the image feature amount of the image data (step S660).

  The distance D of the image feature amount represents the similarity or dissimilarity between the two images. For example, when the Euclidean distance is used, it is calculated by the following formula.

  Ycp, Cbcp, and Crcp indicate the image feature amount (coefficient) of the database image, and Ycq, Cbqq, and Crcq indicate the image feature amount (coefficient) of the image data. In this embodiment, the image feature amount (coefficient) is adopted in MPEG-7. Since the image feature amount is used, the variables i and j both take values of 0 to 7 (n = 7).

  The CPU 31 having calculated the distance D calculates a reliability coefficient TC (step S670). The reliability coefficient TC is a coefficient determined according to the difference between the image feature quantity of the database image and the image feature quantity of the image data, that is, the distance D, and the image feature quantities of both are close (distance D is equal to The smaller the value, the higher the reliability and the higher the coefficient. Specifically, it is calculated by the following formula.

  Dref is one reference value with respect to the distance D, and is set in advance as a reference value with which it is possible to determine that both image data are substantially the same (having high similarity). That is, in this equation, when the distance D is equal to or less than Dref and the reliability coefficient TC exceeds 1, it is determined that the two image data are substantially the same. Therefore, when the reliability coefficient TC exceeds 1, the reliability coefficient TC is set to 1, and the range that the reliability coefficient TC can take is set to 0 ≦ TC ≦ 1.0.

  Subsequently, the CPU 31 corrects the calculated reliability coefficient TC (step S680). Specifically, correction is performed by the following equation using the position information relevance PR.

  In this correction, when the position-related information degree PR is a large value, the reliability coefficient TC calculated in the above step is corrected to a larger reliability coefficient TC1, and the reliability is high (one This is a database image with an appropriate keyword representing the image content of the image data). That is, when the position information based on the GPS information of both is close and the database image is of type A or type B, the position information relevance PR is a relatively large value and the reliability coefficient TC1 is also a large value. (Value close to 1).

  Note that the database image that is a target for calculating the reliability coefficient TC1 is the type B, C, or type A and the position information satisfies the conditions from Steps S645 and S655. Therefore, even if the position information relevance level PR is high, a database image that is determined not to be photographed at a location close to the position information based on GPS information is not a processing target. There is no problem even if correction is performed using.

  Even when the database image is type C, there is no problem in correction because the position information relevance PR is zero and the new reliability coefficient TC1 remains the original reliability coefficient TC. By performing such correction, although the degree of similarity of the image feature amount of the image data itself is high, an appropriate reliability coefficient TC1 can be calculated even when the subjects are completely different subjects photographed at different places.

  Subsequently, the CPU 31 determines whether or not the corrected reliability coefficient TC1 is larger than the currently set reliability coefficient reference value TCmax (step S685).

  If it is determined in step S685 that the reliability coefficient TC1 is greater than the reference value TCmax (Yes), the reference value TCmax is replaced with the reliability coefficient TC1, and the keyword of the current database image to be processed is extracted. The keyword is temporarily stored (step S690), and the process returns to step S630 to repeat a series of processes for the next database image. The extracted keyword is rewritten to a keyword of a new database image to be processed every time the reliability coefficient reference value TCmax is replaced.

  On the other hand, if it is determined in step S685 that the reliability coefficient TC1 is equal to or less than the reference value TCmax (No), the process returns to step S630 without storing the keyword, and a series of processing is performed on the next database image. repeat. Note that the CPU 31 sets a processed flag for the database image to be processed before returning to step S630 by the processing of steps S685 and S690.

  In this way, a series of processes are repeated, and after all the database images in the database 20 are processed, the process returns from step S635 to END, and the image recognition process ends. At this time, the CPU 31 displays the temporarily stored extracted keyword on the display 40.

  According to the above image recognition processing, the processing is repeated for all the database images, and the keyword corresponding to the database image having the highest reliability coefficient TC1 is extracted. Therefore, it is possible to extract an appropriate keyword representing the image content of any one image data.

  Further, in the image recognition processing of the present embodiment, processing for narrowing down the search target according to the position information relevance PR is performed, such as excluding database images that are type A and whose positional information does not satisfy the predetermined condition from the search target. That is, before determining the degree of similarity using the image feature amount, it is possible to reduce the number of objects to be processed in consideration of the position information relevance level PR. Therefore, the keyword search speed can be improved and the search time can be shortened.

  Furthermore, by narrowing down the search target according to the position information relevance PR, it is possible to exclude database images that cannot be appropriate keywords representing the image content of any one image data. Therefore, the probability of extracting an appropriate keyword is improved, and the accuracy (recognition rate) for recognizing the image content of arbitrary image data can be improved.

  In the image recognition processing of the present embodiment, the reliability coefficient TC1 corrected in consideration of the position information relevance PR in addition to the image feature amount of the database image is obtained. In other words, not only the image feature amount but also the correction for improving the reliability of the database image that is determined to be close to the photographed position is performed. Therefore, by simply evaluating the corrected reliability coefficient TC1, an appropriate keyword can be extracted, and the recognition rate of the image content can be improved.

  In this embodiment, the positional information relevance PR = 0.9 is used as one criterion for largely classifying the image data constituting the database image. However, PR is not limited to 0.9. For example, a value larger than 0.9 may be used as a reference.

  The keywords thus extracted by the image recognition processing may be assigned to the image data itself (for example, manufacturer note in Exif format). By doing so, it can be effectively used for image data management and classification at a later date. Further, it may be used for database construction.

  In the present embodiment, the database 20 has been described as being independent of the image recognition device 15, but the image recognition device 15 may be configured to include the database 20. For example, the hard disk 34 may be used as the database 20 by storing various information related to the image data shown in FIG. 4 in the hard disk 34 in the PC 30 in which the image recognition processing program is installed as the image recognition device 15. Even if the image recognition apparatus 15 is configured in this manner, it is possible to quickly and appropriately extract keywords.

E. Variation:
In the present embodiment, description has been made assuming that one keyword is extracted by image recognition processing, but a plurality of keywords may be extracted. For example, in step S690 in FIG. 6, instead of the process of sequentially rewriting the extracted keywords, the extracted keywords corresponding to the third highest reliability coefficient TC1 may be stored and extracted as candidates. By displaying the plurality of extracted keywords on the display 40 and making the application for the user to select one, more appropriate keywords can be extracted for any image data.

  In the present embodiment, the database 20 is described as having stored therein feature quantities that are information such as position related information PR, GPS information, region range R, image feature quantities, and keywords. The data itself may be stored. By doing so, it becomes easy to associate various information with image data.

  In the present embodiment, the processing of one piece of image data designated by the user on the application has been described. However, the image recognition processing is sequentially executed for all image data copied from the digital still camera to the hard disk. May be. By doing so, it is possible to save the user operation in the image recognition process and improve the convenience.

  Furthermore, in this embodiment, the database 20 is constructed on the hard disk of the computer, but for example, the database 20 may be constructed on a server on the network. In this case, the I / F circuit 35 of the PC 30 may be provided with a network connection function. By doing so, a large-capacity database 20 can be constructed.

  As mentioned above, although embodiment of this invention was described, this invention is not limited to such embodiment at all, Of course, it can implement with various forms within the range which does not deviate from the meaning of this invention. is there. In this embodiment, the image recognition process is executed in the form of a software program. However, a hardware circuit including a logic circuit that executes each of the above processes (steps) may be used. By doing so, the load on the CPU 31 can be reduced and each process can be executed at a higher speed.

It is explanatory drawing which shows the image recognition system containing the image recognition apparatus of one Example. It is a flowchart which shows the flow of the construction process of a database. It is explanatory drawing of the area | region range R. FIG. It is explanatory drawing of the various information of the image data accumulate | stored in a database. It is a schematic structure figure of an image recognition device. It is a flowchart of an image recognition process. It is explanatory drawing of the feature-value of an image.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Database creation apparatus 11 ... Keyword setting part 12 ... Area range setting part 13 ... Position information relevance setting part 14 ... Image feature-value extraction apparatus 15 ... Image recognition apparatus 16. .. Image feature extraction device 17 ... Discriminator 18 ... Reliability coefficient corrector 20 ... Database 21 ... Digital video camera 23 ... Mobile phone 24 ... Memory card 25 ... Hard disk 27 ... Digital still camera 30 ... Computer 31 ... CPU
32 ... ROM
33 ... RAM
34 ... Hard disk 35 ... I / F circuit part 40 ... Display 41 ... Keyboard 42 ... Mouse 100 ... Image recognition system PR ... Position information relevance R ... Area range TC ... Reliability factor TC1 ... Reliability factor

Claims (10)

An image recognition apparatus for executing processing for extracting a keyword representing image content of one image data having information on a shooting position from a predetermined search target and recognizing the image content,
For each of the plurality of target image data to be searched, an image feature amount, a keyword representing the image content, position information where an image having the image feature amount is photographed, and an object to which the keyword is assigned are specified. Storage means for associating a position information relevance level indicating a degree unique to a place and storing it as a database;
Image data input means for inputting the one image data as acquired image data and extracting position information where the acquired image data is captured;
Image feature amount extraction means for extracting an image feature amount included in the acquired image data;
Using the degree of relevance of the position information of the target image data stored in the database, the position information of the acquired image data, and the shooting position distance obtained from the position information of the target image data, the target image data is Exclusion determination means for determining whether or not to exclude from the keyword extraction process,
If it is determined that the image is to be processed, the image feature quantity of the acquired image data is compared with the image feature quantity of the target image data, and keywords of the target image data satisfying a predetermined condition are extracted from the database. An image recognition apparatus comprising: a keyword extraction unit. The image recognition apparatus according to claim 1,
The position information relevance is
A) Type A in which the object to which the keyword in the target image data is assigned is unique to a specific place;
B) Type B in which the object to which the keyword in the target image data is assigned is high in a certain place and the place where the object exists is limited;
C) The object to which the keyword in the target image data is assigned is classified and set according to three types of type C that are not unique to a specific place,
The exclusion determination unit determines to exclude the target image data from the processing target when the position information relevance of the target image data is the type A and the distance between the photographing positions is a predetermined amount or more. Image recognition device. The image recognition apparatus according to claim 2, further comprising:
The storage means associates, for each target image data, a region range indicating a geographical range in which an object to which a keyword is assigned in the target image data, and stores the range as the database.
The exclusion determination unit determines that the gap between the photographing positions is greater than or equal to a predetermined value when the position information of the acquired image data does not fall within the range of the target image data stored in the database. apparatus. The image recognition device according to any one of claims 1 to 3,
The keyword extracting means includes
A reliability coefficient calculation unit that calculates a reliability coefficient indicating a degree of similarity between the acquired image data and the target image data from the image feature amount of the acquired image data and the image feature amount of the target image data When,
An image recognition apparatus comprising: an extraction unit that extracts a keyword of the target image data based on the calculated reliability coefficient. The image recognition apparatus according to claim 4, further comprising:
The keyword extracting means includes
A correction unit that corrects the reliability coefficient by adding the position information relevance to the reliability coefficient calculated using the image feature amount;
The extraction unit is an image recognition device that extracts the keyword based on the corrected reliability coefficient. The image recognition device according to claim 5,
The image recognizing apparatus, wherein the extraction unit extracts a keyword of the target image data having the highest reliability coefficient among the corrected reliability coefficients obtained for each target image data to be searched. For each of a plurality of target image data to be searched, an image feature amount, a keyword representing the image content, position information at which an image having the image feature amount is captured, and an object to which the keyword is assigned is a specific location A keyword representing the image content of one piece of image data having the position information of the photographing position is extracted from the database using a database that stores the position information relevance indicating the degree unique to the image. An image recognition apparatus that executes processing and recognizes image contents,
Image data input means for inputting the one image data as acquired image data and extracting position information where the acquired image data is captured;
Image feature amount extraction means for extracting an image feature amount included in the acquired image data;
Using the degree of relevance of the position information of the target image data stored in the database, the position information of the acquired image data, and the shooting position distance obtained from the position information of the target image data, the target image data is Exclusion determination means for determining whether or not to exclude from the keyword extraction process,
If it is determined that the image is to be processed, the image feature quantity of the acquired image data is compared with the image feature quantity of the target image data, and keywords of the target image data satisfying a predetermined condition are extracted from the database. An image recognition apparatus comprising: a keyword extraction unit. An image recognition method for recognizing image content by executing processing for extracting a keyword representing the image content of one piece of image data having information on a shooting position from a predetermined search target,
For each of the plurality of target image data to be searched, an image feature amount, a keyword representing the image content, position information where an image having the image feature amount is photographed, and an object to which the keyword is assigned are specified. Associating with the location information relevance indicating the degree unique to the place, storing it as a database,
The one image data is input as acquired image data, and position information where the acquired image data is photographed is extracted,
Extracting an image feature amount included in the acquired image data;
Using the degree of relevance of the position information of the target image data stored in the database, the position information of the acquired image data, and the shooting position distance obtained from the position information of the target image data, the target image data is Decide whether or not to exclude them from the keyword extraction process,
If it is determined that the image is to be processed, the image feature quantity of the acquired image data is compared with the image feature quantity of the target image data, and keywords of the target image data satisfying a predetermined condition are extracted from the database. Image recognition method. A computer program for controlling an image recognition apparatus for recognizing image content by executing processing for extracting a keyword representing the image content of one piece of image data having information on a shooting position from a predetermined search target,
For each of the plurality of target image data to be searched, an image feature amount, a keyword representing the image content, position information where an image having the image feature amount is photographed, and an object to which the keyword is assigned are specified. A function of associating a location information relevance level indicating a degree unique to a place and storing it as a database;
A function of inputting the one image data as acquired image data and extracting position information where the acquired image data is captured;
A function of extracting an image feature amount included in the acquired image data;
Using the degree of relevance of the position information of the target image data stored in the database, the position information of the acquired image data, and the shooting position distance obtained from the position information of the target image data, the target image data is A function to determine whether to exclude from the keyword extraction process,
If it is determined that the image is to be processed, the image feature quantity of the acquired image data is compared with the image feature quantity of the target image data, and keywords of the target image data satisfying a predetermined condition are extracted from the database. A computer program that causes the image recognition apparatus to realize functions.   A recording medium in which the computer program according to claim 9 is recorded in a computer-readable manner.

Images