JP2007304771A - Subject recognition device, subject recognition program and image retrieval service provision method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technology for achieving the efficiency of the subject recognition of image data. <P>SOLUTION: This subject recognition device is provided with an image input part, a subject dictionary part and a processing part. An image input part captures image data. A subject dictionary part classifies subject conditions for specifying the type of the subject into hierarchies for storage. A processing part successively collates the subject conditions according to the hierarchy, and narrows down subject conditions to be collated, and searches the type of the subject projected on the image data. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a subject recognition device, a subject recognition program, and an image search service providing method.

2. Description of the Related Art Conventionally, a technique for recognizing a face portion shown in image data, such as an image face recognition technique, is known.
Patent Document 1 below discloses an apparatus that recognizes a subject from image data and displays subject information together with the image data.
On the other hand, search sites on the Internet provide an image search service by searching for text information (image file name, shooting date / time information, etc.) attached to image data.
Japanese Patent Laid-Open No. 2001-113097

  Like the face recognition technology described above, a technology for recognizing a subject from image data is progressing day by day, and it is expected that the types of subjects to be recognized will increase in the future.

  On the other hand, in the conventional patent document 1, it is necessary to perform subject recognition one by one for each kind of subject. For this reason, if the types of subjects to be recognized become enormous, the number of times of subject recognition processing becomes enormous and the total processing time for subject recognition becomes longer.

  Therefore, an object of the present invention is to provide a technique for efficiently recognizing a subject of image data.

<< 1 >> The subject recognition apparatus of the present invention includes an image input unit, a subject dictionary unit, and a processing unit.
The image input unit captures image data.
The subject dictionary unit stores subject conditions for specifying the subject type.
The processing unit collates the image data with the subject condition, and identifies the subject type shown in the image data.
In particular, in the present invention, the subject dictionary unit stores subject conditions classified into hierarchies. The processing unit collates the image data with the subject condition of the lower layer while narrowing down the subject conditions located at the lower layer based on the collation result between the subject condition located at the upper layer and the image data.
<< 2 >> Preferably, the processing unit records keyword information predetermined in association with the recognized subject type in association with the image data. Further, the processing unit performs an image search by comparing the input keyword with this keyword information.
<< 3 >> Preferably, the subject dictionary unit has a function of additionally registering the subject type Kd and the subject condition Cd. In response to this additional registration, the processing unit obtains a higher-level subject type Ku that includes the subject type Kd, and additionally performs collation with the subject condition Cd for the image data for which the subject type Ku has been detected.
<< 4 >> Also preferably, the subject dictionary unit has a function of managing version information of subject conditions. The processing unit records the version information of the subject condition used for specifying the subject type in association with the image data.
<< 5 >> The subject recognition program of the present invention is a program for causing a computer to function as the subject recognition device according to any one of claims 1 to 4.
<< 6 >> The image search service providing method of the present invention is a method in which a computer executes an image search service, and includes the following steps.
(1) A step of classifying and storing subject conditions for specifying a subject type in a hierarchy.
(2) A step of obtaining a subject type shown in the image data by sequentially comparing the image data with subject conditions according to the hierarchy.
(3) A step of recording keyword information predetermined in relation to the recognized subject type in association with the image data.
(4) A step of performing an image search by collating the input keyword with keyword information corresponding to the image data.

  In the present invention, subject recognition of image data is performed while tracing subject conditions classified into hierarchies from an upper hierarchy to a lower hierarchy. In this case, subject recognition is performed so as to narrow down the subject condition, and it is possible to omit subject recognition of a lower hierarchy with a low possibility. As a result, the number of subject recognition processes can be reduced, and efficient subject recognition can be performed for many types of subjects.

[Description of configuration]
FIG. 1 is a block diagram showing the configuration of the image search system 11.
In FIG. 1, an image search system 11 is configured by communicably connecting a subject recognition device 12, an image database 13, and a terminal device 14 (such as a user's personal computer) via a communication line such as the Internet.

  In addition, you may construct | assemble the system which searches the image data memorize | stored in a computer at high speed by implement | achieving this image search system 11 with a computer single-piece | unit.

Further, the subject recognition device 12 includes an image input unit 21, a subject dictionary unit 22, and a processing unit 23. Further, the processing unit 23 includes a feature amount calculation unit 31, a feature amount storage unit 32, a control unit 33, and a subject keyword storage unit 34.
Note that a part or all of the subject recognition apparatus 12 may be realized in software by executing a subject recognition program on a computer. Further, part or all of the subject recognition device 12 may be realized by hardware.
Hereinafter, the operation of the image search system 11 will be specifically described.

[Subject recognition for new images]
FIG. 2 is a flowchart illustrating subject recognition for new image data. Hereinafter, the operation will be described along the step numbers shown in FIG.

Step S1: The image input unit 21 periodically monitors the data update of the image database 13. When the image input unit 21 finds new image data to which no keyword is assigned in the image database 13 by this monitoring, the image input unit 21 takes in the image data.

Step S2: The feature amount calculation unit 31 extracts an image feature amount from the captured image data in order to perform subject recognition efficiently. This image feature amount is used for collation with subject conditions described later. The extracted image feature quantity is stored in the feature quantity storage unit 32.
For example, the following data is effective as this kind of image feature amount.

(1) Hue histogram of image: Used when recognizing landscape subject / person subject by color tendency.
(2) Image extraction edge: Used to recognize a specific subject based on the edge shape.
(3) Used for recognizing a closed region such as a face or an object where the contour shows a closed curve.
(4) Partial image of the face (skin color) part ... Used for recognition of specific individuals.
(5) Color near the lips: Used to recognize men and women.
(6) Color near hair: Used to recognize men and women and the elderly.
(7) Texture in image: Used to recognize eyes / nose / mouth / clothing pattern / meadow / forest.
(8) Vertical line frequency: Used to recognize buildings / forests.
(9) Horizontal line across the screen: Used to recognize horizontal / horizontal lines.
(10) Estimated color temperature of light source: Used to recognize indoor / outdoor conditions.
(11) Used for partial image pattern matching of image data.
(12) GPS information, shooting date and time, subject distance, exposure information, audio data, information indicating the vertical and horizontal directions of the image, accompanying information such as focus control information

Step S3: In the subject dictionary of the subject dictionary unit 22, subject conditions for specifying the subject type are registered in advance in a hierarchical structure as shown in FIG. The control unit 33 selects the highest hierarchy from the hierarchical structure of the subject dictionary and sets it as the search hierarchy.

Step S4: The control unit 33 reads out the subject conditions for each subject type belonging to the search hierarchy from the subject dictionary unit 22, and collates them one by one with the image feature amount.
For example, in FIG. 3, in the highest search hierarchy, a subject type of a parallel concept such as a person / landscape (the depth from the highest hierarchy is equal) is defined.
For example, for a person, if a subject condition such as the appearance of a high frequency distribution of skin color appears in the hue histogram, it is determined that the person subject appears in the image data.
For a landscape, if a subject condition such as a high-frequency distribution of blue or green appears in the hue histogram and the subject distance is far away, it is determined that a landscape subject is reflected in the image data.
With the operation as described above, whether each of the parallel concept subject types belonging to the search hierarchy is reflected in the image data is determined.

Step S5: The control unit 33 records a keyword related to the subject type that satisfies the condition in the subject keyword storage unit 34 in association with the image data.

Step S6: The control unit 33 omits the lower hierarchy included in the subject type that does not satisfy the condition from the reference range of the subject dictionary. For example, when a person's subject condition is not satisfied, a lower hierarchy included in the person subject (in FIG. 3, a specific individual, male, female, etc.) is omitted from the reference range of the subject dictionary. In this way, by limiting the reference range of the subject dictionary according to the subject recognition result of the upper layer, subject recognition can be performed efficiently while narrowing down to highly likely subject types.

Step S7: Here, the control unit 33 determines whether or not the search hierarchy of the subject dictionary has reached the lowest hierarchy.
If the lower hierarchy still remains below the search hierarchy, the control unit 33 shifts the operation to step S8 in order to perform subject recognition for this lower hierarchy.
On the other hand, if the search hierarchy has already reached the lowest hierarchy, the control unit 33 regards the new image data as having been subjected to all subject recognition, and proceeds to step S9.

Step S8: The control unit 33 lowers the search hierarchy by one and returns the operation to step S4.

Step S9: The control unit 33 acquires the version information of the subject dictionary from the subject dictionary unit 22, and records it in the subject keyword storage unit 34 in association with the image data.
With the above operation, the subject recognition and keyword assignment processing for image data to which no keyword is assigned are completed.

[Process associated with subject dictionary update]
FIG. 4 is a flowchart for explaining processing associated with dictionary update. Hereinafter, the operation will be described along the step numbers shown in FIG.

Step S21: The subject dictionary unit 22 has a function of additionally registering the following items in the subject dictionary.

(1) Subject type Kd
(2) Subject condition Cd for specifying the subject type Kd
(3) Keyword related to the subject type Kd (4) Hierarchy position in the subject dictionary of the subject type Kd

  The user can customize the subject dictionary by additionally registering these items via the terminal device 14. Moreover, the subject dictionary can be periodically upgraded by performing this additional registration on the manager side of the image search system 11.

Note that the registration of the hierarchical position may be omitted because knowledge of the hierarchical structure of the subject dictionary is required. Registration of the subject condition Cd may be simplified by registering a part or all of a representative image corresponding to the subject type Kd.
On the other hand, each time such additional registration is performed, the subject dictionary unit 22 stores the updated contents of the subject dictionary as a history and updates version information indicating the version number of the subject dictionary.

Step S22: The control unit 33 determines whether or not a layer position is additionally registered for the subject type Kd.
Here, when the registration of the hierarchy position is omitted, the control unit 33 shifts the operation to step S24 in order to check the hierarchy position of the subject type Kd.
On the other hand, when the layer position of the subject type Kd is registered, the control unit 33 shifts the operation to step S23.

Step S23: The control unit 33 obtains a subject type Ku one level higher than the subject type Kd according to the registered hierarchical position. After this operation, the control unit 33 shifts the operation to step S27.

Step S24: Since the hierarchical position of the subject type Kd is unknown, the control unit 33 first performs subject recognition as a test. That is, the control unit 33 performs a comparison between the image feature amount and the subject condition Cd for a predetermined number of image data (test images) that have been subject-recognized in the past.

Step S25: The control unit 33 obtains a subject type Ku that logically includes the subject type Kd based on the collation result in step S24 and the past subject recognition result of the test image.

Step S26: The control unit 33 rewrites the subject dictionary in the subject dictionary unit 22 and sets the layer position of the subject type Kd as the lower layer of the subject type Ku.

Step S27: The control unit 33 selects an image data group that is determined to include the subject type Ku in past subject recognition.

Step S28: The control unit 33 sequentially collates the image feature amount of the image data group selected in Step S27 with the subject condition Cd, and selects image data that satisfies the condition.

Step S29: The control unit 33 records a keyword related to the subject type Kd in the subject keyword storage unit 24 in association with the image data that satisfies the condition in step S28.

Step S30: The control unit 33 acquires the version information of the subject dictionary from the subject dictionary unit 22, and records it in the subject keyword storage unit 34 in association with the image data selected in step S27.
With the above operation, a new subject keyword can be additionally given to the image data in accordance with the update of the subject dictionary.

[Image search operation]
FIG. 5 is a flowchart illustrating image search in the present embodiment. Hereinafter, this operation will be described along the step numbers shown in FIG.

Step S41: The user inputs a keyword related to the subject to the terminal device 14 in order to search for image data showing the desired subject. The terminal device 14 transmits this input keyword to the control unit 33.

Step S42: The control unit 33 queries the subject keyword storage unit 34 for the input keyword, and selects image data that matches or is similar to the input keyword.

Step S43: The control unit 33 arranges the thumbnails of the image data selected in Step S42 and creates a linked thumbnail page. The control unit 33 transfers the created thumbnail page data to the terminal device 14 as an image search result. The user can access the image data as the search result by browsing the thumbnail page using the terminal device 14 and following the link of the thumbnail page.
With this operation, it is possible to quickly search the image data group for the subject type related to the input keyword.

[Effects of the embodiment, etc.]
As described above, in the present embodiment, since a subject dictionary having a hierarchical structure is used, subject recognition can be performed so as to narrow down subject types. Therefore, it is possible to flexibly omit the lower-level subject recognition with low possibility according to the recognition status of the upper layer. As a result, even if the number of types of subjects to be recognized increases, the number of times of subject recognition can be efficiently reduced, and the total processing time required for subject recognition can be shortened.

  Further, in the present embodiment, the subject type reflected in the image data is recognized in advance, and a keyword related to the subject type is recorded. Therefore, it is possible to easily search for image data showing a desired subject by matching keywords.

  In this embodiment, the collation range of the additionally registered subject type Kd is narrowed down to the image data group that has been previously classified into the higher-level subject type Ku. As a result, it is not necessary to re-execute subject recognition on all past image data every time additional registration is performed, and the processing load associated with version upgrade of the subject dictionary can be reduced.

[Supplementary items of the embodiment]
In the above-described embodiment, the description is focused on the use of image search. However, it is not limited to this application. For example, the subject type shown in the image data may be determined, and image processing (including image quality adjustment) corresponding to the subject type may be performed on the image data. With this operation, an image processing apparatus that selects and executes image processing suitable for the type of subject can be realized.

  In the above-described embodiment, the image feature amount is extracted from the image data. However, the present invention is not limited to this. It may be determined whether or not the subject type is reflected in the image data by collating a template image indicating the subject type (corresponding to the subject condition) and image data by pattern matching.

  Furthermore, in the above-described embodiment, the image data may be divided into a plurality of divided areas, and subject recognition may be performed for each of the divided areas.

  Further, in the above-described embodiment, subject types are narrowed down based on the inclusion relationship of the upper and lower layers, and the number of subject recognition is omitted. However, the present invention is not limited to this.

  For example, when subject recognition is performed for a person and a woman, it is possible to classify a subject that is a person and is determined not to be a woman as male or other. In this way, by narrowing down the possibility of the subject type based on the parallel concept exclusive relationship, the number of times of subject recognition can be efficiently eliminated.

  In the embodiment described above, the keyword and version information of the image data are recorded in the subject keyword storage unit 34. However, the present invention is not limited to this. Keyword information and / or version information may be recorded in a file of image data (for example, EXIF data).

  In the above-described embodiment, the description has been made on the assumption of a still image, but the present invention is not limited to this. It can be similarly applied to subject recognition of moving images. For example, it is possible to determine a scene in which a person is shown in one moving image file, and further to determine a scene in which a woman is shown from the person scene.

  As described above, the present invention is a technique that can be used for an image search system, a subject recognition device, and the like.

1 is a block diagram illustrating a configuration of an image search system 11. FIG. 6 is a flowchart for explaining subject recognition for new image data. It is a figure which shows an example of the hierarchical structure of a to-be-photographed dictionary. It is a flowchart explaining the process accompanying a dictionary update. It is a flowchart explaining operation | movement of an image search.

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 ... Image search system, 12 ... Subject recognition apparatus, 13 ... Image database, 14 ... Terminal device, 21 ... Image input part, 22 ... Subject dictionary part, 23 ... Processing part, 24 ... Subject keyword memory | storage part, 31: ... Feature Calculation unit, 31... Feature amount calculation unit, 32... Feature amount storage unit, 33... Control unit, 34.

Claims (6)

An image input unit for capturing image data;
A subject dictionary section for storing subject conditions for specifying a subject type;
A processing unit that collates the image data with the subject condition and identifies the subject type shown in the image data,
The subject dictionary unit classifies and stores the subject conditions in a hierarchy,
The subject recognition apparatus characterized in that the processing unit performs collation while narrowing down subject conditions located in a lower layer based on a collation result of subject conditions located in an upper layer. The subject recognition apparatus according to claim 1,
The processor is
Predetermined keyword information related to the recognized subject type is recorded in association with the image data,
A subject recognition apparatus, wherein an image search is performed by comparing an input keyword with the keyword information corresponding to the image data. The subject recognition apparatus according to any one of claims 1 to 2,
The subject dictionary unit has a function of additionally registering the subject type Kd and the subject condition Cd,
The processing unit obtains a higher-level subject type Ku that includes the subject type Kd according to the additional registration in the subject dictionary unit, and the image data for which the subject type Ku has been detected is compared with the subject condition Cd. An object recognition apparatus characterized by additionally performing collation. The subject recognition apparatus according to any one of claims 1 to 3,
The subject dictionary unit has a function of managing version information of the subject condition,
The processing unit records version information of the subject condition used for specifying the subject type in association with the image data.   A subject recognition program for causing a computer to function as the subject recognition device according to any one of claims 1 to 4. A method for a computer to implement an image search service, comprising:
Categorizing and storing subject conditions for specifying a subject type in a hierarchy;
Obtaining a subject type shown in the image data by sequentially comparing the image data with the subject condition according to the hierarchy;
Recording predetermined keyword information related to the recognized subject type in association with the image data;
A method for providing an image search service, comprising: comparing an input keyword with the keyword information corresponding to the image data and performing an image search.

Images