JP4217954B2 - Image search device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image search apparatus for searching for a specific image portion such as a face portion from image data such as a photograph.
[0002]
[Prior art]
In recent years, it has been considered that a specific object included in a photograph or the like, for example, a part such as a human face is specified and predetermined processing is performed based on the specified part. As an example, detect the face part of each person from the photographed picture and burn only the face part, or detect the face part of the person from the image being captured and use it for face authentication processing, Such a thing can be considered.
[0003]
In a device for recognizing a target object such as a conventional face image, in order to cope with the case where it is difficult to recognize the target object depending on the imaging state (tilt, size, illumination state, etc.) of the target object, Some perform processing adapted to the imaging state (reference state).
[0004]
Conventionally, in this process, specifically, a neural network is learned using learning image data taken while changing the imaging state, and a photograph that is a processing target using the learned neural network is used. It has been considered to detect how much the imaging state deviates from the reference state and perform image processing so as to correct the deviation.
[0005]
As an example of processing for detecting a desired pattern from target image data, a method such as a kernel nonlinear subspace method disclosed in Patent Document 1 is known.
[0006]
[Patent Document 1]
JP 2001-90274 A
[0007]
[Problems to be solved by the invention]
However, for example, in the case of a person's face, there are various changes such as lateral adjustment, upward adjustment, neck caulking condition, lighting condition, etc. When trying to detect a deviation from the conventional reference state, a neural network A large amount of learning image data used for learning is required in accordance with the various changes described above. Moreover, as a result of learning with such a large amount of image data, the scale of the neural network has become enormous, and it has been impossible to complete the processing within a realistic time.
[0008]
Further, in a conventional apparatus for recognizing a target object, processing is performed using the entire photograph or the like as a search source for the target object as a search range. For this reason, the amount of data to be processed also increases, and the processing load is great.
[0009]
The present invention has been made in view of the above circumstances, and it is an object of the present invention to provide an image search apparatus capable of reducing the processing load for searching for an object to be searched from a photograph or the like and improving the search accuracy. I will.
[0010]
[Means for Solving the Problems]
  The invention according to claim 1 is an image search device for searching for a search target image data portion to be searched from within the target image data to be processed, wherein a search region is included in the target image data. Referring to the means for defining at least one and a plurality of first conversion databases learned and acquired in advance for each conversion method, the search partial data included in each of the defined search areas should be applied.N1 piecesA conversion condition including a conversion method and a conversion amount is acquired, and conversion based on the acquired conversion condition is performed at least once for each search partial data, and conversion is performed by the first conversion means. A conversion method for each of the search partial data further compared to the conversion in the first conversion meansN2 (N2 is different from N1)A limited conversion condition that limits the amount of conversion is acquired with reference to a plurality of second conversion databases that have been acquired in advance for each conversion method, and a conversion based on the acquired limited conversion condition is performed for each of the converted search parts. At least once on the dataAnd convert each search partial data to a reference stateSecond conversion means,In advance, the reference stateWith reference to a search database acquired by learning using image data, it is determined whether or not a search target is included in each of the converted search partial data, and the determination result is output. . Here, when the conversion conditions are acquired by the first and second conversion means, feature amount vector information including a predetermined set of feature amounts is calculated based on the respective search partial data, and the feature amount vector information is calculated. The number of feature quantities included in the feature quantity vector information in the first conversion means will be respectively referred to the first and second conversion databases.N1Is the number of feature quantities included in the feature quantity vector information in the second conversion meansN2The accuracy may be reduced by reducing the accuracy. According to this, the processing load in the first conversion means is reduced.
[0011]
According to a second aspect of the present invention, in the image search device according to the first aspect, preliminary search means for determining whether or not a search target is included for each search partial data converted by the first conversion means. In addition, the second conversion means converts only the search partial data that is determined to be included in the search target by the preliminary search means.
[0012]
According to a third aspect of the present invention, in the image search device according to the second aspect, the preliminary search means uses the search target image data example for the search partial data converted by the first conversion means. The search database obtained by learning is referred to and it is determined whether or not a search target is included in each of the converted search partial data.
[0013]
  According to a fourth aspect of the present invention, in the image search device according to the third aspect, the means for demarcating the search area includes entropy, hierarchical entropy, color of image data included in the area to be demarcated as the search area. And at least one of luminance dispersion are used to determine whether or not the search area is actually defined. According to a fifth aspect of the present invention, in the image search device according to the third aspect, the entropy of the image data included in the area to be defined as the search area by the means for defining the search area is a predetermined value. If it is larger than the threshold value, the area to be defined as the search area is actually defined as the search area.
[0014]
  A sixth aspect of the present invention is an image search program for searching for a search target image data portion to be searched from target image data to be processed, wherein the computer is included in the target image data. And applying to each search partial data included in each of the defined search areas with reference to a means for defining at least one search area and a plurality of first conversion databases learned and acquired in advance for each conversion method. ShouldN1 piecesA conversion condition including a conversion method and a conversion amount is acquired, and conversion based on the acquired conversion condition is performed at least once for each search partial data, and conversion is performed by the first conversion means. A conversion method for each of the search partial data further compared to the conversion in the first conversion meansN2 (N2 is different from N1)A limited conversion condition that limits the amount of conversion is acquired with reference to a plurality of second conversion databases that have been acquired in advance for each conversion method, and a conversion based on the acquired limited conversion condition is performed for each of the converted search parts. At least once on the dataAnd convert each search partial data to a reference stateA second conversion means;In advance, the reference stateRefer to the search database learned and acquired using image data, determine whether each of the converted search partial data includes a search target, and function as a means for outputting the determination result That's what it meant.
[0015]
  The invention according to claim 7 is the image search program according to claim 6, wherein for each search partial data converted by the function as the first conversion means, it is determined whether a search target is included, Only search partial data determined to include a search target is caused to function as the second conversion means.Like,Make the computer workFurther comprising meansThat's what it meant.
[0016]
  According to an eighth aspect of the present invention, in the image search program according to the seventh aspect, when functioning as the preliminary search means, the search partial data converted by the first conversion means is an example of search target image data. The search database acquired by learning is referred to to determine whether or not a search target is included in each of the converted search partial data.
[0017]
DETAILED DESCRIPTION OF THE INVENTION
[Basic configuration]
Embodiments of the present invention will be described with reference to the drawings. As shown in FIG. 1, the image search apparatus according to the embodiment of the present invention includes a control unit 11, a storage unit 12, a database unit 13, a display unit 14, an operation unit 15, and an external storage unit 16. It is implemented using a general computer configured to include This computer may be incorporated in another product, such as a camera.
[0018]
The control unit 11 operates according to a program stored in the storage unit 12, and includes a search area demarcation process that demarcates at least one search area from the target image data to be processed, and a reference state. A conversion process for conversion, a search process for detecting a search area including a search target, and a predetermined process using the search result are executed. Specific processing contents of these control units 11 will be described in detail later.
[0019]
The storage unit 12 stores software executed by the control unit 11. The storage unit 12 also operates as a work memory that holds various data required by the control unit 11 during the process. Specifically, the storage unit 12 can be realized as a storage medium such as a hard disk, a semiconductor memory, or a combination thereof.
[0020]
As will be described later, the database unit 13 includes a first conversion database 13a used in the first conversion process of the control unit 11, a second conversion database 13b used in the second conversion process, and a search database 13c used in the search process. It is a database that contains The database unit 13 is specifically a storage medium such as a hard disk, and the storage unit 12 may also serve as the database unit 13, but is illustrated here separately for the sake of explanation.
[0021]
The display unit 14 is, for example, a display device or a printer device, and displays information according to an instruction input from the control unit 11. The operation unit 15 is a keyboard or a mouse, for example, and accepts a user operation and outputs the content of the operation to the control unit 11.
[0022]
The external storage unit 16 reads a program or data from a computer-readable removable medium (a type of storage medium) such as a CD-ROM or a DVD-ROM, and outputs the program or data to the control unit 11. The processing to be stored in is performed. The program according to the present embodiment can be distributed and stored in a portable storage medium such as a CD-ROM, and is copied to the storage unit 12 using the external storage unit 16 and used. Note that the program according to the present embodiment may be copied to the storage unit 12 not only from such a storage medium but also from a server on the network via a communication unit (not shown).
[0023]
[Processing of control unit 11]
Here, the content of the process of the control part 11 is demonstrated concretely. In the present embodiment, image data (target image data) to be processed is input from the outside via the external storage unit 16 or a communication unit (not shown) and stored in the storage unit 12. Here, the target image data may be one or plural. When the user operates the operation unit 15 to instruct the control unit 11 to perform a process of searching for a search target for specific target image data (instruction to start processing), the control unit 11 2 is started.
[0024]
The control unit 11 performs a first conversion process for demarcating the target image data and demarcating search areas for each of the reduced and converted target image data and bringing the image data portion included in each search area close to the reference state. Thereafter, a second conversion process is performed to further bring the image data portion included in each search area closer to the reference state. And the control part 11 performs a search process with respect to the image data part contained in the search area | region after a 2nd conversion process.
[0025]
[First conversion process]
First, the contents of the first conversion process will be described. One of the characteristic features of this embodiment is that the first conversion process is performed step by step, that is, the conversion corresponding to one degree of freedom of conversion is performed sequentially. is there.
[0026]
In order to determine the content of conversion and the amount of conversion to be performed at each stage, the control unit 11 obtains predetermined N1-dimensional feature vector information based on the image data portion included in the currently selected search area. Calculate. Here, the feature amount vector information is a vector amount including N1 feature amount elements selected in accordance with the property of the search target.
[0027]
In the present embodiment, the control unit 11 will be described as specifying the transformation by the kernel nonlinear subspace method using the feature vector information and the feature vector information stored in the transformation database 13a.
[0028]
[Contents of the first conversion database 13a]
Since this kernel nonlinear subspace method is widely known as a method for classifying data into a certain category, a detailed description thereof will be omitted. To put it briefly, a space F1 that is stretched based on N1 feature elements is used. , Each of the plurality of subspaces Ω included in the space F1 is recognized as a category to which data is classified, and feature vector information in the space F created based on the data to be classified (for example, Φ and Are projected onto each subspace Ω (the projection result is, for example, φ), and the subspace Ω that minimizes the distance E between the feature vector information Φ before projection and the feature vector information φ after projection This is a method for detecting that (referred to as a nearest subspace) and determining that the data to be classified belongs to the category represented by the subspace Ω.
[0029]
Therefore, in the learning stage, the non-linear mapping (to the space F1) is performed so that the closest subspace Ω based on the N1 dimensional feature vector information corresponding to the example data for learning (learning sample) that should belong to the same category is the same. At least one of the mapping, that is, the parameter included in the kernel function) and the hyperplane that separates the subspaces Ω corresponding to each category is adjusted.
[0030]
In the present embodiment, this conversion database 13a is formed in order to determine a method (type and amount of conversion) for converting the search target into the reference state. That is, the conversion database 13a has been learned and acquired so that the amount (category) of conversion can be determined for each type of conversion to be performed on image data for which it is unknown whether or not it is in the reference state. In the present embodiment, a conversion database 13a is created for each type of conversion (degree of freedom) to be performed on an image, such as image rotation, parallel movement, and size change. The conversion database 13a corresponding to each degree of freedom of conversion is acquired by learning the conversion amount of the corresponding conversion as a category.
[0031]
In order to acquire this learning, a learning sample is generated as follows in the learning process of the conversion database 13a of the present embodiment. That is, a plurality of examples of image data to be searched in a predetermined reference state are prepared, and for each example of image data, conversion is performed with different conversion amounts for each degree of freedom. A plurality of converted image data is generated. The converted image data generated for each degree of freedom is used as a learning sample for each degree of freedom. Specifically, when a face is a search target, a plurality of face image data in a predetermined reference state (predetermined shooting condition / posture) is prepared as an example, and for each image data, for example, rotation / For each degree of freedom such as parallel movement and size change, if it is a rotation, the converted image data rotated by an angle of 5 degrees in the range from -180 degrees to 180 degrees is used as a learning sample for the degree of freedom of rotation. . In the case of parallel movement, a plurality of converted image data moved by 5 pixels vertically and horizontally are used as learning samples for the degree of freedom of translation. Since these learning samples include conversion degrees of freedom such as movement, the learning samples are generated by moving the area of the reference state from the image data in a region wider than the reference state while moving it by 5 pixels.
[0032]
In this way, for each of a plurality of image data examples, a plurality of image data that has been subjected to a plurality of conversions for each degree of freedom is generated, and information indicating what conversion has been performed on each image data (a large amount of conversion) Etc.).
[0033]
Here, in order to obtain image data having undergone conversions of different conversion amounts, the image data that has been converted while changing the conversion amount by a predetermined step (for example, 5 degrees in terms of rotation) is used as a learning sample. Although they are included, it is also possible to perform conversion while determining the conversion amount by a random number, and include each in the learning sample, instead of changing each predetermined step.
[0034]
Next, the conversion database 13a corresponding to each degree of freedom is learned using the learning sample for each degree of freedom.
[0035]
[Operation of the first conversion process]
The control unit 11 performs the conversion process as follows for each of the search areas defined by the search area defining process, using each of the conversion databases 13a learned in this way. That is, the image data portion included in the search area to be processed (for example, it can be equated with a vector value as a column of pixel values) is converted into N1-dimensional feature vector information (for each conversion database 13a) in the space F1. , That is, a set of feature values defined for each degree of freedom of conversion), and the mapping is projected onto each subspace Ω. Then, a conversion amount that minimizes the distance E between the feature vector information before projection and the feature vector information after projection is determined. Further, the control unit 11 calculates a square value L of the distance E, and stores this in the storage unit 12 as an error.
[0036]
Here, the conversion amount is determined for each degree of freedom based on each conversion database 13a, but the control unit 11 selects one of the conversion amounts corresponding to each degree of freedom as a predetermined condition (for example, corresponding to each conversion amount). The conversion corresponding to the selected degree of freedom is converted by an amount corresponding to the selected conversion amount.
[0037]
That is, from the image data portion included in the search area, the reference state is obtained by, for example, 10 degree rotation conversion with respect to the rotation degree of freedom according to information acquired by learning in each conversion database 13a corresponding to each degree of freedom. Since the error is Lr, and the degree of freedom of translation is approached to the reference state by conversion of 5 pixels to the left, and information such as Lp is obtained, the error is the smallest among these Select a transformation with degrees of freedom. For example, in the case of the above-described example, if Lr <Lp, a rotation search of 10 degrees is performed on the search area to define a new search area. Then, the image data portion included in the new search area is further mapped to the feature vector information in the space F1, and the mapping is further projected to each partial space Ω. And it repeats from the process which determines the conversion amount from which the distance E of the feature-value vector information before projection and the feature-value vector information after projection becomes the minimum.
[0038]
In addition, when the conversion amounts corresponding to the respective degrees of freedom all represent “0” (that is, no conversion), the process ends at that stage, and if there is an unprocessed search area, The conversion process is performed with any of the unprocessed search areas as a processing target.
[0039]
Here, the image data portion included in the defined search area is used as it is in the target image data. However, processing for reducing the resolution of the image data portion is performed to obtain coarse-grained data, and the coarse-grained data. You may perform a conversion process using data. In this case, each conversion database 13a is learned and acquired using a learning sample corresponding to the coarse-grained data.
[0040]
[Second conversion process]
First, the contents of the second conversion process will be described. One of the characteristic features of the present embodiment is that the second conversion process is performed step by step, that is, the conversion corresponding to one degree of freedom of conversion is sequentially performed once. is there.
[0041]
In order to determine the content of conversion and the amount of conversion to be performed at each stage, the control unit 11 obtains predetermined N2-dimensional feature vector information based on the image data portion included in the currently selected search area. Calculate. Here, the feature amount vector information is a vector amount including N2 feature amount elements selected in accordance with the property of the search target. In the present embodiment, first, the first conversion process is used to roughly approximate the reference state, and then the second conversion process is performed to more closely approximate the reference state. For this reason, the number of dimensions of the feature vector information in the second conversion process is made larger than that in the first conversion process (N2> N1). Accordingly, the second conversion process can be performed with higher accuracy.
[0042]
Similar to the first conversion process, the control unit 11 specifies the conversion by the kernel nonlinear subspace method using the N2 dimensional feature vector information and the feature vector information stored in the conversion database 11b. .
[0043]
[Contents of second conversion database 13b]
The second conversion database 13b is learned and acquired by the same method as the first conversion database 13a. However, in the learning process of the first conversion database 13a, based on the learning sample, the feature vector information of predetermined N1 dimensions is converted. In contrast to the conversion, the learning process of the second conversion database 13b differs from the learning sample by converting the learning sample into predetermined N 2 -dimensional feature vector information and performing learning using this.
[0044]
[Operation of second conversion process]
The control unit 11 further performs the second conversion process on each of the search areas converted by the first conversion process, using the second conversion database 13b that has been learned and acquired for each conversion degree of freedom. That is, the image data portion included in the search area to be processed (for example, it can be regarded as a vector value as a column of pixel values) is converted to N2 original feature vector information (each second conversion database) in the space F2. 13b, that is, a set of feature values defined for each degree of freedom of conversion), and the mapping is projected onto each subspace Ω. Then, the conversion amount that minimizes the distance E between the feature vector information before projection and the feature vector information after projection is determined. Further, the control unit 11 calculates a square value L of the distance E, and stores this in the storage unit 12 as an error.
[0045]
Here, as in the case of the first conversion process, the conversion amount is determined for each degree of freedom based on each second conversion database 13b, but the control unit 11 selects one of the conversion amounts corresponding to each degree of freedom. Selection is made based on a predetermined condition (for example, a condition such that the distance E corresponding to each conversion amount is minimum), and conversion corresponding to the selected degree of freedom is converted by the selected conversion amount.
[0046]
That is, in this case as well, in the same way as the first conversion process, from the image data portion included in the search area, for example, by the information acquired by learning in each second conversion database 13b corresponding to each degree of freedom, for example, free rotation For a degree, information is approached to a reference state by a rotational transformation of 10 degrees, and its error is Lr, and for a degree of freedom of translation, information is approached to a reference state by conversion of 5 pixels to the left, and the error is Lp. Therefore, the conversion of the degree of freedom that minimizes the error is selected from these. For example, in the case of the above-described example, if Lr <Lp, a rotation search of 10 degrees is performed on the search area to define a new search area. Then, the image data portion included in the new search area is further mapped onto the feature vector information in the space F2, and the mapping is further projected onto each partial space Ω. And it repeats from the process which determines the conversion amount from which the distance E of the feature-value vector information before projection and the feature-value vector information after projection becomes the minimum.
[0047]
On the other hand, if the conversion amount corresponding to each degree of freedom represents “0” (that is, no conversion), the process ends at that stage, and if there is an unprocessed search area, the unprocessed search area Conversion processing is performed with any one of the processing search areas as a processing target.
[0048]
In this case as well, the image data portion included in the defined search area is used as it is in the target image data. However, processing for reducing the resolution of the image data portion is performed to obtain coarse-grained data. You may perform a 2nd conversion process using data. In this case, each second conversion database 13b is learned and acquired using a learning sample corresponding to the coarse-grained data.
[0049]
In the first and second conversion processes, the control unit 11 does not sequentially perform processes such as calculation of feature vector information, mapping to a partial space, distance evaluation, and error evaluation for each degree of freedom. It may be performed in parallel.
[0050]
Furthermore, although the case where the kernel nonlinear subspace method is used has been described here as an example, other methods such as an auto encoder may be used as long as data classification and error evaluation at the time of classification are possible.
[0051]
Further, here, it has been described that the conversion of each degree of freedom is performed in the second conversion process similarly to the first conversion process. However, in the second conversion process, the number of degrees of freedom related to the conversion may be reduced. For example, only the conversion with the same degree of freedom as the conversion performed last in the first conversion process is performed, or the conversion degree of freedom is first selected in the second conversion process. The conversion process may be repeated only for the degrees of freedom selected in the first second conversion process.
[0052]
Further, the maximum value of the conversion amount as a category determined in the second conversion process may be set smaller than the maximum value of the conversion amount determined in the first conversion process. In other words, the learning sample of the second conversion database 13b used in the second conversion process may be a sample having a narrower conversion amount range than the learning sample of the first conversion database 13a. For example, in the learning sample for the first conversion database 13a, image data rotated by 10 degrees in a range from a rotation angle of −180 degrees to 180 degrees is used, and in the learning sample for the second conversion database 13b, the range and The step amount is reduced, and image data rotated by 3 degrees in a range from −30 degrees to 30 degrees is used.
[0053]
As described above, the conversion condition (at least one of the degree of freedom of conversion or the conversion amount) in the second conversion process may be limited as compared with the first conversion process.
[0054]
[Processing flow of control unit 11]
Specifically, the control unit 11 first sets the reduction rate S to the minimum reduction rate (for example, 1 time, that is, does not reduce) (S1), and reduces the target image data at the reduction rate S (S2). Then, an area of map data having a size equal to the size of the target image data after reduction is secured on the storage unit 12, the value of the area is set to “false”, and the map data is initialized. (S3). For example, if the target image data after reduction is image data of 1000 × 1000 pixels, an area for 1000 × 1000 bits is secured, and each bit value is initialized to “0”.
[0055]
Next, the control unit 11 performs a process of defining at least one search area for the reduced target image data (S4). The search area defining process will be described in detail later. Then, one of the search areas is selected (S5), and the first conversion process is executed for the search area (S6).
[0056]
Subsequently, the control unit 11 further executes a second conversion process for the search area resulting from the first conversion process (S7), and the image data portion included in the search area after the first and second conversion processes. (S8) is executed to determine whether or not a search target is included (S8). If it is determined that the search target is included (Yes), the value of the area on the map data corresponding to the search area after the conversion process is set to “true” (S9). ). Then, it is checked whether or not there is an unselected search area (S10). If there is an unselected search area (if Yes), the process returns to step S5 (A), and the unselected search area is selected. Select one of them and continue processing.
[0057]
On the other hand, when it is determined that the search target is not included as a result of the search process in the process S8 (when No), the process proceeds to the process S10 as it is. In addition, if there is no search area that has not been selected in process S10, that is, if the conversion process and the search process have been completed for all search areas (if No), the currently set reduction rate S is determined in advance. (S11), if not (if No), the reduction rate S is adjusted to be increased (S12), and the process returns to step S2 for processing. Continue (B). Here, the process S12 for largely adjusting the reduction rate S may be, for example, a process for increasing the reduction rate S by a predetermined ratio, or by multiplying the magnification which is the reduction rate S by a predetermined multiplier ΔS, S = A new reduction ratio S may be set as S × ΔS.
[0058]
If the currently set reduction rate S does not exceed the predetermined maximum reduction rate in Step S11 (if Yes), based on the map data corresponding to the target image data at each reduction rate, the original In the target image data (before reduction), an area including a search target is defined (S13), and the process ends.
[0059]
Here, after the first conversion process, the second conversion process is executed immediately. However, after the first conversion process, a preliminary search process (the contents will be described later) is performed, and this preliminary conversion process is performed. As a result of the search process, the second conversion process is performed only when it is determined that the search target may be included in the image data portion included in the search area after the first conversion process. In this case, if it is determined that there is no possibility that the search target is included in the image data portion included in the search area after the first conversion process, the process may be shifted to the above-described process S10 and the process may be continued. According to this, since the second conversion process is not performed on the area not including the search target, the processing load can be further reduced.
[0060]
[Search area definition processing]
Here, a process in which the control unit 11 demarcates a search area (search area demarcation process) will be described. The search area defining process can be performed using information on the start point input by the user or by autonomously defining an area that satisfies a predetermined condition.
[0061]
For example, when using information input from the user, the control unit 11 determines a predetermined size with each start point coordinate as an upper left corner based on information of at least one start point coordinate input from the operation unit 15 or the like. Are defined as search areas.
[0062]
When the area satisfying the predetermined condition is autonomously searched and demarcated, the control unit 11 starts the coordinates (for example, X = 0, Y = 0) of the upper left corner of the target image data (after reduction) as the starting point. As a result, it is checked whether or not a predetermined demarcation condition is satisfied for a rectangular area having a predetermined size. If the delimitation condition is satisfied, the process of setting the rectangular area as a search area When the start point deviates from the width of the target image data (X> the width of the target image data), the start point is moved by a predetermined amount in the height direction. (X = 0, Y = Y + ΔY) and repeat the process in the width direction. Thus, an area that satisfies the demarcation condition is defined as a search area in the entire target image data.
[0063]
Here, the moving amounts of the starting point are ΔX and ΔY in the width direction and the height direction, respectively, but these moving amounts are relative to ΔX and ΔY when the image data is 1 time depending on the reduction rate S of the target image data. ΔX / S, ΔY / S may be used.
[0064]
[Search process]
Next, the search process performed in process S8 will be described. In this search process, it is determined whether or not a search target is included using the search database 13c for each image data portion included in each search area for which the conversion process has been completed. As a specific example of the search process, there is a method disclosed in Japanese Patent Laid-Open No. 2002-329188. Next, the outline will be described.
[0065]
[Learning course of search database]
This search database 13c is formed by learning a neural network using an example of search target image data in a reference state as a learning sample. That is, the control unit 11 receives a plurality of learning samples, calculates a feature amount set (feature amount vector) selected in advance according to the properties of the search target for each of the learning samples, and generates learning data. To do. Next, using this learning data, a lattice space map is formed on the M × M ′ lattice space stored in the storage unit 12 by SOM (self-organizing map). That is, the control unit 11 calculates the distance between the feature vector that is the input learning data and the weight vector assigned to each grid with a predetermined measure (for example, Euclidean measure), and the distance is the minimum. A lattice (most matching node) c is detected. Then, the weight vectors of the plurality of grids near the most matched node are updated using the input feature vector. By repeating this process, a lattice space map is formed on the storage unit 12, and the best matching nodes for similar feature vectors form a continuous region. That is, in this lattice space, a non-linear projection from a feature vector, which is a multi-dimensional input signal, to a two-dimensional map is formed while maintaining the phase, and by updating the weights, feature portions of the data are organized, As a learning result, there are grids that react to similar data in close proximity.
[0066]
When the learning based on each learning data is completed, the control unit 11 classifies each lattice of the lattice space map into a category. This classification can be performed based on, for example, the distance between the lattices (the distance between the weight vectors associated with each lattice), and the category of the lattice group that reacts to image data similar to the search target (search target category). And other categories of lattice groups (non-search target categories).
[0067]
[Search operation]
The control unit 11 secures an area for storing map data having the same size as the target image data in the storage unit 12 and initializes the value of the area to “false”.
[0068]
The control unit 11 calculates a predetermined feature vector based on the image data portion of the search area for which the conversion process has been completed, using the learned and acquired search database 13c. Then, the distance between the calculated feature vector and the weight vector associated with each grid in the search database 13c is obtained, the grid (most matching node) having the minimum distance from the feature vector is specified, and the specified grid If it belongs to the search target category, it is determined that the search area includes the search target. If the specified lattice belongs to the non-search target category, the search area does not include the search target. to decide.
[0069]
[Preliminary search processing]
A preliminary search process for determining whether to perform the second conversion process will be described. This preliminary search process may be the same process as the search process performed in the above-described process S8, but it is also preferable to make the process simpler in consideration of the processing load of the search process itself.
[0070]
For example, as the preliminary search process, a predetermined feature amount is calculated based on the image data portion included in the search area after the first conversion process, and the search is performed based on whether the feature amount satisfies a predetermined condition. It may be determined whether there is a possibility that the target is included. As an example, when a human face is a search target, the entropy is higher in the contour portion of the human face than in other portions. Therefore, the entropy is calculated as a feature amount, and a predetermined threshold (for example, target image data) is calculated. When the calculated entropy value is higher than the threshold value determined based on the overall average entropy value), it is determined that the search target may be included.
[0071]
[Operation of Control Unit 11]
As described above, the control unit 11 according to the present embodiment sequentially extracts the target image data to be searched, extracts the area to be searched from each of the reduced target image data, The first conversion processing is executed to bring the image data portion closer to the reference state, and then a preliminary search is made as to whether or not the search target is included in the image data portion after the first conversion processing.
[0072]
If it is determined that there is a possibility that the search target is included as a result of this preliminary search, the second conversion process is executed so as to be closer to the reference state, and the search target is the search after the conversion. It is determined whether it is included in the image data in the area. In other words, the control unit 11 first performs the process of setting the image data to be searched for as a reference state roughly, and then performs the process finely, thereby reducing the overall processing load.
[0073]
Since the conversion process to the reference state is performed in this way, in the present embodiment, the position of the search area before conversion may be slightly shifted. Even if the reduction ratio is slightly deviated from the reference state, there is no problem. Therefore, conventionally, multi-stage reduced image data reduced by 0.8 times is generated and the search area is extracted while being shifted by one pixel. It is possible to reduce by 5 times, and ΔX and ΔY can be set to 6 pixels or the like even when a region that satisfies a predetermined condition is autonomously extracted when defining a search region. As a result, the number of patterns to be searched can be significantly reduced, and the processing load for searching for a search object from a photograph or the like can be reduced.
[0074]
The control unit 11 generates map data representing an area determined to include the search target as search result information. This map data corresponds to each of the target image data after being reduced at each reduction rate. Are generated. Therefore, an area including the search target is determined by using the plurality of map data (each having the reduced target image data size) in an integrated manner.
[0075]
For example, each map data is enlarged at an enlargement ratio corresponding to the respective reduction ratios, compared with the size of the original target image data, and a region that is “true” in common to all map data ( Based on the target image data of any reduction ratio, it may be determined that the search target is included in an area where the search target is determined to be included. Further, an area that is “true” in any one of the map data may be determined as an area that includes a search target.
[0076]
[Variation of search area definition processing]
Further, in the search area defining process, the control unit 11 determines the entropy, hierarchical entropy, color, luminance distribution, and two or more of these values of the image data included in the area to be defined as the search area. A combination may be used to determine whether or not the search area is actually defined. For example, when searching for a face part of a person, the entropy is high in the peripheral part (outline part) of the face, so if the entropy is larger than a predetermined threshold, the area to be defined as the search area is actually A search area is defined. Otherwise, the area is not set as a search area, and other processes are continued. According to this, the search area to be subjected to the conversion process / search process can be rationally reduced, and the processing load can be reduced.
[0077]
[Other variations]
In the description so far, the conversion performed in the conversion process is the two-dimensional rotation, translation, enlargement / reduction of the search area (the search area is enlarged / reduced, and the original image data portion is enlarged (before the enlargement / reduction). As a result, the image data portion included in the search area is converted. However, other than this, for example, if it is a human face Further, conversion may be included for the image data portion itself, such as a three-dimensional rotation influenced by the posture (growth adjustment or swing direction adjustment). Specifically, such a three-dimensional rotation can be realized by assuming an average three-dimensional model to be searched and projecting an image data portion on the average three-dimensional model.
[0078]
Further, in the search process, for example, even a human face may be further classified into categories and may include conditions such as age, sex, and whether or not the mouth is open.
[0079]
Furthermore, in the flowchart of FIG. 2, the control unit 11 sequentially performs processing at each reduction ratio. However, since the processing at each reduction ratio is independent from each other, the control unit 11 performs processing at each reduction ratio. Processing may be performed in parallel.
[0080]
[Operation]
Next, the operation of the image search device of the present embodiment will be described by taking as an example a case where a human face portion is searched for as a search target from image data of a photograph given as target image data. In the following example, for the sake of simplicity, it is assumed that the degree of freedom of conversion is only translation (x, y) and rotation (θ).
[0081]
The control unit 11 stores target image data input from the outside such as an external interface or a communication unit (not shown) in the storage unit 12, and demarcates the search area by the method described above. For example, the control unit 11 divides the target image data into image blocks of a predetermined size (parts of which may overlap each other), and for example, defines an image block whose entropy is higher than a predetermined value as a search region. Then, one of the defined search areas is selected, and the first conversion process is executed for the selected search area. In this first conversion process, conceptually, from the reference state O, the image data portion in the range R1 as shown in FIG.
[0082]
The control unit 11 further performs a preliminary search to determine whether or not the face of the search target person is included in the image data portion after the first conversion process. This preliminary search process may be the same as the main search process using a neural network, or may be a simple process using a feature quantity such as entropy. When it is determined as a result of the preliminary search process that the image data portion after the first conversion process includes the face of the person to be searched for, the control unit 11 continues to the image data portion. The second conversion process is executed on the search area. In this second conversion process, conceptually, as shown in FIG. 3B corresponding to FIG. 3A, the degree of freedom is, for example, only the θ direction (rotation), and the conversion range is narrow. The process (R2) is applied, and fine adjustment is performed to bring the image data portion closer to the reference state O.
[0083]
The control unit 11 performs a search process on the image data portion included in the search area after the second conversion process, checks whether the search target is included, and if the search target is included, Information for specifying the portion of the target image data corresponding to the search area where the search processing has been performed is generated.
[0084]
As described above, in this embodiment, the processing load can be reduced and the search is performed closer to the reference state as compared to the case of the conversion process using only one conversion database by performing the two-stage conversion process. Therefore, the search accuracy can be improved.
[0085]
In the above description, a case where two-stage conversion processing is performed is shown as an example, but it may be performed three or more times. In this case, the conversion conditions in the conversion process may be further limited as the level is increased, or the number of dimensions of the feature vector information may be further increased.
[0086]
【Example】
FIG. 4A is a distribution diagram showing how much error has occurred in the image data after the conversion to the reference state by the conversion process using one type of conversion database from the actual reference state. Yes, FIG. 4B shows, as in this embodiment, when two types of conversion databases are used, and the conversion processing is performed in two steps, first coarsely and then finely. It is a distribution diagram showing how much error image data has produced from the actual reference state. As can be understood with reference to the drawings, according to the method of the present embodiment, the average error is reduced, and conversion is performed so that the distribution state is closer to the reference state.
[0087]
Further, in the case of the conversion process using the one type of conversion database, an average of about 20 conversion processes are required before convergence (unit conversion for each degree of freedom is performed approximately 20 times). When the two types of conversion databases are used, the rough conversion process is performed about 10 times and the fine conversion process is performed about 5 times, and the processing load of the conversion process itself is reduced as a whole.
[Brief description of the drawings]
FIG. 1 is a configuration block diagram of an image search apparatus according to an embodiment of the present invention.
FIG. 2 is a flowchart illustrating an example of processing of a control unit 11.
FIG. 3 is an explanatory diagram showing an outline of conversion processing performed by the image search apparatus according to the embodiment of the present invention.
FIG. 4 is an explanatory diagram showing an experimental result comparing a result of a conversion process using one type of conversion database and a result of a two-stage conversion process using two types of conversion databases.
[Explanation of symbols]
11 control unit, 12 storage unit, 13 database unit, 14 display unit, 15 operation unit, 16 external storage unit.

Claims (8)

An image search device for searching a search target image data portion to be searched from target image data to be processed,
Means for defining at least one search area in the target image data;
With reference to a plurality of first conversion databases learned and acquired in advance for each conversion method, N1 conversion methods and conversion amounts to be applied to each search partial data included in each of the defined search areas are determined. First conversion means for acquiring a conversion condition including, and performing conversion based on the acquired conversion condition at least once for each search partial data;
Each search partial data converted by the first conversion means is further limited to N2 conversion methods (N2 is different from N1) compared to the conversion in the first conversion means , and the amount of conversion is limited. A conversion condition is acquired by referring to a plurality of second conversion databases that have been acquired in advance for each conversion method, and conversion based on the acquired limited conversion condition is performed at least once for each of the converted search partial data. Second search means for converting each search partial data into a reference state ;
Including
A search database acquired by learning using image data in the reference state is referred to in advance to determine whether a search target is included in each of the converted search partial data, and the determination result is output. An image search apparatus characterized by: The image search device according to claim 1,
For each search partial data converted by the first conversion means, it further comprises preliminary search means for determining whether or not a search target is included,
An image search apparatus characterized in that the second conversion means converts only search partial data determined to contain a search target by the preliminary search means. The image search device according to claim 2,
The preliminary search means refers to the search database acquired by learning using the search target image data example for the search partial data converted by the first conversion means, and sets each of the search partial data after conversion. An image search apparatus for determining whether or not a search target is included. In the image search device according to claim 3,
The means for defining the search area is:
Using at least one of entropy, hierarchical entropy, color, and luminance dispersion of image data included in an area to be defined as a search area, and determining whether to actually demarcate as a search area An image search device. In the image search device according to claim 3,
The means for defining the search area is:
When the entropy of the image data included in the area to be defined as the search area is larger than a predetermined threshold, the area to be defined as the search area is actually defined as the search area An image search device. An image search program for searching for a search target image data portion to be searched from target image data to be processed, the computer comprising:
Means for defining at least one search area in the target image data;
With reference to a plurality of first conversion databases learned and acquired in advance for each conversion method, N1 conversion methods and conversion amounts to be applied to each search partial data included in each of the defined search areas are determined. First conversion means for acquiring a conversion condition including, and performing conversion based on the acquired conversion condition at least once for each search partial data;
Each search partial data converted by the first conversion means is further limited to N2 conversion methods (N2 is different from N1) compared to the conversion in the first conversion means , and the amount of conversion is limited. A conversion condition is acquired with reference to a plurality of second conversion databases that have been learned and acquired in advance for each conversion method, and conversion based on the acquired limited conversion condition is performed at least once for each of the converted search partial data. Second search means for converting each search partial data into a reference state ;
A search database acquired by learning using image data in the reference state is referred to in advance to determine whether a search target is included in each of the converted search partial data, and the determination result is output. Means to
A program characterized by functioning as The image search program according to claim 6,
For each search partial data converted by the function as the first conversion means, it is determined whether or not a search target is included, and only the search partial data determined to include a search target is 2. An image search program further comprising means for causing the computer to function so as to function as two conversion means. The image search program according to claim 7, wherein
When functioning as the preliminary search means, the search portion data after being converted by the first conversion means is referred to a search database acquired by learning using an example of image data to be searched, and the search portion after the conversion An image search program for determining whether or not a search target is included in each piece of data.

Images